Feature Manipulation and Normalization
Feature transformations should be selected according to the property that needs to change.
Choosing the Property to Change
A feature is not always presented in the most useful numerical form. Its values may be shifted away from zero, cover an inconvenient range, contain unusually high or low values, or represent counts whose differences are not equally meaningful. Feature manipulation and normalization provide several ways to change that representation. The central rule is simple: identify the property that needs to change first, then select the transformation that addresses it.
Do not choose a transformation merely because it is familiar. Choose it according to whether the problem concerns location, spread, extreme values, or the meaning of differences across a count scale.
Tracing Location and Spread
Centering, scaling, and standardization all reorganize the numerical position of a feature, but they do different jobs. Centering changes location by subtracting the empirical mean from every value. Scaling changes the range, commonly placing values between 0 and 1 or between −1 and 1. Standardization combines a location adjustment with a variance adjustment: it subtracts the empirical mean and divides by the standard deviation, producing a feature with zero mean and unit variance.
| Transformation | Primary property changed | Typical result |
|---|---|---|
| Centering | Location | The empirical mean becomes zero |
| Scaling | Range | Values are commonly placed between 0 and 1 or between −1 and 1 |
| Standardization | Location and variance | The feature has zero mean and unit variance |
The different jobs performed by location-and-spread transformations
A Small Feature Trace
Selecting a location or spread adjustment
Suppose a generated feature contains values whose average is far from zero, and a separate feature contains values across an inconvenient numerical range. Which transformation matches each problem?
Inspect the first feature: The problem is its location: the values are shifted away from zero. Centering is appropriate because it subtracts the empirical mean from every value.
Inspect the second feature: The problem is its range. Scaling is appropriate because it changes the range, commonly placing values between 0 and 1 or between −1 and 1.
Consider both properties: If both the location and the variance need adjustment, standardization combines subtraction of the empirical mean with division by the standard deviation.
Use centering for a location problem, scaling for a range problem, and standardization when both mean and variance need adjustment.
Reshaping Extreme and Count Values
Clipping, sigmoidal transformation, and logarithmic transformation do not primarily reposition a feature around its mean or place it inside a fixed range. They alter how values and differences are represented, especially when extreme values or unevenly meaningful count differences are a concern.
Clipping limits high or low feature values to a specified range. It directly prevents values beyond that range from remaining as large or small as they originally were.
A sigmoidal transformation is a softer alternative for extreme values. Values close to zero are affected only slightly, while values far from zero behave similarly to clipping.
A logarithmic transformation is especially useful for count features when equal numerical gaps do not carry equal meaning. It compresses the large-count end relative to the small-count end.
Count Features in Context
Why a count may need reshaping
Consider a generated word-count feature. Compare the meaning of moving from zero occurrences to one occurrence with moving from 1000 occurrences to 1001 occurrences.
Compare the numerical gaps: Both changes increase the count by one, but the first change can matter much more than the second.
Identify the mismatch: Equal numerical gaps do not necessarily represent equal importance across the count scale.
Select a transformation: A logarithmic transformation is useful because it compresses the large-count end relative to the small-count end.
For a count feature with unevenly meaningful differences, consider a logarithmic transformation rather than treating every numerical gap as equally important.
What do you think happens?
A feature contains a few unusually high values, and you want a direct limit on how high or low values may be. Which transformation best matches that goal?
Reveal answer
Answer: Clipping
Clipping limits high or low feature values to a specified range. A sigmoidal transformation would be a softer alternative when extreme values should be treated less strongly rather than directly limited.
Matching Problems to Transformations
- Use centering when the feature's location, represented by its empirical mean, needs adjustment.
- Use scaling when the feature's range is inconvenient.
- Use standardization when both the empirical mean and variance need adjustment.
- Use clipping when high or low values should be limited to a specified range.
- Use a sigmoidal transformation when extreme values need a softer treatment: values near zero should change little, while far-from-zero values should behave similarly to clipping.
- Use a logarithmic transformation for count features when large-count differences should be compressed relative to small-count differences.
Common Selection Mistakes
Treating centering, scaling, and standardization as interchangeable.
Centering adjusts location, scaling adjusts range, and standardization adjusts both mean and variance.
Fix:
Name the property that needs to change before choosing the operation.Using a location-and-spread adjustment to solve a value-meaning problem.
Standardization changes mean and variance, while a logarithmic transformation reshapes the count scale by compressing the large-count end.
Fix:
Use a logarithmic transformation when the issue is the relative meaning of differences across a count range.Assuming all extreme-value transformations behave in the same way.
Clipping directly limits high or low values to a specified range, whereas a sigmoid is a softer alternative: values near zero change little and far-from-zero values behave similarly to clipping.
Fix:
Choose clipping for a direct limit and a sigmoid for softer treatment of extremes.Applying a transformation without diagnosing the feature.
Different transformations address different properties, and transformations can also be combined when more than one property needs adjustment.
Fix:
Inspect whether the issue concerns location, range, variance, extreme influence, or unevenly meaningful count differences.
Practice the Selection Rule
For each generated situation, select the transformation that best matches the property needing change: a feature shifted away from zero; a feature covering an inconvenient range; a feature requiring both mean and variance adjustment; a feature with unusually high and low values that must be limited; a feature with extreme values that should be softened rather than directly limited; and a count feature where large-count differences matter less than small-count differences.
Hints
- Separate location and range problems from problems involving extreme-value influence.
- For the two extreme-value cases, distinguish a direct limit from a softer response.
- For the count case, consider whether equal numerical gaps have equal meaning.
- Feature transformation is a property-matching task. Centering changes location by subtracting the empirical mean. Scaling changes the range. Standardization adjusts both mean and variance. Clipping directly limits extreme values, while a sigmoidal transformation treats extremes more softly. A logarithmic transformation is useful for count features when the large-count end should be compressed relative to the small-count end. Transformations may be combined, but the desired change should be identified first.
Key Takeaways
- Choose a feature transformation according to the property that needs to change.
- Centering adjusts location, scaling adjusts range, and standardization adjusts both mean and variance.
- Clipping and sigmoidal transformation address extreme values differently: clipping imposes a direct limit, while a sigmoid provides a softer response.
- Logarithmic transformation compresses the large-count end when count differences are not equally meaningful across the scale.
- Transformations can be combined when more than one property needs adjustment.