Scaling Features for Sensitive Algorithms

Preprocessing adjusts features so that the data representation is more suitable for algorithms, and some algorithms, such as neural networks and SVMs, are very sensitive to the scaling of the data.

Programming

When Raw Numbers Mislead an Algorithm

Almost every real dataset raises the same practical question before any learning happens: are the features measured on comparable scales? A table might record a person's height in centimeters and their annual income in currency units. The numbers in one column can be a hundred times larger than the numbers in another, even though neither column is more important than the other. The size of the numbers is an accident of the units chosen, not a statement about how much each feature matters.

This matters because some algorithms are very sensitive to the scaling of the data. The source names neural networks and support vector machines as examples. For these methods, the raw size of a feature's numbers can shape the result, not just the pattern hidden in the feature. Preprocessing is the step that adjusts the features before the algorithm sees them, so that the data representation is more suitable for the algorithm. This article follows one such adjustment from start to finish: tracing the numbers by hand, writing the logic in Python, and reading what the output means.

Before turning to the mechanics, keep one question in mind as you read: if the algorithm only sees the numbers and not the units they came from, what would a column of large numbers do to its view of the data?

Two Columns on Different Ranges

Consider a tiny table with three samples and two features: height in centimeters and weight in kilograms. The heights run from 150 to 200, and the weights run from 40 to 120. The height column spans 50 units while the weight column spans 80 units, and the absolute numbers live in different ranges altogether. Neither span is the 'right' one; they are simply what the units produce.

The middle sample is not given in the source, so this article uses a generated value for illustration: 175 cm and 80 kg. That sample sits halfway between the extremes in both columns. Your job is to trace what happens to each column when it is rescaled on its own.

Each column's values move when its own minimum is subtracted and the result is divided by its own span. Both columns end up on the same range from 0 to 1, while the order of samples within each column stays the same.

Read the diagram column by column. Before rescaling, the height values and the weight values are not directly comparable: one column is measured around 150 to 200 and the other around 40 to 120. After rescaling, each column runs from 0.0 at its smallest sample to 1.0 at its largest. The rescaling of each column used only that column's own values, which is why both columns can now sit on a common footing.

What a Per-Feature Rescale and Shift Does

The source describes the typical adjustment as a simple per-feature rescaling and shift of the data. The word per-feature matters. Each column is adjusted on its own terms, using the values in that column, rather than the whole table being transformed by one global rule. After the adjustment, every feature sits on a scale that the algorithm can handle comfortably.

A shift moves the values so that they start from a different reference point, and a rescale changes how spread out they are. Together, these operations bring columns measured in very different units onto a comparable footing. The source does not commit to one particular rescaling recipe in this passage, so the recipe used in this article is one concrete way to carry out a per-feature rescale and shift, not the only one.

The recipe used here is the minimum-and-span rescale. For a column j, find its smallest value and its span (largest minus smallest). Then each value in that column is shifted by subtracting the minimum and rescaled by dividing by the span:

Formula

Here $x_{ij}$ is the original value in sample i and feature j, and $x'_{ij}$ is the rescaled value. The minimum $\min_j$ and maximum $\max_j$ are computed from column j alone. The relationships among samples within a column are preserved, since the rescale is a consistent transformation of that column, but the raw magnitudes that would otherwise dominate the algorithm are removed.

The scale and shift are computed from the feature columns only. The rescaled features go to the algorithm, while the label column bypasses scaling entirely and reaches the algorithm unchanged.

The label column never enters the scaling step. This is what the source means when it says scaling methods do not use supervised information. Preprocessing is often paired with supervised learning, where a model is trained on examples that come with known answers, but the scaling itself looks only at the features. That is why scaling counts as an unsupervised technique even when it appears in a supervised workflow.

Tracing the Rescale by Hand

Worked example

Min-and-span rescale on the height column

Heights are 150 cm, 175 cm, and 200 cm. Apply the minimum-and-span rescale to this column and interpret the three results.

1
Find the minimum
The smallest height is 150 cm, so the shift is 150.
2
Find the span
The largest height is 200 cm. The span is 200 minus 150, which equals 50.
3
Rescale the first sample
For 150 cm, the shifted value is 150 minus 150, which equals 0. Divided by 50, the result is 0.0.
4
Rescale the middle sample
For 175 cm, the shifted value is 175 minus 150, which equals 25. Divided by 50, the result is 0.5.
5
Rescale the last sample
For 200 cm, the shifted value is 200 minus 150, which equals 50. Divided by 50, the result is 1.0.

Result

The column now runs from 0.0 to 1.0. The first sample sits at the minimum, the last sample sits at the maximum, and the middle sample lands halfway between them.

Repeat the same steps on the weight column, using its own minimum of 40 kg and its own span of 80 kg. The weight of 80 kg is 40 units above the minimum, so it becomes 40 divided by 80, which is 0.5. Notice that the height column never influenced the weight numbers. Each column was handled on its own terms.

Applying Per-Feature Scaling in Python

The same idea can be carried out by hand in plain Python, which makes the per-feature logic visible. The snippet below applies the minimum-and-span rescale to every column of a small table, computing the statistics for each column separately. The source introduces its scaling discussion with a plotting call from the mglearn helper library, which shows several rescaling options side by side; the snippet here is a plain-Python version of one of those options.

What do you think happens?

Before reading the output, predict what the middle row will look like after both columns are rescaled.

Reveal answer

Answer:

[0.5, 0.5]

The middle sample is 175 cm and 80 kg. In each column it is exactly halfway between the minimum and maximum, so each rescaled value is 0.5.

python

data = [
    [150, 40],
    [175, 80],
    [200, 120],
]

num_features = len(data[0])
scaled = [row[:] for row in data]

for j in range(num_features):
    column = [row[j] for row in data]
    lo = min(column)
    span = max(column) - lo
    for i, row in enumerate(data):
        scaled[i][j] = (row[j] - lo) / span

for row in scaled:
    print(row)

data = [
    [150, 40],
    [175, 80],
    [200, 120],
]

num_features = len(data[0])
scaled = [row[:] for row in data]

for j in range(num_features):
    column = [row[j] for row in data]
    lo = min(column)
    span = max(column) - lo
    for i, row in enumerate(data):
        scaled[i][j] = (row[j] - lo) / span

for row in scaled:
    print(row)

Output (expected)

[0.0, 0.0]
[0.5, 0.5]
[1.0, 1.0]

Walk through it

1
Lines 1 to 5 define the table: three samples, each with a height and a weight.
2
Line 7 counts the features, here 2, and line 8 makes a copy of the table so the original values survive.
3
The loop on line 10 runs once per column, and line 11 collects that column's values.
4
Lines 12 and 13 compute the column's minimum and span from its own values.
5
The inner loop on lines 14 and 15 rescales each entry in that column.
6
The final loop prints each rescaled sample row by row.

Notes on the code

The outer loop visits each feature index j, so every column is handled on its own.
lo and span are computed from column j only, which is the per-feature part of the rule.
Each value is shifted by subtracting lo and divided by span, matching the formula in the previous section.
The original data list is left unchanged; the rescaled values are written into a copy.

Read the output row by row. The first sample sits at the minimum of both features, so it maps to zeros. The last sample sits at the maximum of both, so it maps to ones. The middle sample lands between them, with each value reflecting its own column's range. The 0.0 and 1.0 values are relative to this dataset: they say where a sample falls within the observed range of its column, not an absolute measurement in centimeters or kilograms.

Mistakes When Scaling Features

Common mistake
Treating a larger raw number as a more important feature
What it looks like
Assuming the weight column matters more because its values run up to 120 while heights only reach 200 after a shift.
Why it fails
The source is explicit that the size of a column's numbers reflects its units, not its importance. Neither column is more important than the other.
Do this instead
Rescale each feature to a common range so that the raw magnitude no longer decides how much the algorithm pays attention to it.
Common mistake
Using label information to compute the scale
What it looks like
Computing the minimum and span using the target column along with the features.
Why it fails
Scaling methods do not use supervised information. They look only at the features, which is why they count as unsupervised.
Do this instead
Compute the minimum and span from the feature columns only, and let the label column pass through unchanged.
Common mistake
Scaling the whole table with one global rule
What it looks like
Finding one minimum and one span across every number in the table and applying it to all columns.
Why it fails
The adjustment is per-feature. Each column has its own range, so one shared rule would not put each column on the same footing.
Do this instead
Compute a separate minimum and span for each column, as the loop in the Python example does.
Common mistake
Expecting rescaling to change the relationships within a column
What it looks like
Believing that after rescaling, the middle sample no longer comes between the other two.
Why it fails
The rescale is a consistent transformation of each column, so the ordering and relative positions of samples within a column are preserved.
Do this instead
Check that the rescaled column keeps the same order of samples, now running from 0.0 to 1.0.
Common mistake
Reading 0.0 and 1.0 as absolute measurements rather than positions within the observed range

Checking Your Understanding

EASY

A column holds the values 10, 20, 30, and 50. Apply the minimum-and-span rescale to this column. What are the four rescaled values, and which sample lands halfway between the minimum and maximum?

Hints

The minimum is 10 and the maximum is 50, so the span is 40.
For each value, subtract 10 and then divide by 40.
The value 30 is halfway between 10 and 50, so it should map to 0.5 when computed correctly.

Your attempt (not saved)

Summary of Per-Feature Scaling

1
Some algorithms, such as neural networks and support vector machines, are very sensitive to the scaling of the data, so raw magnitudes can shape the result.
2
Scaling is a per-feature rescale and shift: each column is adjusted using only its own values, so columns measured in different units end up on a comparable footing.
3
Scaling methods do not use label information, which is why they count as unsupervised even inside a supervised workflow.
4
The minimum-and-span rescale subtracts each column's minimum and divides by its span, so the smallest value maps to 0 and the largest to 1 while the order of samples within each column is preserved.

At a glance

Some algorithms, such as neural networks and support vector machines, are very sensitive to the scaling of the data, so raw magnitudes can shape the result.
Per-feature rescaling and shifting adjusts each column using only its own values, bringing features measured in different units onto a comparable range.
Scaling methods do not use label information, so they count as unsupervised even when used in a supervised workflow.
The minimum-and-span rescale maps each column's smallest value to 0 and largest to 1, preserving the order of samples within each column.

Related concepts

More concepts starting with S