Feature Representation in Machine Learning Pipelines

Preprocessing adjusts features so that the data representation is more suitable for algorithms, and some algorithms, such as neural networks and SVMs, are very sensitive to the scaling of the data.

Programming

Two Columns on Different Scales

Most real datasets mix measurements that have nothing to do with each other. A person's height might be recorded in centimeters while their annual income is recorded in currency units. The numbers in one column can be a hundred times larger than the numbers in another, even though neither column is more important than the other. Before any learning algorithm runs, it is worth asking a practical question: are the features measured on comparable scales?

The step that answers this question is called preprocessing. Preprocessing adjusts the features so that the data representation is more suitable for the algorithm that will consume it. This matters because some algorithms, such as neural networks and support vector machines, are very sensitive to the scaling of the data. For those methods, the raw size of a feature's numbers can shape the result, not just the pattern hidden in the feature.

Read the next section as a trace. We will start with a tiny table, watch what happens to each column when it is adjusted, and then see how the same change can be written in Python.

Tracing a Small Feature Table

Consider a tiny table with three samples and two features: height in centimeters and weight in kilograms. The heights run from 150 to 200, and the weights run from 40 to 120. The height column therefore spans 50 units, while the weight column spans 80 units. The absolute numbers sit in different ranges altogether. For the middle sample, use a height of 160 cm and a weight of 70 kg. These middle values are an illustrative choice made for this walkthrough.

Sample	Height (cm)	Weight (kg)
First	150	40
Middle	160	70
Last	200	120

The raw feature table before any adjustment

Now trace one column at a time. In the height column, the first sample is at the minimum, the last sample is at the maximum, and the middle sample sits 10 units above the minimum out of a total span of 50. In the weight column, the first sample is again at the minimum, the last sample is at the maximum, and the middle sample sits 30 units above the minimum out of a total span of 80. Each column is judged against its own range, not against the other column.

Shift and Rescale, Column by Column

The source describes the typical adjustment as a simple per-feature rescaling and shift of the data. The word per-feature matters. Each column is adjusted on its own terms, using the values in that column, rather than the whole table being transformed by one global rule. After the adjustment, every feature sits on a scale that the algorithm can handle comfortably.

A shift moves the values so that they start from a different reference point, and a rescale changes how spread out they are. Together, these operations bring columns that were measured in very different units onto a comparable footing. The source does not commit to one particular rescaling recipe in this passage, so the rule below is one concrete way to carry out a per-feature rescale and shift, not the only one.

Formula

In this rule, x is one raw value from a column, min(x) and max(x) are the smallest and largest values in that same column, and x' is the adjusted value. Subtracting the minimum performs the shift, so the column starts at zero. Dividing by the span performs the rescale, so the column ends at one. Each column needs its own minimum and span, which is why the computation is per-feature.

Before and After the Rescale

The before-and-after picture shows the core idea. Each column is transformed independently, and the result is that both features occupy the same range from zero to one. The relationship between the samples within each column is preserved, since the rescaling is a consistent transformation of that column. What disappears is the raw magnitude advantage that the weight column would otherwise have over the height column.

When each column is min-max scaled, where does each original value land on the new scale, and how do the minimum and maximum move?

Both columns now run from 0 to 1. The minimum of each column became 0, and the maximum of each column became 1. The middle values moved to positions that reflect where they sat inside their own column: 0.2 for height and 0.375 for weight. The weight column, which had the larger raw spread, no longer carries a larger numeric footprint than the height column.

Running the Rescale in Python

The same idea can be carried out by hand in plain Python, which makes the per-feature logic visible. The program below applies the minimum-and-span rescale to every column of the small table, computing the statistics for each column separately. The samples are stored as rows, so the program first transposes the table into columns, rescales each column, and then transposes back to rows.

Code walkthrough

samples = [
    [150, 40],
    [160, 70],
    [200, 120],
]

columns = list(zip(*samples))
scaled_columns = []
for col in columns:
    lo, hi = min(col), max(col)
    span = hi - lo
    scaled_columns.append([(v - lo) / span for v in col])

scaled = [list(row) for row in zip(*scaled_columns)]
for row in scaled:
    print(row)

1
Line 1 to 5: the table is stored as three rows, each holding a height and a weight.
2
Line 7 zip(*samples) groups the first values together and the second values together, giving two columns.
3
Line 9 to 12: for each column, the code finds its own minimum and maximum, computes the span, and rescales every value in that column.
4
Line 14 zip(*scaled_columns) turns the rescaled columns back into rows, one row per sample.
5
Line 15 to 16: each printed row is one sample, and each position inside the row is one feature.

samples holds one row per sample, with height first and weight second
zip(*samples) turns rows into columns so each feature can be handled on its own
min and max are taken from one column at a time, which is what makes the rule per-feature
each value is shifted by the column minimum and divided by the column span

Output

[0.0, 0.0]
[0.2, 0.375]
[1.0, 1.0]

Read the output row by row. The first sample sits at the minimum of both features, so it maps to zeros. The last sample sits at the maximum of both, so it maps to ones. The middle sample lands between them, with each value reflecting its own column's range. Inside a printed row, the first position is the height feature and the second position is the weight feature.

What do you think happens?

After the per-feature rescale, what value does the middle sample take for weight?

Reveal answer

Answer:

0.375

The weight of 70 sits 30 units above the column minimum of 40, and the weight column spans 80 units, so 30 divided by 80 gives 0.375.

Mistakes When Rescaling Features

Common mistake
Computing one minimum and one span for the whole table and applying it to every column
What it looks like
Using the smallest value anywhere in the table as the minimum for both height and weight
Why it fails
The source emphasizes that the word per-feature matters. Each column is adjusted on its own terms, so a single global rule ignores the different ranges of the columns.
Do this instead
Compute the minimum and span from each column's own values, as the per-column loop in the Python example does.
Common mistake
Thinking that the target labels are needed to rescale the features
What it looks like
Passing the answers for each sample into the scaling step alongside the features
Why it fails
Scaling methods do not use supervised information. They look only at the features, which is why they count as unsupervised even inside a supervised workflow.
Do this instead
Rescale using the feature columns alone, and keep the labels separate for the learning step.
Common mistake
Assuming that the rescaling recipe shown here is the only correct one
What it looks like
Treating the minimum-and-span rule as the definition of preprocessing
Why it fails
The source does not commit to one particular rescaling recipe in this passage. The minimum-and-span rule is one concrete way to shift and rescale.
Do this instead
Learn the goal, which is making the features comparable, and treat the minimum-and-span formula as one method for reaching it.
Common mistake
Forgetting that a column with a span of zero (all values equal) cannot be divided by its span without special handling

Where Scaling Sits in a Pipeline

Preprocessing is a step that runs before the algorithm. Raw features enter the pipeline, the per-feature rescale adjusts them, and only then does the transformed data reach the learning algorithm. The algorithm therefore never sees the raw table in this setup. It sees the adjusted representation, which is why the adjustment can change what the algorithm sees.

The source names neural networks and support vector machines as methods that are very sensitive to scaling. For those methods, a feature with a much larger raw range can influence the result more than its importance warrants. Rescaling is the step that removes that artificial advantage before the algorithm starts working.

Practice: Rescale a New Column

EASY

Using the minimum-and-span rule, rescale a single weight column with the values 60, 90, and 120. Predict the three scaled values before you compute them, then check your answers. Explain in one sentence what the middle value tells you about where that sample sits inside its column.

Hints

Find the minimum and the maximum of the column first. Here the minimum is 60 and the maximum is 120.
The span is the maximum minus the minimum, which is 60.
For each value, subtract the minimum and then divide by the span.

Your attempt (not saved)

Key Takeaways

Preprocessing adjusts features so that the data representation is more suitable for the algorithm, and some methods such as neural networks and SVMs are very sensitive to feature scale.
A per-feature rescale and shift adjusts each column using its own values, so columns with large raw numbers no longer dominate by magnitude alone.
Scaling methods use only the feature values, not the labels, which is why they count as unsupervised even inside supervised workflows.
In Python, zip can turn rows into columns so that each feature's minimum and span can be computed and applied separately before the data reaches the learning algorithm.

More concepts starting with F