Understanding the Pipeline Class

A pipeline stores its steps as a list of (name, estimator) tuples, and the estimator for step i sits at index 1 of that tuple.

Programming

Why Trace a Pipeline Step by Step

A pipeline is a chain of steps that data flows through, one step after another. Its behavior is easiest to understand by watching the data move, not by reading its definition alone. This article uses a small pipeline whose two steps are a transformer that adds one to every number, and a last step that labels each number as low or high by comparing it with a cutoff. Read the code, predict what each step receives, and then check your prediction against the trace of what actually happens.

By the end of this article you should be able to follow the data through both fitting and prediction, explain why only the last step must have a fit method, and find the exact step where a surprising output first appears. Those three skills cover most of the debugging you will do with pipelines.

Steps Stored as a List of Tuples

A pipeline keeps its steps in a list. Each entry in that list is a tuple of the form (name, estimator). The name is a label you choose for the step, and the estimator is the object that does the work. The estimator for step i sits at index 1 of the tuple at position i, so pipeline.steps[0][1] is the first estimator, pipeline.steps[1][1] is the second, and so on. The name for step i sits at index 0 of that same tuple.

Which index gives the name of a step, and which index gives the estimator? Each position in the list holds one (name, estimator) tuple, so the estimator is always at index 1 of the tuple at that position.

Reading the layout carefully prevents a common confusion. The name and the estimator are both in the same tuple, so steps[0][0] gives the string name and steps[0][1] gives the object you can call fit, transform, or predict on. Calling methods on the tuple itself, rather than on index 1 of it, is an error.

Fitting Passes Data Forward

Fitting a pipeline has two parts. The first part goes over every step except the last one. Each of those steps is fitted and used to transform the data at the same time, which the source describes as running fit_transform. The transformed result then becomes the input to the next step. The second part fits the final step on whatever data it finally receives.

When pipeline.fit(X, y) runs, the first step fits and transforms the raw input, and its output becomes the input the last step is fitted on.

The important point is that the last step never sees the raw input when the pipeline is fitted. It is fitted on the output of the steps before it. This is why fitting a pipeline is not the same as fitting each estimator separately on the original data.

Tracing a Small Pipeline During Fitting

The sketch below is an illustrative version of the toy pipeline. It keeps the two steps in a list of (name, estimator) tuples and runs the fitting loop by hand, so you can see every hand-off. The first step adds one to each number. The last step labels each number as high when it is at least the cutoff of 6, and low otherwise.

python

class AddOne:
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return [x + 1 for x in X]

    def fit_transform(self, X, y=None):
        self.fit(X, y)
        return self.transform(X)


class Labeler:
    def __init__(self, cutoff):
        self.cutoff = cutoff

    def fit(self, X, y=None):
        self.n_seen_ = len(X)
        print('Labeler.fit received:', X)
        return self

    def predict(self, X):
        return ['high' if x >= self.cutoff else 'low' for x in X]


steps = [('add_one', AddOne()), ('label', Labeler(cutoff=6))]

X = [3, 5, 8]

# Fitting: every step except the last uses fit_transform
for name, est in steps[:-1]:
    X = est.fit_transform(X)

# The last step is fitted on the data it finally receives
steps[-1][1].fit(X)

class AddOne:
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return [x + 1 for x in X]

    def fit_transform(self, X, y=None):
        self.fit(X, y)
        return self.transform(X)


class Labeler:
    def __init__(self, cutoff):
        self.cutoff = cutoff

    def fit(self, X, y=None):
        self.n_seen_ = len(X)
        print('Labeler.fit received:', X)
        return self

    def predict(self, X):
        return ['high' if x >= self.cutoff else 'low' for x in X]


steps = [('add_one', AddOne()), ('label', Labeler(cutoff=6))]

X = [3, 5, 8]

# Fitting: every step except the last uses fit_transform
for name, est in steps[:-1]:
    X = est.fit_transform(X)

# The last step is fitted on the data it finally receives
steps[-1][1].fit(X)

Output (expected)

Labeler.fit received: [4, 6, 9]

Notes on the code

steps is a list of (name, estimator) tuples
steps[:-1] selects every step except the last
The loop reassigns X, so each step's output becomes the next input
The final fit receives [4, 6, 9], not the original [3, 5, 8]

Worked example

What the last step is fitted on

The raw input is X = [3, 5, 8]. The first step adds one to every number, and the last step is fitted on the result. What data does the last step receive during fitting?

1
Start with the raw input
X = [3, 5, 8] enters the pipeline. It is the input to the first step only.
2
Run fit_transform on the first step
AddOne fits, then transforms. Each number increases by one, giving [4, 6, 9].
3
Pass the output forward
The loop assigns the transformed list to X, so X is now [4, 6, 9].
4
Fit the last step
The labeler's fit runs on [4, 6, 9]. The labeler never sees [3, 5, 8].

Result

The last step is fitted on [4, 6, 9].

Prediction Uses Transform, Not Fit

What do you think happens?

When pipeline.predict([3, 5, 8]) runs, which numbers reach the labeling step's predict method?

The original numbers 3, 5, and 8
The numbers 4, 6, and 9
The labels low, high, and high

Reveal answer

Answer:

The numbers 4, 6, and 9

Prediction runs transform on every step except the last. The first step adds one, so the labeling step receives 4, 6, and 9. The pipeline did not hand the raw numbers to the labeling step.

Prediction follows a shorter path than fitting. Data passes through every step except the last one, and each of those steps only transforms the data. It does not refit anything. The earlier steps are called with transform, not fit_transform, and that is the key difference. The last step then calls predict on the data it has received.

When pipeline.predict(X) runs on new data, the first step only transforms the input, and the last step calls predict on the transformed values.

Output

pipeline.predict([3, 5, 8])  ->  ['low', 'high', 'high']

The labeler compared 4 with the cutoff 6 (low), 6 with 6 (high), and 9 with 6 (high).

Why Only the Last Step Needs fit

The pattern above assumes that the final step has a predict method, but the source notes that this is not required. Every step except the last is used through fit_transform during fitting and transform during prediction. Only the last step is fitted directly on the data it receives, so the last step is the only one that needs a fit method in the sense the pipeline uses.

Because the last step does not have to be a classifier, it does not have to have predict. A pipeline built from a scaler and PCA has no classifier at all. PCA has a transform method, so you can call transform on the whole pipeline and get the output of PCA applied to the data the earlier step has already processed. Pipelines are therefore not limited to classifiers, and they can be built from transformers only.

Mistakes When Reading Pipeline Output

Common mistake
Assuming the last step receives the raw input during fitting or prediction
What it looks like
Expecting the labeler to see 3, 5, and 8 when the first step has already added one.
Why it fails
Each step passes its output to the next, so the last step receives the transformed values.
Do this instead
Trace the data through each step in order and write down the values that reach the final step.
Common mistake
Using fit_transform on earlier steps during prediction
What it looks like
Refitting the first step on the new data inside predict.
Why it fails
Prediction only transforms the data in earlier steps. Refitting would change what the pipeline learned during fitting.
Do this instead
During prediction, earlier steps are called with transform. Only fitting calls fit_transform.
Common mistake
Calling predict on a pipeline whose last step is a transformer
Why it fails
A transformer such as PCA has no predict method, so the final call has nothing to run.
Do this instead
Use transform on the whole pipeline when its last step is a transformer.
Common mistake
Confusing the name in steps[i][0] with the estimator in steps[i][1]

Finding Where the Output Diverges

When the final output is not what you expected, you do not need to guess which step is at fault. Follow the data through the steps in order and compare each intermediate value with what you expected at that point. The first step where the values stop matching is where your model of the pipeline breaks down.

1
Write down the raw input you passed to the pipeline.
2
Predict the output of the first step by hand and compare it with the value the trace shows entering the second step.
3
Move down the steps in order, checking each output against your expectation.
4
Stop at the first step where the values differ. The fault is in that step or in the expectation you had for it.
5
If every intermediate value matches, compare the last step's predict or transform call with what you expected from it.

Practice Changing the Cutoff

MEDIUM

Keep the same pipeline with the first step adding one to every number. Change the labeler's cutoff from 6 to 9. Before running anything, predict the labels for pipeline.predict([3, 5, 8]). Then state which step's output is the first one you need to check to explain each label.

Hints

The labeling step receives the transformed values 4, 6, and 9, not 3, 5, and 8.
A value is labeled high when it is greater than or equal to the cutoff.
Compare each transformed value with 9 one at a time.

Your attempt (not saved)

Key Takeaways

A pipeline stores its steps as a list of (name, estimator) tuples, and the estimator for step i sits at index 1 of that tuple.
During fitting, every step except the last runs fit_transform, and each output becomes the next step's input. The last step is fitted on the data it finally receives.
During prediction, every step except the last only runs transform, and the last step calls predict on the transformed data.
Only the last step needs a fit method. A pipeline can end in a transformer such as PCA and be used through transform instead of predict.
To find a surprising output, follow the intermediate values step by step and stop at the first one that differs from your expectation.

Related concepts

More concepts starting with U