Building a Simple Pipeline in Python

A pipeline stores its steps as a list of (name, estimator) tuples, and the estimator for step i sits at index 1 of that tuple.

Programming

Why Watching the Data Matters

A pipeline is a chain of processing steps. Each step takes data in, does something to it, and hands the result to the next step. When a pipeline gives a surprising answer, the cause is almost always hidden in the hand-offs between steps. The fastest way to build a correct mental model of a pipeline is to watch data move through it, one step at a time.

This article starts with a small, runnable pipeline with two steps. The first step is a transformer that adds one to every number. The last step labels each number as low or high by comparing it with a cutoff. Read the code first and notice how the steps are stored and how the two method calls are made. Then trace the values by hand, write down what you expect each step to receive, and compare your guess with what actually happens.

How the Steps Are Stored

A pipeline keeps its steps as a list of (name, estimator) tuples. Each tuple pairs a name you chose with the object that does the work. The estimator for step i sits at index 1 of that tuple, so pipeline.steps[0][1] is the first estimator, pipeline.steps[1][1] is the second estimator, and so on. The name at index 0 is only a label, which is useful for looking steps up by name and for reading the pipeline's structure.

How is the estimator for step i found inside the pipeline's list of (name, estimator) tuples, and which tuple position holds it?

python

class AddOne:
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return [x + 1 for x in X]

    def fit_transform(self, X, y=None):
        return self.fit(X, y).transform(X)


class LowHighLabeler:
    def __init__(self, cutoff):
        self.cutoff = cutoff

    def fit(self, X, y=None):
        return self

    def predict(self, X):
        return ['low' if x < self.cutoff else 'high' for x in X]


pipe = SimplePipeline([
    ('add_one', AddOne()),
    ('labeler', LowHighLabeler(cutoff=4)),
])

print(pipe.steps[0][1])
print(pipe.steps[1][1])

class AddOne:
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return [x + 1 for x in X]

    def fit_transform(self, X, y=None):
        return self.fit(X, y).transform(X)


class LowHighLabeler:
    def __init__(self, cutoff):
        self.cutoff = cutoff

    def fit(self, X, y=None):
        return self

    def predict(self, X):
        return ['low' if x < self.cutoff else 'high' for x in X]


pipe = SimplePipeline([
    ('add_one', AddOne()),
    ('labeler', LowHighLabeler(cutoff=4)),
])

print(pipe.steps[0][1])
print(pipe.steps[1][1])

Notes on the code

AddOne has no learned parameters, so its fit method only returns self
LowHighLabeler stores its cutoff when it is created
The pipeline's steps list holds (name, estimator) tuples
pipe.steps[0][1] reaches the AddOne estimator through index 1 of the first tuple

Tracing the Fitting Pass

Fitting a pipeline has two parts. The first part goes over every step except the last one. Each of those steps is fitted and used to transform the data, and the transformed result becomes the input to the next step. The second part fits the final step on whatever data it finally receives. The loop over the earlier steps is the part that moves data forward.

When pipeline.fit(X, y) runs, what does each step receive as input, what does it return, and how does that output become the next step's input?

Worked example

Tracing pipe.fit([3, 5, 8], None) step by step

Trace the values that each step receives and returns during fitting, with the labeler's cutoff set to 4.

1
Start
The pipeline receives X = [3, 5, 8]. The first step is not yet fitted, so the loop begins with the first tuple.
2
Step 0 (AddOne)
The loop calls fit_transform on the AddOne estimator with [3, 5, 8]. Each number is increased by one, so the output is [4, 6, 9].
3
Hand-off
The transformed list [4, 6, 9] becomes the input for the next step. The raw numbers 3, 5, and 8 never reach the labeler during fitting.
4
Last step (LowHighLabeler)
The final step is not transformed by the loop. It is fitted on the data it receives, [4, 6, 9]. Its fit method only returns self in this example.

Result

Fitting leaves the labeler fitted on [4, 6, 9]. The earlier step was used to produce that data through fit_transform.

What do you think happens?

After fitting, you call pipe.predict([3, 5, 8]) with cutoff=4. Which list does the labeler's predict method receive?

[3, 5, 8], the raw numbers
[4, 6, 9], the output of the AddOne step
[2, 4, 7], the numbers after subtracting one

Reveal answer

Answer:

[4, 6, 9], the output of the AddOne step

Prediction runs transform on every step except the last. AddOne therefore turns [3, 5, 8] into [4, 6, 9] before the labeler's predict is called.

Tracing the Prediction Pass

Prediction follows a shorter path than fitting. Data passes through every step except the last one, and each of those steps only transforms the data. It does not refit anything. The last step then calls predict on the data it has received. The key difference from fitting is that the earlier steps are called with transform, not fit_transform.

When pipeline.predict(X_new) runs, which method is called on each intermediate step versus the final step, and in what order?

python

pipe.fit([3, 5, 8])
print(pipe.predict([3, 5, 8]))

pipe.fit([3, 5, 8])
print(pipe.predict([3, 5, 8]))

Output (expected)

['high', 'high', 'high']

Notes on the code

fit runs fit_transform on AddOne and fits the labeler on [4, 6, 9]
predict runs transform on AddOne, giving [4, 6, 9] again
The labeler then calls predict on [4, 6, 9] with cutoff 4

Why Only the Last Step Needs fit

The pattern above assumes that the final step has a predict method, but this is not required. The last step only needs a fit method. Earlier steps need a way to produce transformed data, which is why they are used through transform or fit_transform. The final step is the only one whose output the pipeline does not feed forward, so it may be a classifier with predict, or it may be a transformer with transform.

A pipeline built from a scaler and PCA has no classifier at all. PCA has a transform method, so you can call transform on the whole pipeline. You get the output of PCA applied to the data that the earlier step has already processed. Pipelines are therefore not limited to classifiers, and they can be built from transformers only.

Position in pipeline	Method called during fit	Method called during predict	Why
Every step except the last	fit_transform	transform	The output must be passed forward as input to the next step
Last step	fit	predict, or transform if the last step is a transformer	The pipeline does not feed its output to another step

How each position in the pipeline is used

Worked example

A transformer-only pipeline

Suppose a pipeline has a scaler followed by PCA, with no classifier. What happens when you call transform on the whole pipeline?

1
Fitting
The scaler is fitted with fit_transform, and PCA is fitted on the scaled data. PCA is the last step, so it needs only fit.
2
Transform call
The scaler receives the new data and runs transform. Its output is passed to PCA.
3
Final output
PCA runs transform on the scaled data and returns the reduced representation. No predict method is needed anywhere in the pipeline.

Result

The pipeline is used through transform, and its output is the PCA output on the data the scaler has already processed.

Finding Where the Output Diverges

When a pipeline's output is not what you expected, the question to ask is at which boundary the data first stopped matching your mental model. You can answer it by printing the data at each hand-off. Because each step's output becomes the next step's input, the first boundary where the values differ from your expectation is the place to look.

python

def trace_predict(pipe, X):
    data = X
    for name, est in pipe.steps[:-1]:
        data = est.transform(data)
        print(name, '->', data)
    final_name, final_est = pipe.steps[-1]
    result = final_est.predict(data)
    print(final_name, 'predicts', result)
    return result

trace_predict(pipe, [3, 5, 8])

def trace_predict(pipe, X):
    data = X
    for name, est in pipe.steps[:-1]:
        data = est.transform(data)
        print(name, '->', data)
    final_name, final_est = pipe.steps[-1]
    result = final_est.predict(data)
    print(final_name, 'predicts', result)
    return result

trace_predict(pipe, [3, 5, 8])

Output (expected)

add_one -> [4, 6, 9]
labeler predicts ['high', 'high', 'high']

Notes on the code

The loop mirrors the prediction path, so each boundary is printed
The first printed line where the values differ from your expectation shows the divergent step
Compare the printed labeler input with the raw numbers you started with

Common mistake
Assuming the last step receives the raw input
What it looks like
Expecting the labeler to label 3 as low with cutoff 4, because 3 is less than 4
Why it fails
Prediction passes the data through every earlier step first. The labeler received [4, 6, 9], so 3 became 4 before the comparison.
Do this instead
Trace the data at each boundary and use the value the last step actually receives.
Common mistake
Calling fit_transform on earlier steps during prediction
What it looks like
Writing predict so that AddOne is refitted on the new data before transforming it
Why it fails
Prediction does not refit anything. Earlier steps only transform, so learned state from fitting is reused.
Do this instead
Use transform on every step except the last during prediction.
Common mistake
Expecting every pipeline to end in a predict method
What it looks like
Calling predict on a pipeline that ends in PCA and getting an error
Why it fails
The last step only needs a fit method. A final transformer is used through transform.
Do this instead
Check which method the last step provides, and call transform on a transformer-ending pipeline.
Common mistake
Looking up the estimator at index 0 of a steps tuple instead of index 1

Practice Tracing a New Pipeline

MEDIUM

Suppose a pipeline has two steps. The first step, DoubleIt, multiplies every number by two during transform and has no learned state. The last step, a labeler with cutoff 5, labels numbers as low when they are less than 5 and high otherwise. Before running anything, write down the list the labeler receives during predict(X_new) when X_new is [1, 2, 4], and then predict the final output. Then check your answer by printing the data at each boundary.

Hints

During predict, DoubleIt is called with transform, so the labeler receives the doubled values
Double each input number first: 1 becomes 2, 2 becomes 4, and 4 becomes 8
Compare each doubled value with the cutoff of 5 to find the label

Your attempt (not saved)

Pipeline Behavior in Summary

1
A pipeline stores its steps as a list of (name, estimator) tuples, and the estimator for step i is at index 1 of that tuple.
2
During fit, every step except the last runs fit_transform and passes its output forward, and the last step is fitted on the data it finally receives.
3
During predict, every step except the last runs transform, and the last step calls predict on the transformed data.
4
Only the last step needs a fit method. A pipeline can end in a transformer such as PCA and be used through transform instead of predict.
5
When the output is unexpected, print the data at each boundary to find the first step where the values differ from your expectation.

At a glance

A pipeline stores its steps as a list of (name, estimator) tuples, and the estimator for step i is at index 1 of that tuple.
During fit, every step except the last runs fit_transform and passes its output forward, and the last step is fitted on the data it finally receives.
During predict, every step except the last runs transform, and the last step calls predict on the transformed data.
Only the last step needs a fit method, so a pipeline can end in a transformer and be used through transform.
To find where output diverges, print the data at each boundary and identify the first step where the values differ from your expectation.

Related concepts

Principal Component Analysis in Practice

More concepts starting with B

Why Watching the Data Matters

How the Steps Are Stored

Tracing the Fitting Pass

Tracing the Prediction Pass

Why Only the Last Step Needs fit

Finding Where the Output Diverges

Practice Tracing a New Pipeline

Pipeline Behavior in Summary

At a glance

Related Topics

Related concepts