Fitting and Transforming Data in a Pipeline

The named_steps attribute of a pipeline is a dictionary from step names to estimators, and it is the easiest way to reach individual steps.

Programming

Why Look Inside a Fitted Pipeline

A pipeline bundles several processing steps into one object, and you often fit the whole pipeline at once. Fitting the pipeline is usually the only call you need to make on the data. Once it is fitted, though, you may want to look inside it. You might want the coefficients of a linear model that sits at the end of the pipeline, or the components extracted by a PCA step in the middle.

This article shows how to reach into a pipeline and retrieve the learned attributes of one of its steps. Before reading the details, try to predict what you would expect to find when you ask a pipeline for the components of its PCA step. Would the answer come from the pipeline as a whole, or from one specific step inside it? Keep that question in mind as you read.

Tracing Data Through the Steps

A pipeline passes data through its steps in order. When you call fit_transform on a pipeline, the data enters the first step, that step's output becomes the input to the second step, and so on. The attribute you read later belongs to one of those steps at a particular position, not to the pipeline as a whole.

When pipe.fit_transform(X) runs, the data passes through each step in order, so the components are learned from the output of the first step.

Notice what this means for the PCA step. Its components are computed from the data after the scaler has already processed it. The components therefore describe the scaled features, which is why reading them through the pipeline's position in the chain matters. The next section explains how you select that step by name.

named_steps as a Dictionary

Definition

The named_steps attribute of a pipeline is a dictionary from step names to estimators, and it is the easiest way to reach individual steps. Each key is the name you chose when you built the pipeline, and each value is the estimator object stored under that name.

How does the string name 'pca' map to the specific PCA object stored inside the pipeline? Looking up the key pca returns the PCA estimator, and only that estimator carries components_.

Because named_steps is an ordinary dictionary lookup, you pass the step name in square brackets, as in pipe.named_steps['pca']. The result is the estimator object for that step. That means you can read any attribute the estimator has learned, not only the ones the pipeline itself exposes.

python

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=2)),
])

pipe.fit(X)

pca_step = pipe.named_steps['pca']
print(type(pca_step).__name__)

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=2)),
])

pipe.fit(X)

pca_step = pipe.named_steps['pca']
print(type(pca_step).__name__)

Output (expected)

PCA

Notes on the code

The names 'scaler' and 'pca' are the keys we chose when building the pipeline.
named_steps['pca'] returns the PCA estimator itself, not a copy of its data.
The pipeline was fitted before the lookup, so the PCA step has learned its components.

Reading components_ Through the Step

The source pattern for this idea fits a pipeline that was defined earlier to the cancer dataset, then pulls out the first two principal components from the step named pca. The full line is pipe.named_steps['pca'].components_. It looks like one expression, but it is really three operations joined together. Python evaluates them from left to right, so each piece takes the result of the piece before it.

Which object owns components_, and how does the attribute chain move from the pipeline to the step to the attribute? The pipeline gives you the dictionary, the dictionary gives you the PCA step, and the PCA step owns the components_ array.

Worked example

Following the chain pipe.named_steps['pca'].components_

Starting from a fitted pipeline called pipe, which object's data does pipe.named_steps['pca'].components_ return, and why?

1
Step 1: pipe.named_steps
The pipeline object exposes a dictionary called named_steps. This is the only piece of the chain that belongs to the pipeline itself.
2
Step 2: ['pca']
The string key 'pca' is looked up in that dictionary. The value stored under that key is the PCA estimator object, which is the step at that position.
3
Step 3: .components_
The attribute components_ is read from the PCA estimator. Only the PCA object has this attribute, so the array returned is the PCA step's own learned data.

Result

The chain returns the components learned by the PCA step, not any data the pipeline computes for itself. The pipeline is used only to reach the step.

What do you think happens?

After pipe.fit(X) on the breast cancer data, with 30 input features and PCA set to 2 components, what is the shape of pipe.named_steps['pca'].components_?

Reveal answer

Answer:

(2, 30), meaning two components, each with one weight per original feature.

PCA with n_components=2 learns two principal components. Each component is a direction in the original 30-feature space, so the components array has one row per component and one column per feature.

Watching components_ Appear After Fitting

Learned attributes such as components_ exist only after fitting. Before fit runs, the PCA step has been configured with its settings, such as the number of components it should keep, but it has not yet looked at any data. Once fitting runs, the step learns its components and stores them on itself. That is the moment the attribute becomes available to read.

Before

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=2)),
])
pipe.named_steps['pca'].components_

The pipeline is built but not fitted. The PCA step holds its settings but has learned no components, so components_ is not yet available.

After

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=2)),
])
pipe.fit(X)
pipe.named_steps['pca'].components_

The pipeline has been fitted to X. The PCA step has learned its components and stores them, so reading components_ through the step returns a two-row array.

The same ordering rule applies to every step. The whole pipeline is fitted first, which lets each step learn from the data that reaches it, and only then do you read the learned results. Reading before fitting is a timing mistake, not a naming mistake, and the next section covers both kinds.

Common Errors with Step Names and Attributes

Most problems with this technique come from three sources: a step name that does not match, an attribute read before the pipeline was fitted, and an attempt to select a step by its position instead of its name. Each one fails in a different way, so it helps to recognize the difference.

Common mistake
Misspelling the step name or changing its case
What it looks like
pipe.named_steps['PCA'] when the step was built as 'pca'
Why it fails
Step names are dictionary keys, so they must match the names used when the pipeline was built, including case. A key that does not exist cannot be found.
Do this instead
Use the exact name from the pipeline definition, such as pipe.named_steps['pca']. Print pipe.named_steps.keys() if you are unsure of the names.
Common mistake
Reading a learned attribute before fitting
What it looks like
pipe.named_steps['pca'].components_ directly after building the pipeline
Why it fails
Learned attributes such as components_ are created during fitting. Before fitting, the attribute has not been set on the step.
Do this instead
Fit the pipeline first, for example with pipe.fit(X), and then read the attribute through the step.
Common mistake
Reading the attribute from the pipeline instead of the step
What it looks like
```
pipe.components_
```
Why it fails
The attribute belongs to the estimator at the step, not to the pipeline object that holds the steps.
Do this instead
Go through named_steps to reach the step first: pipe.named_steps['pca'].components_.
Common mistake
Selecting a step by its position instead of its name
What it looks like
```
pipe.named_steps[1]
```
Why it fails
named_steps is a dictionary keyed by step names, so an integer position is not a key that it holds. The position of the step is not what you look up.
Do this instead
Use the step's name, such as pipe.named_steps['pca']. If you need the position, find the name in the keys first.

Error type	Typical cause	What it looks like	Fix
Misspelled step name	The key does not match the name used when building the pipeline, including case	A failed dictionary lookup on named_steps, usually reported as a missing key	Use the exact name, and check the keys of named_steps
Attribute before fitting	The learned attribute is created only during fitting	The attribute is missing on the step when you read it	Fit the pipeline first, then read through the step
Step accessed by position	named_steps is keyed by names, not by integer positions	A failed lookup when an integer is used as the key	Look up the step by its name

Three different ways to fail when accessing steps and their attributes

Practice Reaching a Step's Components

EASY

You build a pipeline with a step named 'scale' that is a StandardScaler, followed by a step named 'reduce' that is a PCA with 3 components. You fit it to a feature matrix X. Write the line that returns the components learned by the PCA step. Then name one error you would avoid by checking the step name.

Hints

Start from pipe.named_steps and select the step by its exact name, 'reduce'.
The attribute components_ belongs to the PCA estimator, so it comes after the step lookup.
A name that differs by case from 'reduce' will not be found, so copy the name exactly.

Your attempt (not saved)

Summary of Reaching Into Pipelines

1
The named_steps attribute is a dictionary from step names to estimators, and it is the easiest way to reach individual steps.
2
pipe.named_steps['pca'].components_ is three operations read left to right: get the dictionary, look up the step, then read the learned attribute from that step.
3
Learned attributes such as components_ exist only after the pipeline has been fitted, and they belong to the step's estimator rather than to the pipeline object.
4
Step names must match exactly, including case, and steps are found by name rather than by position.

At a glance

named_steps is a dictionary from step names to estimators, and it gives the easiest access to individual steps.
The chain pipe.named_steps['pca'].components_ returns the PCA step's own learned data, because the lookup reaches the step before the attribute is read.
Learned attributes exist only after fitting, so fit the pipeline before reading components_ through the step.
Errors come from mismatched step names, reading before fitting, and selecting a step by position instead of by name.

Related concepts

More concepts starting with F