Creating Custom Estimators in scikit-learn

By this point you have seen a wide range of tools and algorithms that scikit-learn provides. They cover many common machine learning tasks, and for most projects you can assemble a workflow from them without writing anything new. Real projects, however, often have a structure, a rule, or a transformation that is specific to the problem at hand. When that happens, no built-in class may do exactly the work your data needs.

This article follows one practical question: when does a missing processing step push you toward writing your own estimator, and when can you avoid that? The answer depends on a single property of the step, namely whether it depends on the data it is given. Keep that property in mind as we trace what happens to a step as it moves through a workflow.

The simplest case is a processing step you can carry out on your data before it reaches a scikit-learn model or pipeline. You transform the data first, then pass the result to the estimator. Nothing about the model's training process has to know how the transformation was produced, so the rest of your code can treat the output as ordinary input.

Because the transformation is a fixed operation that happens outside the modeling workflow, writing a custom estimator is usually not required for a step like this. The key idea is that the transformation's output does not depend on what the rest of the dataset looks like. Each row is processed by the same fixed rule, so the step can be run once over everything you have.

The source identifies this as the difficult case. Here the processing you apply is determined by the data itself, and that changes the picture. If you want to use grid search or cross-validation, the simple approach of preprocessing everything up front becomes unreliable, and the work gets trickier.

To make this concrete, consider an illustrative step that shifts each value so the data is centered around zero. The amount it subtracts is the average of the values it was given. This is a teaching example created for this article, not one taken from the source, and it is chosen only to show why the step depends on the data.

Here $n$ is the number of values the step was given, and each $x_i$ is one of those values. The centering step learns $\bar{x}$ from the data and then subtracts it from every value. Notice that the offset is not a fixed number you could write down in advance. It exists only because of the particular values the step saw.

Grid search and cross-validation both repeatedly split data and evaluate a model on different portions of it. Each time, one portion is used to train and another is held back to evaluate. A data-dependent step has to be fitted again for each of these portions, because the values it learns belong to the portion it was fitted on.

The diagram below follows one fold. The step learns its offset only from the training portion of that fold. The held-out portion is then transformed using that offset, and the held-out rows never contribute to the value the step learned.

The order matters. The offset must be learned before the held-out rows are transformed, and it must be learned only from the training rows of that fold. If the held-out rows were allowed to influence the offset, the evaluation would no longer measure how the step behaves on data it has not seen.

Grid search adds another layer. It tries many parameter settings, and for each setting it runs a full cross-validation. So a single data-dependent step may have to learn its state many times over: once for each parameter setting, and once for each fold within that setting. The learned state from one run must not carry over into another.

Reading the flow from the top, the parameter setting is never the source of the learned state. The state always comes from the training rows of a particular fold, and it is recomputed for every fold of every candidate. This is the reason the source says that data-dependent preprocessing makes grid search and cross-validation harder to combine with a preprocess-first workflow.

The two kinds of step can now be set side by side. What separates them is whether the output for a given row would change if the rest of the dataset were different.

The fixed transformation can be carried out once on the whole dataset, and the rest of the workflow treats its output as ordinary input. The data-dependent step cannot be settled in advance, because its output depends on the statistics it learns from whichever rows it is given. That is the situation where, according to the source, a custom estimator is needed in order to work with scikit-learn's tools.

A custom estimator is the natural way to express a data-dependent step in a form that scikit-learn's tools can work with. The reason is that the step then carries its own learned state with it. In the centering example, the offset of 7 from the earlier worked example would be stored as part of the step itself, rather than being computed by hand outside the workflow and passed along separately.

Because the state travels with the step, the library's tools can learn it from the training portion of each fold and reuse it for the held-out portion, without you rewriting the procedure for every split. The source does not go into implementation details at this point, so the purpose to remember is the condition itself: a custom estimator is needed when a step's behavior is data-dependent and the workflow must also handle grid search or cross-validation.