Estimator Classes in scikit-learn
Every machine learning model in scikit-learn is implemented in its own class, called an Estimator class.
A Classifier in Two Lines
This article starts with a small piece of code rather than a definition. The algorithm you will build is the k-nearest neighbors classifier, a classification method that assigns a category label to a data point. It is easy to understand, which is why it is a common first model to study. Most of what you need to know about it is visible in just a few lines of scikit-learn code, and only two of those lines do the real work: the line that creates the object and the line that calls fit.
- The import line brings in the class from the neighbors module
- The class name tells Python which algorithm to build, and
n_neighbors=1 sets the number of neighbors - fit passes the training features and labels to the classifier
- X_train and
y_trainare assumed to already exist as NumPy arrays; how they were prepared is outside this article
What do you think happens?
After knn.fit(X_train, y_train) runs, what does the call return?
Reveal answer
The classifier object itself, modified in place.
fit passes the training data and labels, stores the training set, and returns the classifier object. It does not return predictions or a score. This matters later when a variable that holds the result of fit is mistaken for a list of labels.
How a Neighbor Decides the Label
The core idea is simple. To make a prediction for a new data point, the algorithm finds the point in the training set that is closest to it, and then assigns that training point's label to the new point. The training set is the model. Building the model consists only of storing the training set, so the actual work happens when a prediction is requested.
A k-nearest neighbors classifier stores the training set, and it labels a new point by finding the closest training point and copying that point's label.
The diagram shows the one-neighbor version of the rule, which is the version this article uses first. Everything the classifier knows about the data lives in the stored training set. When more than one neighbor is consulted, the prediction uses a majority class among those neighbors instead of copying a single label. The step that changes with k is therefore the step that combines neighbor labels.
Where the Model Lives Before and After fit
Every machine learning model in scikit-learn is implemented in its own class, called an Estimator class. The k-nearest neighbors algorithm lives in the KNeighborsClassifier class, which is in the neighbors module. Before you can use the model, you must instantiate the class into an object. Instantiation is the step where parameters are set, such as n_neighbors. Assigning the result to a name like knn gives you a handle on that object for fitting and for later steps.
Read the instantiation line as two things happening together. The class name chooses the algorithm, and the keyword argument sets a parameter. After instantiation the object exists and knows its parameters, but it has not yet seen any data. Calling fit passes the training data and labels, stores the training set inside the object, and returns the classifier object itself, modified in place. The object changes state; fit does not hand back a prediction.
Creating and Fitting the Classifier Step by Step
Building a 1-neighbor classifier
Create a k-NN classifier that uses one neighbor and fit it to a training set that already exists as X_train and y_train.
- 1Import the class
The class KNeighborsClassifier is found in the neighbors module, so the import reads from sklearn.neighbors.
- 2Instantiate with a parameter
Calling the class with
n_neighbors=1 builds the object and sets the number of neighbors to one. - 3Store the object in a name
Assigning the result to knn gives a handle on the object for fitting and later steps.
- 4Fit to training data
knn.fit(X_train, y_train)stores the training features and labels in the object. The return value is the classifier itself.
knn is a fitted KNeighborsClassifier that holds the training set and is ready to label new points.
What the Number k Changes
The parameter k, written n_neighbors in scikit-learn, sets how many neighbors are consulted when a label is assigned. It is a parameter you choose, and changing it changes which training points influence the answer. With k equal to 1, a new point takes the label of its single closest training point. With more than one neighbor, the prediction uses a majority class among the consulted neighbors. The source introduces the single-neighbor case first and notes that more detail on choosing k comes later in the book.
| Setting | Neighbors consulted | How the label is produced |
|---|---|---|
| k = 1 (n_neighbors=1) | The single closest training point | Copy that point's label |
| k greater than 1 | Several closest training points | Take the majority class among those neighbors |
How the value of k changes the prediction rule
Tracing an Unexpected Result Back to Its Source
When a k-NN workflow produces a surprising label, the cause usually sits at one of a few stages along the path from data to prediction. Tracing that path stage by stage is more productive than rereading the classifier's code, because the classifier itself only does two things: it stores the training set during fit and it searches that stored set when new data arrives.
Two of these stages deserve particular attention. The training data passed to fit defines the entire model, so a mismatch in what was stored there will shape every later answer. The variable that holds the result of fit is also a common source of confusion. Because fit returns the classifier and not a prediction, code that expects labels from that variable will show a result that looks wrong even though fit ran without error.
Mistakes Beginners Make with fit
Common mistake Expecting fit to return predictions
What it looks likelabels = knn.fit(X_train, y_train) followed by using labels as if it were an array of predicted categoriesWhy it failsfit returns the classifier object itself, modified in place, not a list of predictions or a score.
Do this insteadKeep the object in a name such as knn, and use it for later steps that produce predictions.
Common mistake Assuming the work happens during fit
Why it failsThe model is the stored training set. fit stores it, and the neighbor search happens when a new point is labeled.
Do this insteadThink of fit as preparing the stored data and the prediction step as where the rule is actually applied.
Common mistake Leaving n_neighbors out and assuming a default meaning
Why it failsk is the parameter that controls how many neighbors influence the answer, and the source frames it as a parameter you set.
Do this insteadSet n_neighbors explicitly at instantiation so the number of consulted neighbors is a deliberate choice.
Predicting How a Change to k Alters the Result
Suppose you create two classifiers from the same X_train and y_train: one with n_neighbors=1 and one with n_neighbors=5. For a new point whose single closest training point has a label different from the majority label of its five closest training points, which classifier would give the new point the closest-point label, and which would give it the majority label?
Hints
- Recall which setting copies one neighbor's label and which takes a majority among several.
- Only the number of consulted neighbors differs between the two classifiers; the training data is the same.
Key Points to Carry Forward
- Every scikit-learn model is an Estimator class; k-NN lives in KNeighborsClassifier in the neighbors module.
- Instantiation builds the object and sets parameters such as
n_neighbors; fit stores the training set and returns the classifier itself. - A k-NN classifier labels a new point by finding the closest training point and copying its label when k is 1, or by taking a majority class among k neighbors when k is greater than 1.
- When a result looks wrong, check the training data passed to fit, the new data passed to predict, and whether the variable holding the output is the classifier rather than predictions.
Key Takeaways
- All scikit-learn machine learning models are Estimator classes, and k-NN is implemented in KNeighborsClassifier in the neighbors module.
- Instantiation creates the object and sets parameters such as
n_neighbors; fit stores the training set and returns the classifier object, not predictions. - A k-NN classifier labels a new point by finding its closest training point and copying that label, or by taking a majority class when more than one neighbor is consulted.
- Unexpected results most often trace back to the data passed to fit, the new data passed to predict, or a variable that was assumed to hold predictions.