Classification and Regression

The type of task is determined by the output: if the possible outcomes are continuous, the task is regression; if they are discrete categories, it is classification.

Programming

Starting From a Working Program

Before any theory, read a short program that trains a classifier and prints one number. The program is the starting point for this article. Read it line by line, predict what the final line will print, and then learn where that number comes from. The snippet assumes that train_test_split, KNeighborsClassifier, and the iris dataset (iris_dataset) are already available in your session.

python

X_train, X_test, y_train, y_test = train_test_split(iris_dataset['data'], iris_dataset['target'], random_state=0)
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))

X_train, X_test, y_train, y_test = train_test_split(iris_dataset['data'], iris_dataset['target'], random_state=0)
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))

Notes on the code

Split the dataset into training examples and test examples, kept apart from each other
Create a k-Nearest Neighbors classifier that considers one neighbor
Build the model using the training inputs and the training outputs
Measure accuracy on the test inputs and test labels and print it

What do you think happens?

The last line passes two arguments to score. Which pair of variables should it receive so that the printed value reflects how the model handles data it never learned from?

X_train and y_train
X_test and y_test
X_train and y_test

Reveal answer

Answer:

X_test and y_test

The test set holds examples kept apart from the training step, so scoring on it shows how the model handles data it did not learn from. The source example that uses this pattern with KNeighborsClassifier(n_neighbors=1) reports a test set score of 0.97.

Tracing the fit and score Calls

Follow the program one call at a time. The first meaningful call is fit. It receives the training data and the training outputs, and it builds the model from those examples. Supervised learning works this way whenever you want to predict an outcome from an input and you have examples of input and output pairs. These pairs make up the training set.

The second call is score. It computes accuracy on the test set data and the test set labels. In the source example this call prints a test set score of 0.97, which means the model was correct 97% of the time on the test set. The score is only as meaningful as the data it is given, which is why the choice of arguments matters so much.

In what order do fit, then score, get called, and what data does each one receive and produce?

The fit and score methods are part of a common interface to supervised models in scikit-learn. Once you know this pattern, the same core workflow applies to many different machine learning tasks, even when the underlying algorithm changes. The k-nearest neighbors classifier is one model that follows this pattern.

How a Nearest Neighbor Vote Produces a Class

The word neighbors in the name refers to the training points that are closest to a new point. The k-nearest neighbors classification algorithm predicts a new data point by considering its closest neighbor or neighbors in the training set. The KNeighborsClassifier class implements it. In the program above, the parameter n_neighbors=1 tells the classifier to consider one neighbor. The source does not describe other values of this parameter, so this article does not make claims about them.

Which training points are closest to a new point, and how do their labels determine the predicted class?

Where the Printed Number Can Go Wrong

Now trace a change. Suppose the last line is edited so that it passes the training data instead of the test data. The fit call is unchanged, so the model is built exactly as before. The divergence happens entirely in the score call. The accuracy is now measured on data the model was built from, so the printed value is not expected to match 0.97. The source does not give a value for that case, so the new number cannot be predicted from the source alone. What can be stated with confidence is that the evaluated data changed.

Common mistake
Assuming the score is a property of the model alone
Why it fails
The score method computes accuracy on the data it is given, so the same fitted model can produce different numbers for different pairs of variables.
Do this instead
Always check which inputs and labels are passed to score, and confirm they are the test set when you want a test set score.
Common mistake
Passing the test data to fit
Why it fails
fit receives the training data and training outputs and builds the model from them. Test examples are meant to stay apart from the training step.
Do this instead
Call fit with the training pair and call score with the test pair.

Continuous Outputs Versus Discrete Categories

The type of task is determined by the output. If the possible outcomes are continuous, the task is regression. If the possible outcomes are discrete categories, the task is classification. Both classification and regression are types of supervised learning tasks. The source uses classification for the iris example, which classifies iris flowers into several species using physical measurements. This section focuses on classification and does not describe regression in detail.

The central question is whether the possible outcomes form a continuous spread. Instead of asking what the outputs look like, ask how close two outcomes are to each other, and whether being a little off from the right answer matters. If the possible answers sit along a scale where neighboring values blend into one another, the task is a regression task. If the answers are discrete and no answer lies meaningfully between two others, the task is a classification task.

Continuity means small differences between outcomes are not significant. Predicting a person's annual income is a regression task. Income is an amount of money, and amounts form a continuous scale. Whether someone earns 40,000 or 40,001 dollars a year makes no tangible difference in practice, so a prediction of 39,999 when the correct value is 40,000 is usually an acceptable miss.

In the real world

Recognizing the language of a website is a classification task. A site is written in one language or another, and there is no matter of degree between them. There is no language halfway between English and French. Predicting French for an English site is simply wrong, not slightly wrong, so no notion of closeness helps rank one mistake as smaller than another.

How do discrete category outputs differ from continuous numeric outputs, and which kind of output makes a problem a classification task versus a regression task?

Labeling a New Problem by Its Outcomes

To label a new problem, examine how its possible outcomes relate to one another. Do not decide based on how the data is stored. Two prediction problems can look almost identical on the surface, with the same kind of input and a similar sounding goal, and still call for different treatment because their outputs behave differently. Ask two questions: are the outcomes on a continuous scale where nearby values are interchangeable, and does being slightly off matter? If the answers are continuous and small errors are acceptable, the task is regression. If the outcomes are discrete categories with nothing meaningful between them, the task is classification.

Worked example

Labeling a house price prediction

A system predicts the sale price of a house in dollars. Is this a classification task or a regression task?

1
List the possible outcomes
A sale price is an amount of money, so the possible outcomes span a range of amounts rather than a fixed set of named categories.
2
Check neighboring outcomes
A price one dollar away from the correct price is essentially as good as the correct price for practical purposes, so neighboring values blend into one another.
3
Apply the continuity test
Because outcomes form a continuous scale and small misses are acceptable, the output is continuous.
4
Conclude the label
The task is regression, using the same reasoning as the annual income example.

Result

Regression, because the possible outcomes are continuous and nearby values are interchangeable.

EASY

A system reads a photo of a fruit and decides whether it is an apple, a pear, or a banana. Is this classification or regression? Justify your label by saying whether there is any meaningful outcome between apple and pear.

Hints

Ask whether the answers form a scale where neighboring values blend together.
Consider whether predicting pear for an apple is slightly wrong or simply wrong.

Your attempt (not saved)

Key Takeaways

The type of task is determined by its output: continuous possible outcomes make a regression task, and discrete categories make a classification task.
The fit method builds a model from training inputs and training outputs, and the score method computes accuracy on the test inputs and test labels it is given.
The k-nearest neighbors classifier predicts a new point using its closest training neighbors, and KNeighborsClassifier(n_neighbors=1) reported a test set score of 0.97 in the source example.
To label a problem, examine how the possible outcomes relate to one another, including whether a meaningful outcome lies between two others, rather than how the data is stored.

Related concepts

More concepts starting with C