Binary Classification Basics

In the binary classification case, decision_function returns an array whose shape is (n_samples,).

Programming

Asking a Fitted Model for Scores

A fitted classifier offers more than a predict method. It also offers decision_function, which returns a number for every sample you pass in. Predict gives you a label for each sample. Decision_function gives you a raw score that describes which way the model leans for that sample. This article traces what that score looks like, how its sign relates to the predicted label, and why the raw numbers cannot be read as probabilities.

What do you think happens?

You pass a test set of 25 points, each with 2 features, to decision_function on a binary classifier. What shape should the output have?

(25, 2), the same as the input
(25,), one number per sample
(2,), one number per feature
(25, 2, 2), one value per feature pair

Reveal answer

Answer:

(25,), one number per sample

In the binary case, decision_function returns an array of shape (n_samples,). The test input has shape (25, 2), meaning 25 samples with 2 features each, and the output has shape (25,). The feature dimension disappears because each sample's features are collapsed into a single score.

Before you read on, compare your guess with the answer above. If your guess was (25,), you already have the central idea: asking for one score per sample means the output has one entry for each sample and no entry for the features.

Tracing the Shape from Samples to Scores

In the binary classification case, decision_function returns an array whose shape is (n_samples,). Each entry is a single floating-point number describing one sample. The input has a second dimension, the features, and the output does not. Each sample arrives with several feature values, and the classifier reduces those values to one score for that sample.

How each sample's 2 input features turn into exactly one decision_function number, and why the feature dimension disappears from the output shape.

Worked example

Reading the shape of a 25-sample test set

The test input has shape (25, 2). What is the shape of the decision_function output, and what does each entry represent?

1
Identify the sample count
The first dimension of the input is 25, so there are 25 samples.
2
Identify the feature count
The second dimension is 2, so each sample has 2 features.
3
Apply the binary rule
For binary classification, decision_function returns one floating-point number per sample, so the output is shaped (n_samples,).
4
Write the output shape
The output shape is (25,). Each of the 25 entries is one score describing one sample, and the 2-feature axis is gone.

Result

The output has shape (25,), with one floating-point score for each of the 25 samples.

Each score measures how strongly the model believes a point belongs to the positive class, which in this example is class 1. A large positive score means the model is confident the point belongs to the positive class. A negative score means the model leans toward the negative class, which is the other class in the problem. The magnitude tells you how strong the lean is, while the sign tells you its direction.

Reading the Sign as a Decision

The sign of the decision function is enough to reproduce the predicted labels. If you compare each score with 0, you get a boolean array that is True for positive scores and False for negative scores. In the source example, that boolean array is True for the samples that predict returns as red and False for the samples that predict returns as blue. The thresholded decision function and the predictions therefore agree on which samples belong to the positive class.

If a decision value is positive or negative, which branch the model takes and which class that branch corresponds to.

Recovering Labels with classes_

For binary classification, the negative class is always the first entry of the classes_ attribute, and the positive class is always the second entry. If you want to rebuild the full output of predict from decision_function, you must consult classes_. The procedure in the source converts the True and False values into 1 and 0, then uses those integers as indices into classes_. Indexing with 0 selects the negative class, and indexing with 1 selects the positive class.

Which entry of classes_ a positive decision value points to, and which a negative value points to, so the rebuilt output matches what predict returns.

1
Compare each decision score with 0 to get True for positive and False for negative.
2
Convert True to 1 and False to 0.
3
Use those integers as indices into classes_.
4
Index 0 returns the negative class, and index 1 returns the positive class.
5
Compare the rebuilt array with the output of predict for every sample. In the source run, the check reports True only when every element matches, and the result was True.

Reading the Scores in Order

The source shows the first six scores for the test set: 4.136, -1.683, -3.951, -3.626, 4.29, and 3.662. Read them as a ranking of leanings rather than as labels. The sample with 4.29 leans toward the positive class more than the sample with 4.136, and the sample with -3.951 leans toward the negative class more than the sample with -1.683. The table below shows how the sign of each score maps to a direction.

Sample position	Score	Sign	Leans toward	Strength of lean
1	4.136	positive	positive class	strong
2	-1.683	negative	negative class	moderate
3	-3.951	negative	negative class	strong
4	-3.626	negative	negative class	strong
5	4.29	positive	positive class	strongest of these six
6	3.662	positive	positive class	strong

Sign and relative magnitude of the first six test-set scores. Strength is judged only by ordering within this model.

Why the Raw Range Is Not a Fixed Scale

The numeric range of decision_function is not fixed. It can be arbitrary, and it depends on both the data and the model parameters. The source run reports a minimum of -7.69 and a maximum of 4.29 on the test set. A different dataset or different model settings could produce a different spread, so a score of 2 from one model does not necessarily mean the same thing as a score of 2 from another.

What you read	Reliable use	Unreliable use
Sign of the score	Gives the direction of the lean and reproduces the predicted side of the boundary	None needed beyond reading which side of 0 the value falls
Relative ordering within one model	Compares how strongly samples lean relative to each other	Comparing raw values across different models or datasets
Raw magnitude	Shows that one sample leans further than another in the same model	Treating the number as a calibrated confidence or probability

Which parts of a decision_function value can be trusted for interpretation.

Mistakes Learners Make with Scores

Common mistake
Expecting one score per feature instead of one per sample
What it looks like
Input of shape (25, 2) produces output of shape (25, 2) in the learner's head
Why it fails
The binary case returns shape (n_samples,), so the feature dimension is collapsed.
Do this instead
Expect (25,) for 25 samples, one float each.
Common mistake
Reading the sign and assuming the label names come from the sign itself
What it looks like
A positive score is assumed to mean the string red or the number 1 directly
Why it fails
The sign gives only True or False. Label names come from classes_.
Do this instead
Map True and False to 1 and 0, then index classes_, where index 0 is the negative class and index 1 is the positive class.
Common mistake
Treating a score of 2 as a fixed amount of confidence
What it looks like
Believing that 2.0 always means the same strength of belief across models
Why it fails
The range is arbitrary and depends on data and model parameters.
Do this instead
Use the sign for direction and the ordering within one model for comparison.
Common mistake
Assuming classes_ is only needed for multiclass problems rather than for mapping binary signs back to labels

Applying the Shape and Sign Rules

EASY

A binary classifier's decision_function is called on a test set of 40 samples, each with 3 features. Predict the shape of the output. Then state which of the two classes a score of -0.5 leans toward, and which entry of classes_ it maps to.

Hints

Count the samples, not the features, to get the length of the output.
A negative score is False when compared with 0, and False converts to index 0.
Index 0 of classes_ is the negative class in the binary case.

Your attempt (not saved)

Key Takeaways

In the binary case, decision_function returns one floating-point score per sample, so its shape is (n_samples,) and the feature dimension disappears.
The sign of each score gives the direction of the lean: positive leans toward the positive class and negative leans toward the negative class.
The sign alone gives True or False, so classes_ is required to recover the label names, with index 0 as the negative class and index 1 as the positive class.
The numeric range of decision_function is arbitrary and depends on data and model parameters, so read raw values by sign and by relative ordering, not as probabilities.

Related concepts

Introduction to Classifiers and predict

More concepts starting with B