Decision Boundary Basics

A classifier assigns every point in the input plane to a class. If you draw only the line where the predicted class changes, you learn where the predicted outcome switches from one class to the other. You learn nothing about how sure the classifier is at a given location. A point sitting right at the edge of a region and a point sitting deep inside it would look identical to a boundary-only plot, because both receive the same label.

This article works from a plot that answers both questions at once. It places two views side by side. The left view shows the decision boundary. The right view shows a decision function, colored across the whole two-dimensional plane. The example model is a gradient boosting model trained on a two-dimensional toy dataset, where training points appear as circles and test points appear as triangles. Keep one question in mind as you read: what extra information does the colored view give that a boundary line alone cannot?

Imagine two points that both fall on the same side of the boundary. The left panel treats them identically: each gets the label of its region. Now consider where each point sits. One is just inside the edge of its region. The other is far from the edge, well inside its region. Trace each point through the figure in two steps. First, find its region on the left panel and read its label. Second, find its location on the right panel and read the color, which corresponds to a decision function value on the colorbar.

The left panel shows the decision boundary, which is the boundary between the two classes. It answers a categorical question: for each location in the plane, which class does the model predict? The right panel shows the decision function for all points in the plane, colored according to its values. It answers a graded question: how strongly does the model hold its prediction at each location? The figure caption describes the same pairing, with the boundary on the left and the decision function on the right, both for a gradient boosting model on the two-dimensional toy dataset.

Read the two panels as a pair. The left panel tells you where the label switches. The right panel tells you how the strength of the prediction changes as you move across that switch. Because the right panel is continuous rather than made of two flat regions, it carries more information than the boundary alone.

The colorbar is the key to reading the right panel. It maps colors to the values of the decision function, which gives the color scale its meaning. Without it, you could see that one area is colored differently from another but could not tell how far apart their decision function values are. With it, you can compare how strongly one area leans toward a class compared with another area.

In the plotting code, the colorbar takes its values from the object returned by plot_2d_scores. That object is what supplies the scale for the decision function, so the colorbar is tied to that specific returned object rather than to the boundary.

The plotting code is assumed to run after a gradient boosting model named gbrt has already been fitted, and after the arrays X, X_train, X_test, y_train, and y_test exist from earlier in the chapter. The snippet does not create these objects itself, so it only runs inside that setup. When you read the code, do it first, then compare what it produces with the description of the two panels.

When a two-panel plot looks wrong, the cause is usually one specific line rather than the whole script. Trace the path a result takes from the code to the screen, and check each step against what you expected. The first step where the output diverges is where the problem begins. For the colorbar, that means checking the object that plot_2d_scores returns, since the colorbar depends on it. For the boundary, check the inputs the boundary depends on, since the boundary is what the left panel is built to show.