Gradient Boosting Classifiers

Suppose you have fitted a gradient boosting model to a two-dimensional toy dataset and want to see what it has learned. The most familiar picture is a single line that separates the region predicted as one class from the region predicted as the other. That line is useful, but it answers only one question. It tells you where the predicted outcome changes, and nothing about how sure the classifier is at any particular location.

This article works from a plot that answers both questions at once. Before reading any further, keep one question in mind: what extra information does the colored view give that a boundary line alone cannot? You will return to that question at the end of the article.

A classifier assigns each point in the input plane to a class. Imagine every location in the two-dimensional plane receiving a label. Some locations get the first class, others get the second, and the places where the label switches form the boundary between the classes. Drawing only that boundary tells you where the predicted outcome changes. It does not tell you how strongly the classifier holds any of those predictions.

The left panel of the figure shows the decision boundary, which is the boundary between the two classes. Read it as a map of predicted outcomes. Each location in the plane belongs to one predicted class, and the line marks where that assignment flips. If you want to know which class the model predicts for a given spot, this is the panel to read.

The example uses training points drawn as circles and test points drawn as triangles. Both kinds of point sit on top of the same boundary, so you can check whether the learned line separates the classes where the data actually lies. The boundary panel is the place to check that separation directly.

The right panel shows the decision function for all points in the plane, colored according to its values. The decision function encodes not only the predicted outcome but also how certain the classifier is. Where the function is strongly positive or strongly negative, the model leans firmly toward one class. Where the value is close to the middle of its range, the model is less decisive. Because the colors vary continuously across the plane, the panel does not split into two flat regions the way the boundary panel does.

The colorbar is the key to reading this panel. The colorbar takes its values from the object returned by the plotting helper, so it supplies the scale that maps each color to a value of the decision function. Without the colorbar you can see that one area is colored differently from another, but you cannot tell how far apart their scores are. With it, you can compare how strongly one area leans toward a class against another area.

It is tempting to draw the boundary lines on top of the certainty color map so that everything appears in one picture. This often backfires. A certainty color map fills the whole plane with color, and a boundary line drawn over a busy color gradient can blend into the surrounding shades. The line that was the most important feature of the plot becomes the hardest thing to see.

Separate panels fix the problem because each view keeps its own visual space. To find the boundary, read the left panel directly. To judge certainty, read the right panel and its colorbar. The two readings are then linked by location: the same point in the plane appears at the same position in both panels.

When a two-panel plot looks wrong, the cause is usually one specific line in the plotting code rather than the whole script. The result travels from the code to the screen along a path. The boundary and the score map each depend on particular objects and arguments, so a change to one of them changes only that part of the figure.

The plotting snippet assumes that a gradient boosting model named gbrt has already been fitted, and that the arrays X, X_train, X_test, y_train, and y_test exist from earlier in the chapter. The snippet does not create these objects, so it will only run inside that setup. If the figure fails or looks different from the description, first confirm that these objects exist and hold the data you expect.

Once the setup is confirmed, locate the colorbar line. The colorbar takes its values from the object returned by the plotting helper that draws the decision function. If the colorbar shows an unexpected scale, the line that attaches it to that returned object is the place to look. Changing which object the colorbar is attached to changes the scale shown beside the right panel, while the colors themselves come from the decision function values.