Interpreting Linear Regression Coefficients

A trained linear regression model is more than a prediction machine. It stores one number per input column, and those numbers are called coefficients. Reading them tells you what the model has learned about each input. The practical payoff is that you can often explain a model's behavior by looking at a short list of numbers instead of treating the model as a black box.

This article follows one concrete case. An earlier step combined day-of-week and time-of-day inputs into interaction features. That produced a linear model that performed about as well as a random forest on the same problem. The interaction features made the model easy to inspect, because there is one coefficient for each day and time combination. This article is about reading those coefficients correctly, and about the small code steps that carry them from the model into a plot.

A coefficient list is only meaningful if you know which input each number belongs to. The first step is therefore to build a list of human-readable names for the original inputs. In this case there are two groups: a list of hour labels generated in steps of three hours across the day, and a list of seven day abbreviations. The two lists are joined into one list of base feature names.

The program that follows does three things in sequence. It asks the transformer for the names of all the interaction features it generated. It then applies a boolean test to the learned coefficients to keep only the features whose coefficient is not zero. Finally, it extracts those nonzero coefficients as a separate list. As you follow the execution, keep track of two lists that must stay in step: the feature names and the coefficients. Each position in one list describes the same position in the other.

The filtering relies on a NumPy idiom. Comparing an array to a value produces an array of True and False, and indexing with that array keeps only the True positions. The example below is not the program from the source. It uses made-up numbers to show the same mechanism in isolation, so you can see what the mask does before applying it to real coefficients.

Before you run code like this on a real model, predict its size. If every hour label is paired with every day, the number of interaction columns is the product of the two group sizes. In the setup described here, that is 8 hour labels multiplied by 7 day abbreviations, giving 56 columns. The nonzero filter then removes any of those whose learned coefficient is exactly zero, so the final list can be shorter than 56. Working out the expected count first gives you a check: if the output length does not match your prediction, something upstream changed.

A coefficient is the expected change in the target for a one-unit change in one input column. The phrase 'expected change' matters: the coefficient describes the average effect the model learned, not a guaranteed result for a single case. The phrase 'one input column' matters too: the rule is stated for one input at a time, with the other inputs held fixed. The coefficient's sign gives the direction of the change, and its size gives how much the target moves per unit of that input.

Once the names and coefficients are aligned, the plotting step draws one marker for each nonzero coefficient. The figure is set wide and short so many labels fit along one line. The tick labels are rotated vertically so long names do not overlap, and the feature names are placed at the tick positions. The vertical spread of the markers shows which combinations pushed the prediction up or down.

A plot is only as trustworthy as its labels, and this one can mislead. In the plotting code, the coefficient values are passed as the first and only data argument, so they are drawn on the vertical axis. The feature names are placed on the horizontal axis through the tick settings. Yet the horizontal axis is labeled as the feature magnitude, and the vertical axis is labeled as the feature. Read literally, the labels describe the opposite of what each axis shows.

To locate the problem, trace the three things that determine what a reader sees: which array is passed as the data, which values are used for the tick positions and tick labels, and which text is given to each axis-label call. A swap shows up when the label text for an axis does not match the variable plotted on it. The fix is to correct the axis-label calls so the vertical axis names the coefficient and the horizontal axis names the feature, then check that the markers and their labels still line up.

A trained logistic regression model is not only a prediction tool. It also stores a number, called a coefficient, for every feature it learned from. In a text classifier, the features are the words in the documents, so each word has its own coefficient. Those coefficients record what the model picked up from the training data. Each coefficient belongs to one feature, which in this setting is one word.

This raises a practical question. A text model trained on movie reviews in the source had 27,271 features left after infrequent words were removed. That is far too many coefficients to read one by one, so the approach is to look only at the largest coefficients and see which words they belong to. The rest of this section explains how to read those coefficients once you have picked them out.

Read the sign first. Each coefficient is attached to one word and expresses how the model treats that word when it decides a label. Negative coefficients belong to words the model considers indicative of negative reviews, while positive coefficients belong to words that indicate positive reviews. The sign therefore tells you which side of the decision the word pushes toward. The source's movie-review examples fit this intuition: on the negative side, worst, waste, disappointment, and laughable are indicators of bad reviews, while excellent, wonderful, enjoyable, and refreshing indicate positive reviews.

Some words look unclear when read alone. The source names bit, job, and today as examples. Their weight may come from phrases such as good job or best today, where the surrounding words carry the meaning. A coefficient for a lone word is therefore a summary of how that word behaved in the training data, and it may depend heavily on the phrases it appeared in.

The source's sorted example makes this concrete. The word bit has a negative coefficient, but its magnitude is very small, so under the reading rules it is a weak signal and reads as slightly negative rather than as a strong indicator. A careful reader would still check the word's neighbors in the data before drawing a conclusion, because bit may only carry meaning inside a phrase.

The table below is a small illustration of the reading process. The coefficient values are invented for this lesson so the pattern is easy to see. They are not the source model's actual numbers. The words are sorted from the most negative coefficient to the most positive, and each word is labeled with the review type its sign indicates.