Random Forest Classifiers

Earlier in a machine learning workflow, a common habit is to tune one model's settings and compare the results by hand. That works, but it answers only one question: which hyperparameter values work best for this model? A more complete approach asks a bigger question: which combination of model, preprocessing step, and hyperparameter values works best? GridSearchCV can answer that bigger question, and this article follows how it does so for a Random Forest Classifier and a competing model.

The central object in this workflow is the parameter grid. It is a Python list, and each entry in that list is a dictionary describing one family of candidates. Each dictionary maps a key to a list of values to try. Because each entry is independent, the search never forces one family's settings onto another family's model. The grid is therefore a list of separate small menus rather than one large menu that mixes everything together.

Read the tree from the top. The list has two entries. The first entry is the SVC family: it names the classifier object, a preprocessing choice, and two hyperparameters, gamma and C. The second entry is the RandomForestClassifier family, which names its own classifier object, the same preprocessing choice, and its own hyperparameter. The RandomForestClassifier entry does not receive gamma or C, and the SVC entry does not receive the Random Forest's settings. Each family defines only the keys that apply to its own model.

The grid above is an illustration written for this article: the SVC values and the StandardScaler-or-None choice follow the source's pattern, while the Random Forest key max_features is a generated example of a family-specific setting. The shape is the lesson, so you can copy the pattern and substitute your own values.

Before running anything, it helps to know how much work a grid will do. The source's method is to count the candidates in each entry and then add the entries together. Within one entry, the number of candidates is the product of the lengths of its value lists. Across entries, you add those counts. Each candidate is then fitted once per cross-validation fold, so the number of fits is the number of candidates multiplied by the number of folds.

Once the grid is defined, the source runs the search as usual. First, the data is split into training and test sets with train_test_split, using random_state=0 so the split is repeatable. The grid is then built around a pipeline object, which the source calls pipe, and GridSearchCV is asked to use five cross-validation folds. Calling fit runs the whole search on the training data only. The test set is held back until the end.

Read the loop as nested. For each candidate, the search fits and scores it on every fold, and the scores for that candidate are combined. Only after every candidate has been evaluated does the search pick the winner. The winner is the candidate with the best cross-validation score, which the search uses for selection. Notice that the outer loop covers the grid's candidates across all families, so a preprocessing choice and a classifier are tried side by side in one search.

After fit, three attributes and methods matter. The best_params_ attribute reports the winning combination across both models and the preprocessing step, not just the best hyperparameter value. The best_score_ attribute reports the best cross-validation score from the training data. The score method, called on the held-out test set, reports how the chosen model performs on data the search never saw.

The best parameters come back as a dictionary that includes the winning classifier object, the winning preprocessing object, and the winning hyperparameter values. In the source's run, the selected classifier is an SVC, the selected preprocessing is a StandardScaler, and the selected values are the ones stored under classifier__C and classifier__gamma. Reading this dictionary tells you which branch of the grid won, not only which number was best.

The two scores answer different questions. The best cross-validation score is the score the search used to choose the winner, so it is somewhat optimistic about the chosen model. The test-set score is computed on data that played no part in the search. When the two are close, as in the source's results, that suggests the search did not overfit the cross-validation folds. A large gap between them would be a reason to be cautious about the reported performance.