Gradient Boosted Trees
Gradient boosted trees are a different kind of ensemble from a random forest.
A Model That Memorizes Its Training Data
Tree-based models come with settings, called parameters, that you choose before training starts. Changing a setting can change how well the model performs, sometimes in ways that are hard to predict. This article begins with a model that already runs and asks what each setting is doing inside it.
Gradient boosted trees give a clear example of why settings matter. The source describes a case where a boosted model overfits badly with its default settings. Its training accuracy reaches 1.000, which is a sign that the model has memorized the training data rather than learned a pattern that generalizes. Two remedies are available: stronger pre-pruning through a smaller maximum depth, or a lower learning rate. To understand why these remedies work, you first need to understand how boosting builds its trees.
Trees Built in a Chain
Gradient boosted trees are a different kind of ensemble from a random forest. A random forest builds its trees independently. Gradient boosting builds them one after another, and each new tree tries to correct the mistakes of the tree before it. The individual trees are usually shallow, with depth between one and five in the source's description. The main idea is to combine many simple weak learners, each of which fixes a small part of what the others got wrong.
The chain works in a loop. The first tree makes its predictions. Some of those predictions are wrong, and the remaining error is what the next tree learns from. The second tree is added to the overall prediction, which changes the error that remains. A third tree then learns from that updated error, and so on. Each step is small and focused on what is still wrong, so the combined prediction improves gradually.
Three Settings That Shape the Ensemble
Three parameters control most of the behavior discussed in this article. n_estimators decides how many trees exist. max_depth decides how complex each tree is. max_features decides how different the trees are from one another. The sections below take each one in turn and explain why it matters for the final model.
learning_rate: The Step Size of Each Correction
In gradient boosting, learning_rate controls how strongly each tree corrects its predecessors. A higher learning rate lets each tree make a stronger correction, which allows a more complex model. A lower learning rate makes each correction smaller. The two settings are linked: a lower learning rate needs more trees to reach a model of similar complexity. This is why n_estimators and learning_rate are tuned together rather than separately.
Read the diagram as a trade-off rather than a rule about which setting is better. The source describes the learning rate as the lever that controls how strongly each tree corrects its predecessors. Changing it shifts how far every individual tree moves the prediction, so the count of trees needed to reach a given complexity changes with it.
Predicting the Effect of a Lower Learning Rate
Suppose a boosted model overfits, with training accuracy at 1.000, and you lower its learning rate while leaving n_estimators unchanged. Before reading on, predict what happens to the model's complexity and to the gap between training and test accuracy.
What do you think happens?
With n_estimators fixed, what happens to the model's complexity when you lower learning_rate?
Reveal answer
Complexity decreases, because each tree makes a smaller correction.
The source says a higher learning rate lets each tree make stronger corrections, which allows more complex models. With the same number of trees, a lower rate means smaller corrections, so the model is less complex. This can reduce the gap where training accuracy is high and test accuracy lags behind, which is the overfitting pattern the source describes.
Random Forest Tuning Versus Boosting Tuning
The two ensemble methods are tuned in different directions. Both use the same kinds of tree-level settings, but the role of the number of trees differs, and only boosting has a learning rate. The comparison below shows how each method builds its trees and which parameter controls the influence of each new tree.
| Aspect | Random forest | Gradient boosting |
|---|---|---|
| How trees are built | Independently, each on a bootstrap sample | In sequence, each correcting the previous trees |
| Typical tree depth | Trees are deeper than a single decision tree in the source's description | Shallow trees, usually depth one to five |
| Effect of more trees | Always better, with diminishing returns | Adds complexity and can overfit |
| Key extra parameter | None beyond the tree-level settings | learning_rate controls each correction's strength |
| Tuning approach | Build as many trees as budget allows | Fix n_estimators by budget, then search learning rates |
Tuning differences between the two ensemble methods, as described in the source
Mistakes Beginners Make When Tuning
Common mistake Treating more trees as always better for gradient boosting
What it looks likeRaising n_estimators from 100 to 1000 without watching test accuracy, then assuming the bigger model is safer.
Why it failsThe source says that for gradient boosting, more trees add complexity and can overfit. The random forest rule does not transfer.
Do this insteadWatch training and test accuracy together as n_estimators grows, and pair the number of trees with a learning_rate search.
Common mistake Changing learning_rate without adjusting n_estimators
What it looks likeLowering learning_rate and keeping the same number of trees, then expecting the same complexity.
Why it failsA lower learning rate needs more trees to reach a model of similar complexity.
Do this insteadTreat learning_rate and n_estimators as a linked pair when you change either one.
Common mistake Assuming a surprising score must come from a bug in the code
What it looks likeTwo runs with the same settings give different test scores and the cause is blamed on the algorithm.
Why it failsThe source advises checking whether the split and
random_stateare identical across runs, since those settings make a comparison fair.Do this insteadFind the first line where randomness enters, and fix random_state before comparing settings.
Common mistake Tuning max_features before having a baseline score to compare against
Tracing a Chain of Trees by Hand
A boosted model starts with a prediction of 0 for a single training point whose true value is 10. Tree 1 corrects 60 percent of the remaining error, and tree 2 corrects 60 percent of what is left after tree 1. Describe in words how the prediction changes after each tree, and say what would happen to the speed of progress if learning_rate were lowered so that each tree corrected only 20 percent.
Hints
- Track the remaining error after each tree, not only the prediction.
- Each tree learns from the errors left by the trees before it, as shown in the chain diagram.
- A smaller correction per tree means more trees are needed to reach the same prediction.
Key Takeaways
- Gradient boosted trees are built one after another, and each new tree corrects the mistakes left by the previous ones, usually using shallow trees.
n_estimators,max_depth, andmax_featureseach control a different part of the model: how many trees exist, how complex each tree is, and how different the trees are from one another.- For random forests, more trees is always better with diminishing returns, while for gradient boosting more trees add complexity and can overfit.
learning_ratecontrols how strongly each boosted tree corrects its predecessors, and a lower rate needs more trees to reach a model of similar complexity.- When a result surprises you, watch training and test accuracy together and check the line where randomness or a setting enters the model.