AdaBoost with Decision Stumps
Try it: AdaBoost with Decision Stumps
How AdaBoost keeps a distribution D over the training examples, asks the weak learner for the decision stump with the smallest weighted error eps_t, gives it weight w_t = 1/2 ln(1/eps_t - 1), raises the weight of its mistakes, and combines the stumps into sign(sum w_t h_t(x)).
How it works
- Start with the uniform distribution D(1) = 1/m.
- Weak learner: try every stump b·sign(theta - x) (thresholds around and between the points, both signs) and keep the one with the smallest weighted error eps_t.
- Give it the weight w_t = 1/2 ln(1/eps_t - 1).
- Reweight: D(t+1)_i ∝ D(t)_i exp(-w_t y_i h_t(x_i)), so misclassified examples gain weight.
- The ensemble predicts sign(sum_t w_t h_t(x)); its training error is shown after each round.
Default run (12 steps): 9 labelled points on a line. Start AdaBoost with the uniform distribution D(1) = 1/9 on every example; run up to T = 4 rounds. … Reweight: D(5)_i = D(4)_i exp(-w4 y_i h4(x_i)) / Z. Misclassified D, E, F gain weight. The ensemble sign(0.35 h1 + 0.55 h2 + 0.8 h3 + 0.69 h4) gets 0/9 training examples wrong.
Simplified: 1-D toy data (2–12 labelled points) and at most 10 rounds. Ties between equally good stumps go to the smaller threshold (then b = +1); a score of exactly 0 predicts +1; if a stump has eps = 0 boosting stops because its weight would be infinite.
Loading the simulation…