Softmax and Cross-Entropy
Interactive lab
Try it: Softmax and Cross-Entropy
How softmax exponentiates and normalizes logits into class probabilities, why subtracting the largest logit gives identical probabilities without overflow, and how categorical cross-entropy scores the true class.
How it works
- Find the largest logit m.
- Exponentiate every logit, both naively (exp(z)) and shifted (exp(z - m)).
- Sum the exponentials and divide each by the sum to get probabilities that add to 1.
- Predict the class with the highest probability (first one on ties).
- Score the true class t with categorical cross-entropy L = -log p_t, computed via log-sum-exp.
Default run (10 steps): 4 logits z = [2, 1, 0.1, -1]; true class 0. Softmax turns them into probabilities that sum to 1. … Categorical cross-entropy for true class 0: L = -log p_0 = -(2 - 2) + log(1.5672) = 0.4493.
Simplified: A single example with 3 to 6 classes; float64 arithmetic as in NumPy. Real models compute this for a whole batch.
Educational simulation
Loading the simulation…