Calculating Classification Loss
Lowering the cross-entropy loss is meant to push the model toward higher accuracy. The loss and the accuracy are not the same number, and the training procedure never reads the accuracy while it adjusts the model.
Try it: Indexing and Slicing
How Python indexes a sequence from 0 (or from the end with negative numbers) and how a slice start:stop:step picks positions — with the exact CPython rules for omitted and out-of-range bounds.
How it works
- Positions run 0 … len−1; negative positions count back from the end (−1 is the last).
- An index outside the string raises IndexError.
- A slice resolves start and stop (omitted → the ends; out of range → clamped), then walks by step, stopping before stop.
- A negative step walks backwards; a step of 0 is a ValueError.
Default run (7 steps): 'PYTHONIC'[1:6:2]: a slice copies characters from start up to (not including) stop, moving by step. … 'PYTHONIC'[1:6:2] → 'YHN' (positions 1, 3, 5).
Simplified: Works on a short string (up to 16 printable ASCII characters); lists slice by exactly the same rules.
Loading the simulation…
The Report Card Is Not the Training Target
Imagine you are building a spam classifier. Your report card is the fraction of messages it labels correctly. That fraction is the number you ultimately care about, and it is the number you would report to anyone who asks how well the model works.
When you train the model, though, you do not hand the training procedure that fraction and ask it to climb. Instead you hand it a different quantity, a loss, and ask it to shrink that. This article explains why that swap happens, what the replacement quantity is, and how a small adjustment in the batch loss function points training at the right part of the model's output.
What do you think happens?
Suppose a training step nudges the model's internal numbers slightly, and no message changes from wrong to right. What happens to the accuracy score?
Reveal answer
It stays exactly the same
Accuracy is a count of correct decisions, so if no decision flips, the count does not change. The next section explains why this matters for training.
Why Accuracy Gives Training No Slope
Training adjusts a model by nudging its internal numbers in a direction that makes the score better. That nudging works smoothly only when the score changes gradually as the numbers change. Classification accuracy does not behave that way. It is a count of correct decisions, so it changes in discrete steps: a message either flips from wrong to right, or it does not.
Between two nearby settings of the model, the accuracy may stay exactly the same and then jump. A function with that stepped behavior is not differentiable. In plain terms, there is no usable slope to follow when deciding which way to adjust the numbers. Training is a procedure that follows slopes, so accuracy gives it nothing to follow between jumps.
Cross-Entropy Loss as a Proxy
Because accuracy cannot be followed directly, the training procedure uses cross-entropy loss as a proxy. A proxy here means a different quantity that is chosen because it is workable to optimize while still being connected to the goal. Cross-entropy loss is a loss function, so it produces a value that training tries to reduce.
Lowering the cross-entropy loss is meant to push the model toward higher accuracy. The loss and the accuracy are not the same number, and the training procedure never reads the accuracy while it adjusts the model.
The two quantities therefore play different roles. Accuracy is the measure of performance you report. Cross-entropy loss is the quantity that training reduces, chosen because it can be optimized where accuracy cannot. Keeping these roles separate is the key to reading everything else in this article.
Following a Batch to the Loss Function
The function that computes the loss for a batch of data is named calc_loss_batch. Its structure stays the same when the model is adapted to classification. The one change concerns which part of the model output feeds into the loss. Before the change, the loss was computed over the output for all token positions, written model(input_batch). After the change, only the output for the last token is passed in.
- 1The model is applied to the batch of inputs, producing an output with three axes.
- 2The indexing operation that follows selects along those three axes, one at a time.
- 3The first colon means keep all batch items, so no example is dropped.
- 4The -1 means the final position along the token axis.
- 5The last colon means keep all remaining values at that position.
model(input_batch)produces output for every token position- [:, -1, :] is the slice that keeps only the final token position
- The first colon keeps every item in the batch
- The last colon keeps all values at that position
Reading the Slice [:, -1, :] One Axis at a Time
A slice with three positions separated by colons selects along three axes. Read the expression one axis at a time. The first colon keeps everything along the batch axis, so every item in the batch remains. The -1 chooses one position along the token axis. Python counts negative indices from the end, so -1 means the final position. The last colon keeps everything along the remaining axis.
[[5, 6], [11, 12]]- The outer list stands in for the batch axis, so there are two items
- Each item has three token positions, each holding two values
- item[-1] keeps only the final token position of each batch item
- Both batch items survive, and every value at the kept position is retained
Tracing the Kept Values
A batch has two items. Each item has three token positions, and each position holds two values. Which values does the slice model(input_batch)[:, -1, :] keep?
- 1Batch axis
The first colon keeps both batch items, so nothing is dropped along the batch axis.
- 2Token axis
The -1 counts from the end of the three positions, so it selects position 2, the final token.
- 3Remaining axis
The last colon keeps both values at that position.
- 4Result
The kept output is the final token's two values for each of the two batch items, while token positions 0 and 1 are set aside.
Two batch items, each contributing the two values at the final token position. Earlier token positions are not passed to the loss.
Common Misreadings of the Last-Token Adjustment
Common mistake Thinking the adjustment changes the model itself
Why it failsThe source says the structure of
calc_loss_batchstays the same. The change concerns which part of the model output feeds into the loss, not the model's internal numbers.Do this insteadRead the change as a change to which slice of the output is passed to the loss function.
Common mistake Reading -1 as the second position
Why it failsPython counts negative indices from the end, so -1 is the final position, not index 1.
Do this insteadWith three token positions, -1 selects position 2, the last one.
Common mistake Assuming the first colon drops batch items
Why it failsThe first colon keeps everything along the batch axis, so every item in the batch remains.
Do this insteadOnly the token axis is narrowed by the slice.
Common mistake Believing the last-token slice is the same as using all tokens
Why it failsThe original
calc_loss_batchpasses the output for all token positions. The adjusted version passes only the final token position.Do this insteadCompare model(input_batch) with model(input_batch)[:, -1, :] to see which positions reach the loss.
Common mistake Treating the loss value as the reported accuracy
Why it failsThe loss is the quantity training reduces, while accuracy is the performance measure you report. They are related by design but are not the same number.
Do this insteadKeep the reported accuracy and the training loss as two separate numbers with two separate roles.
Accuracy and Loss Compared
| Property | Classification accuracy | Cross-entropy loss |
|---|---|---|
| Role in the workflow | Performance measure the practitioner reports | Quantity the training procedure reduces |
| What it counts or measures | Count of correct decisions | A loss value computed for training |
| Behavior as weights change | Changes in discrete steps | Changes smoothly enough to follow |
| Usable slope for gradient training | No usable slope | A slope training can follow |
| Relationship to the goal | The goal being measured | A proxy meant to push toward higher accuracy |
Accuracy and cross-entropy loss play different roles even though training is designed to improve accuracy through the loss.
Checking Your Reading of the Slice
A batch has three items, and each item has four token positions. Without running any code, state how many token positions survive the slice model(input_batch)[:, -1, :] for each batch item, and say which position that is. Then say how many batch items survive.
Hints
- Start with the batch axis, where the first colon keeps everything.
- Count the token positions as 0, 1, 2, 3. Python's -1 refers to the last one.
- Only one token position survives per batch item, and all three batch items remain.
Key Takeaways
- Classification accuracy is the performance measure a practitioner reports, but it changes in discrete steps and has no usable slope, so gradient-based training cannot follow it directly.
- Cross-entropy loss is used as a proxy: training reduces it in the hope of improving accuracy, even though the two are not the same number.
- In
model(input_batch)[:, -1, :], the first colon keeps all batch items, -1 selects the final token position, and the last colon keeps all values at that position. - The adjustment changes only which part of the model output feeds into the loss inside
calc_loss_batch; the structure of that function stays the same. - Common misreadings include treating -1 as index 1, assuming the slice drops batch items, and confusing the training loss with the reported accuracy.