Loss Functions
A derivative explains how a small input change affects the output of a continuous function.
From Input to Outcome
A loss function helps describe how far a candidate result is from a target. To understand how such a result can change, begin with a simpler situation: a function receives one real number and produces another. If the input is adjusted slightly and the function is continuous and smooth, its output changes slightly as well. A derivative gives us a way to describe that local input-output change.
The important idea is local change: the derivative is not merely the output itself. It describes how the output responds when the input changes slightly.
Reading a Derivative
The basic derivative example uses a real-number input x and an output y. We can describe the relationship as a function that receives x and produces y. The derivative relates a small change in x to the resulting change in y. It therefore answers a local question: if the input moves slightly, what happens to the output?
Following a Single-Input Change
A continuous, smooth function receives one real-number input and produces one output. Explain what the derivative contributes when the input is adjusted slightly.
Identify the input: The function receives one real number, represented by x.
Observe the output: The function produces an output, represented by y.
Make a small adjustment: Consider what happens when x is moved slightly rather than replaced by an unrelated value.
Use the derivative: The derivative describes the local relationship between the small change in x and the resulting change in y.
The derivative supplies a description of how the output responds locally to a small input change.
Why Tensors Need Gradients
The single-input picture changes when an operation receives multidimensional data. Instead of one real number, a tensor operation may involve several values arranged as tensors. The question remains the same: how does a small change in the input affect the result? A gradient extends the derivative idea to this multidimensional setting.
A gradient is the derivative of a tensor operation. It extends the single-input derivative idea so that the operation can be considered in a multidimensional input setting.
Tracing a Loss Calculation
A common tensor-based sequence begins with an input vector x and a matrix W. The matrix is used with x to compute a target candidate called y_pred. That candidate is then compared with the target y through a loss function. The loss represents the mismatch between the candidate and the target.
Following the Candidate and Target
Trace the roles of x, W, y_pred, y, and the loss in a tensor-based loss calculation.
Start with x: x is the input vector supplied to the operation.
Use W with x: The matrix W is used with the input vector to compute a result.
Name the result: That computed result is the target candidate y_pred.
Bring in y: y is the target against which the candidate is compared.
Calculate the loss: The loss represents the mismatch between y_pred and y.
The sequence is x and W produce y_pred, then y_pred and y participate in the loss calculation.
| Item | Role in the sequence |
|---|---|
| x | Input vector supplied to the operation |
| W | Matrix used with x |
| y_pred | Target candidate computed from the operation |
| y | Target used for comparison |
| Loss | Representation of the mismatch between y_pred and y |
Mistakes in the Change Story
Treating the derivative as the function's output
The output and the derivative answer different questions. The output is what the function produces; the derivative describes how that output changes when x is adjusted slightly.
Fix:
Keep the two roles separate: the function maps x to y, while the derivative describes the local input-output change.Using the single-number picture for every input
A tensor operation may involve several values arranged as tensors, so the input is multidimensional.
Fix:
Use the term gradient for the derivative idea in this multidimensional setting.Confusing the target candidate with the target
The sequence distinguishes the computed candidate y_pred from the target y.
Fix:
Track the names carefully: x and W produce y_pred, and y_pred is compared with y through a loss.Treating the loss as the candidate itself
The loss represents the mismatch between the candidate and the target.
Fix:
Describe the loss as the result of comparing y_pred with y.
Practice the Trace
Explain the following sequence in your own words: an input vector x and a matrix W are used to produce y_pred, and y_pred is compared with y through a loss. Then state why a gradient is relevant when the operation accepts tensor inputs.
Hints
- First identify which items are inputs and which item is the computed candidate.
- Separate the candidate y_pred from the target y.
- Connect the gradient to the multidimensional nature of tensor inputs.
What do you think happens?
Before reading the explanation, predict which quantity represents the mismatch: x, W, y_pred, y, or the loss.
Reveal answer
Answer: The loss
The input vector and matrix participate in producing the target candidate. The candidate is compared with the target, and the loss represents the resulting mismatch.
Key Takeaways
- For a continuous, smooth function, a small input adjustment produces a small output adjustment.
- A derivative describes the local relationship between a change in one real-number input and the resulting change in its output.
- A gradient extends the derivative idea to operations with multidimensional tensor inputs.
- A matrix W and input vector x can produce a target candidate y_pred.
- The candidate y_pred is compared with target y, and the loss represents their mismatch.
Key Takeaways
- A derivative explains how a small change in a single input affects a continuous function's output.
- A gradient applies the same change-focused idea when an operation receives multidimensional tensor inputs.
- In a common loss sequence, x and W produce y_pred.
- The candidate y_pred is compared with target y, and the loss represents their mismatch.