Gradient Descent and Stochastic Gradient Descent
A gradient at w produces the first-order Taylor approximation of f around w.
Choosing the Next Parameter
Suppose the goal is to minimize a function f by changing its parameter. Looking at the entire function may be difficult when deciding what to do next. Gradient descent begins with a narrower question: what does f look like near the parameter value we currently have?
The central idea is local decision-making. Use information at the current parameter to estimate which nearby direction could produce a lower function value.
What do you think happens?
If an approximation is created at the current parameter, should it be trusted equally far away from that parameter?
Reveal answer
Answer: No, because the approximation is local and can become loose farther away.
The approximation is formed around the current point. Its usefulness is local, so gradient descent also needs a reason to limit how far the candidate parameter moves.
The Local Taylor View
Let w be the parameter around which the approximation is built, and let u be a nearby candidate parameter. The gradient at w produces a first-order Taylor approximation of f around w. The approximation starts with the current function value f(w), then adds the gradient-based term 〈u − w, ∇f(w)〉. In symbols, the estimate is f(u) ≈ f(w) + 〈u − w, ∇f(w)〉.
f(u) ≈ f(w) + 〈u − w, ∇f(w)〉
Reading the Approximation
The approximation gives a local way to compare candidate parameters. Starting from f(w), it estimates how the function changes when moving from w toward u. A candidate that receives a lower estimate according to this local expression is a candidate for reducing the function value.
Interpreting a Candidate Move
Use the local approximation to describe what is being estimated when the current parameter is w and the candidate parameter is u.
Start at the current value: The estimate begins with f(w), the function value at the parameter where the gradient was measured.
Measure the parameter change: The difference u − w describes how the candidate differs from the current parameter.
Use the local gradient: The term 〈u − w, ∇f(w)〉 estimates the change associated with that candidate movement using the gradient at w.
Form the local estimate: Add the gradient-based term to f(w) to obtain the first-order estimate of f(u).
The expression estimates f(u) using only the current value f(w), the candidate displacement u − w, and the gradient ∇f(w) measured at w.
Limiting the Step
A tempting strategy would be to minimize the local approximation directly. The difficulty is that the approximation can become loose when the candidate w is far from the current parameter w(t). A low value predicted by the approximation is therefore not enough by itself to justify an arbitrarily distant move.
f(w) ≈ f(w(t)) + 〈w − w(t), ∇f(w(t))〉
Balancing Distance and Descent
Gradient descent addresses the problem by balancing two goals. It seeks a lower value according to the local approximation, while also keeping the new parameter close to the current parameter. The update is motivated by jointly minimizing the distance from the current parameter and the approximation of f around that parameter.
The parameter η controls the tradeoff between these two terms. In the usual gradient-descent form, the resulting move is represented as w(t + 1) = w(t) − η∇f(w(t)). The negative gradient direction is the locally downhill direction, while η controls how strongly the update responds to that local gradient. The important motivation is not to trust the approximation arbitrarily far away; it is to use the approximation while limiting the movement.
The update rule is best understood as a compromise: improve the local estimate, but do not leave the neighborhood in which that estimate was formed without a distance-based constraint.
Convexity and Reliability
Convex functions have a special role in this motivation. For a convex function, the first-order Taylor approximation lower bounds the function. This connects minimizing the local approximation with searching for a lower value of the original function.
Gradient Descent Variants
| Approach | Gradient information described in the source pack | What the comparison highlights |
|---|---|---|
| Gradient descent | The gradient at the current parameter is used to form the local approximation. | The update is motivated by balancing the local approximation with distance from the current parameter. |
| Stochastic gradient descent | The source pack names this approach but does not specify its sampling or update procedure. | Do not infer operational details from the name alone; those details require additional material. |
Common Reasoning Mistakes
Treating the first-order approximation as an exact description of the whole function
The approximation is useful locally but can become loose when the candidate is far from the point where the gradient was formed.
Fix:
Interpret the expression as a local estimate and keep the candidate update near the current parameter.Minimizing the local approximation without considering distance
The prediction may no longer be reliable far from the current parameter.
Fix:
Balance lowering the local approximation with staying close to the current parameter.Ignoring the role of the gradient's evaluation point
The gradient in the local approximation is measured at the current parameter w(t).
Fix:
Track the evaluation point: the approximation around w(t) uses ∇f(w(t)).Assuming convexity is irrelevant to the motivation
The source specifically gives the lower-bound property for convex functions.
Fix:
Associate the lower-bound connection with convex functions while still remembering that locality matters.
Practice the Motivation
Explain, in your own words, why gradient descent does not simply minimize the first-order Taylor approximation without a distance constraint.
Hints
- Identify where the gradient was measured.
- State what can happen when the candidate parameter moves far from that point.
- Explain what η controls in the update motivation.
Complete the reasoning chain: current parameter → local gradient → first-order approximation → distance-aware update. For each arrow, state what information is passed to the next step.
Hints
- The current parameter identifies where the approximation is formed.
- The gradient supplies the gradient-based term.
- The approximation estimates candidate function values locally.
- The distance-aware update limits how far the next parameter moves.
Key Takeaways
- The gradient at w produces the first-order Taylor approximation of f around w.
- The approximation estimates a candidate value by combining f(w) with a gradient-based change term.
- The estimate is local and can become loose when the candidate moves far from the point where the gradient was measured.
- Gradient descent is motivated by jointly lowering the local approximation and keeping the new parameter close to the current parameter.
- For a convex function, the first-order approximation lower bounds the function, which connects local minimization with the search for a lower original function value.
Key Takeaways
- A gradient provides a first-order local approximation around the current parameter.
- The approximation helps identify a locally downhill direction, but it should not be trusted arbitrarily far away.
- Gradient descent balances improvement according to the local approximation with distance from the current parameter.
- The parameter η controls the tradeoff in the update.
- Convexity is important because the first-order approximation lower bounds a convex function.