Differentiable Optimization

From Learning Goal to Update

Learning begins with a broad objective: minimize the risk function L_D. Stochastic Gradient Descent, or SGD, addresses this objective by using one fresh example at each iteration. Instead of directly constructing the full gradient of the risk from all examples, SGD constructs a random gradient vector from the sampled example and uses that vector as an estimate of the direction represented by the risk gradient at the current parameter value.

One Sample, One Stochastic Vector

At iteration t, SGD considers the current point w(t) and draws one fresh example, written as z. The sampled example determines a sampled loss, written as ℓ(w, z). The gradient is taken with respect to w, and the resulting gradient is evaluated at the current point w(t). That evaluated gradient is the random vector v_t.

Reading the Gradient Precisely

Why Sampling Still Represents Risk

A single sampled gradient does not necessarily equal the exact gradient of the risk function. Its relevance comes from considering the expected value of the random vector over the sampling of z. The expected sampled gradient matches the gradient of the risk function. Therefore, although an individual v_t can differ from the full risk gradient, the sampling construction is unbiased with respect to that risk gradient.

Repeated Parameter Updates

SGD repeats the same connection at successive iterations. At each iteration, the current point w(t) is paired with one freshly sampled example. That example produces a sampled loss and, from it, the random vector v_t. The vector supplies an estimate of the direction represented by the gradient of the risk at the current point. The process therefore connects the broad objective of minimizing L_D with information obtained one fresh example at a time.

The important mechanism is not that every individual sampled vector is the exact risk gradient. The mechanism is that each vector is constructed at the current point from a fresh example, and the expected value of this random construction corresponds to the risk gradient.

Common Reasoning Mistakes

• Treating v_t as the exact gradient of L_D for every sampled example.

  The sampled gradient is an unbiased estimate, not necessarily the exact risk gradient on one iteration.

  Fix: Distinguish an individual sampled vector from its expected value over the sampling of z.

• Differentiating the entire risk function when describing the construction of v_t.

  The construction is local to the sampled example.

  Fix: Start with the sampled loss ℓ(w, z), differentiate with respect to w, and evaluate at w(t).

• Leaving out the evaluation point.

  The random vector is the gradient evaluated at the current point, not an unspecified gradient.

  Fix: Name all three parts: the function ℓ(w, z), the variable w, and the point w(t).

• Confusing the learning objective with the sampling procedure.

  The objective is to minimize L_D; the sampled example is the source of an estimate used to address that objective.

  Fix: Label L_D as the target and v_t as the estimate.

Practice the Construction

Essential Takeaways

1. Learning seeks to minimize the risk function L_D.
2. SGD uses one fresh sampled example at each iteration.
3. The sampled example produces the loss ℓ(w, z), which is differentiated with respect to w and evaluated at w(t) to construct v_t.
4. An individual v_t need not equal the exact gradient of L_D.
5. The expected value of the sampled gradient matches the gradient of the risk function, making the sampled construction an unbiased estimate.