Approximate Value Functions
SGD updates can be used with linear function approximation.
Why the Update Simplifies
An approximate value function estimates a value by using a parameter vector θ. When that approximation is linear, stochastic gradient descent updates remain applicable, but the update becomes easier to work with. The important idea is not that linear approximation replaces stochastic gradient descent. Instead, the linear form makes the relevant gradient simple enough that the general update takes a particularly simple form.
The linear case changes the form of the update, but it does not remove the gradient's role.
From Features to an Estimate
The linear approximate value function combines a feature vector with the parameter vector θ to produce an approximate value estimate. The feature vector supplies the inputs used by the approximation, while θ supplies the parameters that determine how those inputs contribute to the estimate. This representation is the reason the linear case is especially convenient for stochastic gradient descent: the dependence of the estimate on θ has a simple structure.
Read the diagram from left to right. The features and θ are not separate value estimates. They are the two ingredients used by the linear approximate value function. The result is one approximate value estimate, which is the quantity whose dependence on θ matters for the update.
The Gradient's Job
The relevant gradient is the gradient of the approximate value function with respect to θ. This wording identifies exactly what is being differentiated: the approximate value function is the quantity being considered, and θ is the variable with respect to which its sensitivity is measured. In the linear case, the gradient has a form that directly reflects the feature vector. That simple relationship is what allows the general stochastic gradient descent update to simplify.
Tracing the General Rule
Conceptual Substitution in the Linear Case
Explain the reasoning sequence that changes a general stochastic gradient descent update into its linear-function form.
Start with the general update: A general stochastic gradient descent update contains a gradient term. At this stage, the update is described in its general form rather than in the simplified linear form.
Identify the differentiated function: The relevant gradient is the gradient of the approximate value function with respect to θ. This identifies both the function being differentiated and the parameter vector affected by the update.
Use the linear representation: Because the approximate value function is linear in θ, its gradient has a form that is directly related to the feature vector.
Substitute the linear gradient: Replacing the general gradient with its linear-case form produces the simplified update. The simplification comes from the gradient's structure, not from removing the gradient.
Interpret the result: The resulting update is easier to work with because the direction supplied by the gradient is represented by the feature vector in the linear case.
The general stochastic gradient descent rule becomes a simpler linear-function update because the gradient of the approximate value function with respect to θ takes a simple feature-related form.
The before-and-after view highlights the precise change. The general rule already has the correct learning structure. The linear assumption supplies a simpler expression for its gradient, and substituting that expression yields the linear update. The learning objective is not changed by this substitution.
How Error Reaches Parameters
The simplified update can be understood as a flow. A prediction error indicates that the current approximate value needs adjustment. The gradient determines how the approximate value responds to changes in θ. In the linear case, that gradient is directly related to the feature vector, so the error is carried through the features to determine how the components of θ are changed. This is why the feature vector appears in the simplified linear update: it represents the direction supplied by the value-function gradient.
This flow does not mean that the error alone determines the update. The error supplies the need for correction, while the gradient supplies the parameter direction. In a linear approximation, the feature vector makes that direction explicit.
Substitution as a Reasoning Chain
The reasoning chain has three essential links. First, begin with a general stochastic gradient descent update. Second, take the gradient of the approximate value function with respect to θ. Third, use the fact that the value function is linear so that this gradient has a simple feature-related form. The final update is therefore a specialized expression of the general rule, not a separate mechanism.
Mistakes in Reading the Update
Treating the linear update as unrelated to stochastic gradient descent
The linear update is obtained by using the linear-case form of the gradient inside the general stochastic gradient descent structure.
Fix:
View the simplified update as the general gradient update after the approximate value function's gradient has been specialized to the linear case.Forgetting that the gradient is taken with respect to θ
The relevant gradient is the gradient of the approximate value function with respect to the parameter vector θ.
Fix:
Name both parts explicitly: differentiate the approximate value function with respect to θ.Saying that linearity removes the gradient
The feature-related form is the result of evaluating the gradient in the linear case.
Fix:
Explain that linearity simplifies the gradient's form while preserving its role in the update.Focusing on numerical substitution when the reasoning is symbolic
The key lesson is the order of reasoning: identify the general gradient, use the linear form, and substitute it into the update.
Fix:
Trace the structural substitution without inventing numerical values.
Practice the Reasoning
A learner says: The linear approximation makes the gradient unnecessary because the update can be written using the feature vector. Explain what is missing from this statement and rewrite it accurately.
Hints
- Identify what the feature vector represents in the linear case.
- State what quantity is differentiated and with respect to which variable.
- Explain how the general update becomes simpler.
What do you think happens?
When a general stochastic gradient descent update is specialized to a linear approximate value function, does the gradient disappear or take a simpler form?
Reveal answer
Answer: It takes a simpler form related to the feature vector.
The linear approximation makes the gradient of the approximate value function with respect to θ particularly simple. Substituting that form into the general update produces the simplified linear update.
The Essential Pattern
- SGD updates can be used with linear function approximation.
- The relevant gradient is the gradient of the approximate value function with respect to θ.
- Linearity gives this gradient a simple form related to the feature vector.
- The simplified linear update is produced by substituting that gradient form into the general SGD update.
- The update becomes simpler, but the gradient remains the reason the update has its direction.
Key Takeaways
- Linear approximate value functions can be trained using stochastic gradient descent updates.
- The key derivative is taken with respect to the parameter vector θ.
- In the linear case, the gradient of the approximate value function has a simple feature-related form.
- Substituting that form into the general update produces the simplified linear update.
- The gradient's role is preserved even though the update's expression becomes easier to use.