Linear Function Approximation

From Unknown Shape to Useful Estimate

A reinforcement learning system may need to estimate a value function whose complete shape is unknown. Rather than represent that entire shape directly, linear function approximation constructs an estimate from features and weights. Fourier basis functions are one important choice of features: they use sine and cosine functions with different frequencies, then combine their contributions through weights.

Reading a Repeating Period

A periodic function is defined by repetition after a period T. The period identifies the length of the repeating structure: after one period, the function repeats the same pattern. Fourier series are naturally associated with this setting because they describe periodic functions as weighted sums of sine and cosine basis functions.

Combining Fourier Features

A Fourier basis uses sine and cosine functions as building blocks. Different frequencies make the basis functions vary in different ways. Each basis function supplies a feature, and its associated weight scales that feature's contribution. The approximation is the sum of the weighted contributions.

This example is deliberately about structure rather than a numerical result. The important trace is that frequencies determine which patterns are available, while weights determine how much those patterns affect the final approximation.

Turning Features into a Value

A linear value estimate turns feature values and weights into one prediction through a weighted sum. In a Fourier-based representation, the features may be values supplied by sine and cosine basis functions. The same broad mechanism applies more generally: represent the state through features, pair those features with weights, and combine the paired contributions into an estimate.

Why Representation Matters

Feature selection places prior domain knowledge into a reinforcement learning system. The designer chooses a representation that emphasizes patterns considered useful for the problem instead of presenting the learning method with an undifferentiated description. Because the feature representation strongly influences what the system can learn and generalize, the choice of features matters alongside the choice of learning method.

Fourier Features in Reinforcement Learning

Fourier basis functions are valuable in reinforcement learning because they are easy to use and can perform well even when the function being approximated is unknown. This matters because reinforcement learning does not begin with a fully known function whose Fourier-series weights can simply be obtained from formulae. Instead, the system must learn about an unknown function while using a chosen representation.

The same broad feature-and-weight idea is relevant to multi-dimensional continuous state spaces. The source also describes a form of the Fourier basis for reinforcement learning problems in which the functions do not have to be periodic. Therefore, periodic Fourier-series intuition is useful for understanding the basis, but it does not by itself limit every reinforcement learning application to periodic functions.

Efficiency and Model Class

Linear approximation methods also differ in their practical trade-offs. LSTD favors data efficiency, but it has a higher computational scaling cost than the other linear methods described in the source. This makes LSTD a useful example of a broader design tension: using data efficiently can come with greater computational cost.

Nonlinear methods take a different modeling direction. The source includes artificial neural networks trained by backpropagation and variations of stochastic gradient descent among these methods. Their popularity in reinforcement learning has led to the term deep reinforcement learning. The key contrast is therefore between a linear feature-weight estimate and a nonlinear model based on neural networks and their training methods.

Mistakes in Feature Reasoning

• Treating every basis function as if it had the same frequency.

  Different frequencies distinguish the patterns available to the approximation.

  Fix: Track frequency as a property that determines the kind of variation each basis function can provide.

• Confusing a feature with its weight.

  The basis function supplies the feature, while the weight controls that feature's contribution.

  Fix: Separate the feature value from the weight that scales it, then add the weighted contributions.

• Assuming that linear approximation is automatically too simple for useful value estimates.

  The source reports that a linear approximator can work well when its features are chosen appropriately.

  Fix: Evaluate the representation and the learning method together.

• Assuming that reinforcement learning can always obtain Fourier weights from simple formulae.

  Simple formulae for Fourier-series weights apply when the function being approximated is known, while reinforcement learning commonly involves unknown functions.

  Fix: Recognize Fourier basis functions as a useful representation for learning about an unknown function.

• Describing LSTD as both data-efficient and computationally cheaper without qualification.

  The source states that LSTD has a higher computational scaling cost than the other linear methods described.

  Fix: Remember both sides of the trade-off: data efficiency and higher computational scaling cost.

Practice the Trace

Key Takeaways

1. A periodic function repeats its structure after a period T.
2. Fourier basis functions use weighted sine and cosine features, with frequencies determining available patterns and weights controlling contributions.
3. A linear value estimate combines feature values and their weights through a weighted sum.
4. Feature selection injects prior domain knowledge and strongly influences learning and generalization.
5. LSTD favors data efficiency at a higher computational scaling cost, while nonlinear neural-network methods follow a different modeling direction associated with deep reinforcement learning.