Function Approximation for Continuous State Spaces

From Samples to a State-Space Function

A learning problem may provide function values only at some samples, while the desired result is a function defined between those samples or even beyond them. Reinforcement learning faces a closely related challenge when its state space is continuous and has multiple dimensions: the learning system needs a way to approximate a function across that space. Polynomial basis functions are relevant because they provide one family of candidate functions for this approximation task.

A Candidate Family in Action

Think of approximation as choosing a candidate function that can stand in for the function the learning system wants to represent. A polynomial family supplies several possible candidate shapes. The important introductory question is not whether you can memorize every polynomial family. It is whether a polynomial family is an appropriate approximation tool for the problem at hand.

Interpolation, Regression, and Reinforcement Learning

Interpolation and regression provide a useful way to understand the structure of reinforcement-learning function approximation. In the interpolation and regression setting, function values are available at some samples, and the goal is to define a function between those samples or even beyond them. In reinforcement learning, the corresponding challenge appears over a continuous, multi-dimensional state space: the system must approximate a function across states using limited or sampled information. The transfer is useful because the underlying challenge is related in both settings: a function must be defined from incomplete or sampled information.

Basis Functions Versus Approximation

Function approximation is the broader task: define an approximate function when the desired function is known only through limited or sampled information. Polynomial basis functions belong to the toolset for carrying out that task. They provide a family of candidate functions, but naming a family is not the same as completing the approximation. Keeping these roles separate prevents a common misunderstanding: polynomial basis functions are a possible representation choice, while function approximation is the overall problem being solved.

Choosing the Right Mental Model

• Treating polynomial basis functions as the same thing as function approximation.

  A polynomial family is one family of candidate functions, while function approximation is the broader task of defining an approximate function from limited or sampled information.

  Fix: Describe the polynomial family as a possible tool within the larger approximation process.

• Missing the connection to interpolation and regression.

  The relevance of polynomial families to reinforcement learning comes from this shared approximation structure.

  Fix: Begin with the sampled-information setting, then connect it to approximation over continuous, multi-dimensional state spaces.

• Assuming that continuous, multi-dimensional state spaces do not change the problem.

  The challenge is to approximate a function over the space when information is limited or sampled.

  Fix: Emphasize the need for a function that extends across the state space rather than only recording the observed samples.

• Focusing on memorizing polynomial families instead of recognizing when a family is appropriate.

  The central introductory skill is identifying when a polynomial family is an appropriate approximation tool.

  Fix: First identify the approximation problem and its sampled information; only then consider a candidate family.

Key Takeaways

1. Polynomial basis functions provide one family of candidate functions for approximation.
2. Interpolation and regression involve defining a function from values available at some samples, between those samples or even beyond them.
3. Reinforcement learning encounters a related challenge when it approximates a function over a continuous, multi-dimensional state space.
4. Function approximation is the broader task; selecting a polynomial family is one possible tool within that task.
5. The key introductory skill is recognizing when a polynomial family is an appropriate approximation tool rather than memorizing every family.