Function Approximation and Reinforcement Learning
Function approximation turns examples of a desired function into a broader estimate.
From Limited Experience to a Broader Estimate
A reinforcement-learning system may have only some examples of the function it needs. The central problem is then not simply how to store those examples, but how to use them to estimate the function more broadly. Function approximation addresses this problem by turning examples of a desired function into a broader estimate.
The defining operation is generalization. Function approximation is not simple example storage: it uses available examples to construct a representation intended to cover more of the desired function.
Three Different Objects
It is important to keep three parts of the process separate. The desired function is the function the system needs to represent. The examples are the evidence currently available from that function. The approximation is the broader representation constructed from those examples. In reinforcement learning, the desired function can be a value function.
Separating Evidence from the Estimate
Suppose a reinforcement-learning system has a few examples associated with a value function. Identify what each part represents.
Desired function: The value function is the function the system is trying to represent.
Examples: The available value-related examples are the evidence the system currently has.
Approximation: The broader estimate constructed by generalizing from those examples is the function approximation.
Boundary: The examples are inputs to the process; the approximation is the result produced by generalization.
The approximation is not identical to the examples. It is a broader representation built from them.
Why This Is Supervised Learning
Function approximation is an instance of supervised learning used within reinforcement-learning algorithms. The examples associated with the desired function provide the evidence from which an approximation is constructed. This connection means that reinforcement learning can combine its own methods with established generalization methods rather than requiring an entirely separate approach to generalization.
The word supervised identifies the role of examples in the process: they are associated with the function being represented and are used to construct a broader estimate. The reinforcement-learning setting determines where those examples come from and how the resulting approximation is used.
Inside the Reinforcement-Learning Method
Within a reinforcement-learning algorithm, a function approximator takes the role of the generalization component. Reinforcement-learning methods provide or use examples associated with the desired function, and the approximator constructs a broader estimate. That estimate can represent a value function and can then support the reinforcement-learning method. The exact practical fit depends on the particular approximation method and the particular reinforcement-learning algorithm.
When evaluating a possible approximator, ask two separate questions. First, can the method serve as a function approximator in principle? Second, how naturally does it fit inside the particular reinforcement-learning algorithm? Theoretical suitability does not imply equal practical convenience.
Available Approximation Traditions
The source identifies a broad range of fields that provide possible approximation methods. These include machine learning, artificial neural networks, pattern recognition, and statistical curve fitting. In theory, a method studied in any of these fields can take the role of a function approximator within a reinforcement-learning algorithm.
| Field | Possible contribution |
|---|---|
| Machine learning | General methods for constructing approximations from examples |
| Artificial neural networks | Methods that can be used as function approximators |
| Pattern recognition | Methods that can provide approximation approaches |
| Statistical curve fitting | Methods for fitting a broader representation to examples |
Fields identified as possible sources of function-approximation methods
Mistakes That Blur the Process
Treating function approximation as example storage
The defining operation of function approximation is generalization, not simple example storage.
Fix:
Describe the approximation as a broader representation constructed from available examples.Calling the available examples the desired function
The desired function, the examples, and the approximation are distinct parts of the process.
Fix:
Identify the examples as evidence from the desired function and the approximation as the constructed estimate.Assuming reinforcement learning must invent its own generalization methods
Function approximation is an instance of supervised learning, and methods from several machine-learning and curve-fitting fields can provide approximators.
Fix:
Recognize that reinforcement-learning methods can be combined with existing generalization methods.Assuming every possible approximator fits every algorithm equally well
The practical fit differs across methods and reinforcement-learning algorithms.
Fix:
Separate theoretical possibility from how naturally a method fits a particular algorithm.
Check Your Understanding
A reinforcement-learning system has several examples associated with a desired value function. Explain what function approximation contributes beyond retaining those examples.
Hints
- Name the role of the examples.
- Describe the result produced by generalization.
- State why the result is broader than simple storage.
Classify each item as the desired function, available evidence, or constructed approximation: the value function the system needs to represent; a set of examples associated with that function; a broader estimate built from those examples.
Hints
- The desired function is the target of representation.
- Examples are the evidence available to the learner.
- The approximation is the result of generalization.
A proposed method comes from statistical curve fitting. What two questions should you ask before using it as a function approximator in reinforcement learning?
Hints
- First ask whether it can serve as an approximator in principle.
- Then ask how naturally it fits the particular reinforcement-learning algorithm.
The Central Idea
- Function approximation turns examples of a desired function into a broader estimate.
- In reinforcement learning, the desired function can be a value function.
- The examples are evidence; the approximation is the generalized representation constructed from that evidence.
- Function approximation is an instance of supervised learning used within reinforcement-learning algorithms.
- Possible approximators can come from machine learning, artificial neural networks, pattern recognition, and statistical curve fitting, although practical fit differs across methods.
Key Takeaways
- Function approximation generalizes from limited examples to represent a desired function more broadly.
- The desired function, its available examples, and the constructed approximation are different things.
- In reinforcement learning, the desired function can be a value function.
- Function approximation is supervised learning used within reinforcement-learning algorithms.
- Methods from several related fields can provide approximators, but their practical fit depends on the reinforcement-learning algorithm.