Planning Backups
State-space planning can be understood as sequences of backups.
Planning as a Sequence
State-space planning can be understood as a sequence of backups. Instead of treating a planning method as a single indivisible procedure, examine it one backup at a time. Each backup performs an update, and the planning strategy also determines which state receives attention next.
Two Planning Decisions
The phrase planning backup contains two separate decisions. First, what kind of backup is performed? This is the computation dimension: the update made during a backup. Second, which state receives attention next? This is the ordering dimension: the sequence in which backups are directed toward states. Two planning strategies may therefore differ because they perform different kinds of backups, because they order attention differently, or because they differ along both dimensions.
Separating Update from Attention
Imagine two planning procedures that use the same kind of backup but choose states in different orders.
Identify the backup type: Ask what update each procedure performs. This identifies the computation dimension.
Identify the backup order: List which state receives attention first, second, and later. This identifies the ordering dimension.
Compare the procedures: If the update is the same but the state order differs, the strategies differ in backup order rather than backup type.
A planning strategy is easier to analyze when its update rule and its attention order are examined separately.
Backward Focusing
Backward focusing is one way to organize attention during planning. The central idea is to begin with a target state or outcome and direct attention toward states that lead backward from it. In a conceptual sequence, the target receives attention first, followed by relevant predecessor states, with the order continuing according to the chosen backward organization.
Forward Focusing
Forward focusing organizes attention according to reachability from frequently visited states under the current policy. The starting point is therefore connected to the behavior currently being followed: states visited frequently under that policy receive importance, and other states are prioritized according to how reachable they are from those frequently visited states.
Following Reachability
Suppose a current policy frequently visits state A and can reach states B and C from states that are frequently visited. A different state, D, is not identified as reachable from those frequently visited states.
Find the reference states: Use the states frequently visited under the current policy as the starting points for attention.
Consider reachability: Compare other states according to whether they are reachable from those frequently visited states.
Order attention: Prioritize reachable states such as B and C rather than treating every state as equally connected to the policy's frequently visited states.
This illustrates the organizing principle of forward focusing: policy-based frequent visitation supplies the reference, and reachability supplies the basis for prioritizing attention.
Two Ways to Order Attention
Backward and forward focusing answer the same broad planning question—where should attention go next—but use different reference points. Backward focusing organizes attention by looking from a target or outcome toward predecessor states. Forward focusing organizes attention from frequently visited states under the current policy and uses reachability to prioritize other states.
| Dimension | Backward focusing | Forward focusing |
|---|---|---|
| Reference point | A target state or outcome | Frequently visited states under the current policy |
| Ordering idea | Organize attention toward predecessor states | Prioritize states according to reachability |
| Role in planning | One way to organize backup attention | One way to organize backup attention |
| What it does not determine by itself | The backup type | The backup type |
Common Reasoning Errors
Treating backup type and backup order as the same thing.
A planning strategy includes both the kind of backup being performed and the order in which backups receive attention.
Fix:
Analyze the update computation and the attention sequence separately.Assuming backward focusing is the definition of planning.
Backward focusing is presented as one way to organize attention during planning.
Fix:
Treat backward focusing as one possible ordering strategy and compare it with forward focusing.Describing forward focusing as simply moving forward through every state.
Forward focusing selects attention according to reachability from frequently visited states under the current policy.
Fix:
Identify the policy-based frequently visited states first, then consider reachability from them.Using the names of methods without examining their underlying operations.
The sequence-of-backups perspective directs attention beneath individual method names.
Fix:
Describe each strategy in terms of backup type and backup order.
Practice the Distinction
A planning description says: “The method uses a particular backup update, begins by considering states frequently visited under the current policy, and then prioritizes other states according to reachability from those states.” Identify the backup type dimension and the backup order dimension. Which focusing strategy does the ordering description resemble?
Hints
- The backup type is identified by asking what update the method performs.
- The backup order is identified by asking which states receive attention and how they are prioritized.
- Frequent visitation under the current policy followed by reachability describes forward focusing.
- To analyze the description, separate the update from the ordering. The particular backup update belongs to the backup type dimension. Starting from frequently visited states under the current policy and prioritizing according to reachability belongs to the backup order dimension and resembles forward focusing.
Practical Reading Strategy
When encountering a new planning method, read beneath its name by asking two questions. What kind of backup is being performed? Which state receives attention next, and why? This method of analysis keeps computation and attention distinct. It also makes it possible to compare methods without assuming that one ordering strategy is the only valid approach.
Key Takeaways
- State-space planning can be understood as a sequence of backups.
- A planning strategy has two important dimensions: the kind of backup performed and the order in which backups receive attention.
- Backward focusing organizes attention from a target or outcome toward predecessor states.
- Forward focusing prioritizes states according to reachability from frequently visited states under the current policy.
- Backward and forward focusing are alternative ways to organize attention, not a reason to ignore the separate backup type dimension.