Uncertainty in Environment
Goal-directed agents pursue explicit goals through sensing and action.
Why Uncertainty Matters
A reinforcement-learning agent is not merely solving a detached calculation. It is a goal-directed system that senses what is happening, selects actions, and continues operating even when the environment is not completely predictable. The central difficulty is that each action leads to a new situation that must be sensed and evaluated again.
A goal-directed agent is an agent that pursues an explicit goal through sensing and action.
The environment is uncertain when the agent cannot treat its future situation as completely known or predictable. The agent must therefore keep sensing, choosing, and responding as interaction continues.
The Decision Loop
The agent's behavior can be understood as a loop. An explicit goal gives the agent direction. Sensing provides information about the current situation. Action selection uses that information to determine a response. The selected action influences the environment, producing a new situation for the next sensing step.
Tracing One Interaction
A Goal-Seeking Agent in an Uncertain Situation
Consider a generated example in which an agent has an explicit goal and must continue operating in an environment whose next situation is not completely predictable.
Set direction: The explicit goal gives the agent a direction for its behavior. Without that goal, sensing and action selection would not be connected to a stated purpose.
Sense: The agent receives information about aspects of the current environment. This information is the basis for its next response.
Select an action: The agent chooses an action using what it can sense while pursuing the goal.
Face uncertainty: The action does not guarantee one completely known future situation. The agent must be prepared for the resulting situation to require another decision.
Continue the loop: The action affects the environment. The agent then senses the resulting situation and selects a further action.
The agent's behavior is an ongoing goal-directed interaction: goal, sensing, action selection, environmental change, and renewed sensing.
This example does not require a particular number of possible outcomes. The important idea is that the agent cannot assume that the environment is fully predictable after every action. It must keep interacting and making decisions under incomplete certainty.
The Whole-Agent Perspective
Reinforcement learning focuses on complete, interactive agents operating in uncertain environments. This does not mean that reinforcement learning ignores individual capabilities. Planning, prediction, and other capabilities can matter, but they matter because of their roles in the larger agent's interaction with the environment.
| View | Main focus | Relationship to reinforcement learning |
|---|---|---|
| Isolated subproblem | One detached capability, such as prediction or planning | Can be useful, but does not by itself describe the complete interaction |
| Whole-agent perspective | A goal-seeking agent senses, selects actions, and continues operating | Includes the connected interaction with an uncertain environment |
Agents Inside Larger Systems
The word complete refers to the agent's interaction problem, not necessarily to an entire organism or an entire robot. A component inside a robot or another behaving system can still be treated as a complete agent if it directly interacts with the rest of that larger system.
In this arrangement, the component interacts directly with the larger system and indirectly with the larger system's environment. The component's boundaries are therefore defined by the interaction being studied.
Uncertainty Changes Decisions
Uncertainty does not remove the goal or the need to act. Instead, it makes action selection an ongoing control problem. The agent must use what it can sense, choose an action, and continue from the situation that follows rather than acting as if the environment were fully known.
What do you think happens?
If an agent takes an action in an uncertain environment, should it assume that exactly one completely known situation will follow?
Reveal answer
Answer: No, because the agent must continue interacting without treating the environment as fully known.
The uncertainty is represented by the possibility of different resulting situations. The agent must sense the resulting situation and make another decision.
Common Misunderstandings
Treating reinforcement learning as mainly one detached task.
Prediction and planning can be useful capabilities, but the broader focus is a complete agent interacting with an uncertain environment.
Fix:
Connect every capability to the agent's goal-directed sensing and action loop.Treating the agent as a passive observer.
The agent also sends actions that influence the environment.
Fix:
Track both directions: the environment provides aspects to sense, and the agent sends actions that affect what happens next.Assuming that uncertainty means the agent cannot act.
Reinforcement learning concerns agents that must operate despite significant uncertainty.
Fix:
Use available sensing, select an action, and continue from the resulting situation.Assuming that a component cannot be a complete agent.
Complete refers to the interaction problem being studied, not necessarily to an entire organism or robot.
Fix:
Identify the component's direct interaction with the larger system and its indirect relationship with the larger environment.
Check Your Understanding
Explain why reinforcement learning is broader than solving an isolated prediction or planning task. In your answer, describe the roles of the goal, sensing, action selection, and uncertainty.
Hints
- Start with the explicit goal that gives the agent direction.
- Explain what information sensing provides.
- Explain how the selected action changes the environment and why another decision is needed.
A component operates inside a larger behaving system. What would you need to identify before treating that component as a complete agent for an interaction problem?
Hints
- Focus on the component's direct interaction with the rest of the larger system.
- Also identify how it relates indirectly to the larger system's environment.
Key Takeaways
- A goal-directed agent pursues an explicit goal through sensing and action.
- Reinforcement learning studies complete, interactive agents operating in uncertain environments.
- Goals, sensing, action selection, and environmental change form a connected loop.
- The agent participates in the problem because its actions influence the environment.
- A component inside a larger behaving system can still be treated as a complete agent for its interaction problem.
Key Takeaways
- A goal-directed agent uses sensing and action to pursue an explicit goal.
- Environmental uncertainty means the agent cannot assume that the result of an action is completely known.
- Reinforcement learning takes a whole-agent view of the continuing interaction rather than isolating prediction or planning.
- The environment affects the agent through sensing, and the agent affects the environment through selected actions.
- An agent can be a component inside a larger behaving system while remaining complete for the interaction problem being studied.