Secondary Reward

From Event to Signal

Suppose an animal works to obtain food. In ordinary language, the food may be called a reward. In reinforcement learning, however, the learning system does not directly manipulate the food as its mathematical input. It receives a numerical evaluation, such as R_t, that scores what happened from the agent's learning perspective. That number may be positive, negative, or zero.

Behavioral Meaning of Reward

In the context of animal behavior, an object or event counts as a reward when the animal tends to approach it and work to obtain it. This definition focuses on observable behavior rather than on whether people describe the object as pleasant, valuable, or a prize. The classification depends on what the animal does.

Primary and Secondary Origins

Primary and secondary rewards describe two different ways something can become rewarding. A primary reward comes from machinery built into an animal's nervous system through evolution. Primary rewards are therefore called innate. They are connected with outcomes that supported survival, reproduction, or predicted reproductive success across ancestral history.

A secondary reward acquires its rewarding quality during an individual's learning. The animal learns that a stimulus or event predicts a primary reward. A stimulus can also become secondary by predicting another learned reward. Its value is not supplied with the same built-in status as a primary reward; it is acquired through prediction.

Acquiring Secondary Value

A previously neutral stimulus can become a secondary reward when an individual learns that it predicts a primary reward. The important change is not necessarily a change in the stimulus itself. The change is in what the stimulus predicts for that individual.

Reinforcement Directs Learning

A reinforcement signal is defined by its instructional role. It is the quantity that most directly directs changes in the agent's policy or value function in response to the current situation. In parameterized systems, it multiplicatively modulates parameter updates.

The reward signal and reinforcement signal can coincide. If R_t is the critical multiplier in the parameter-update equation, then R_t is the reinforcement signal at time t. In other algorithms, the update-driving quantity contains more than the immediate reward.

Reading the TD Error

TD error = R_t + [γV(S_t) − V(S_{t−1})]

The TD error is therefore not necessarily just the reward received at the current moment. It can include an immediate reward contribution and a contribution from how predicted value changes. Because both terms can appear together, the TD error often represents a mixture of reinforcement sources.

A Careful Neuroscience Interpretation

The comparison between reinforcement signals and internal influences on decision making is useful, but it should not be taken too literally. A reward signal may be associated with something perceived in the outside world, yet internal experiences such as memories, ideas, or hallucinations can also trigger such a signal.

The mathematical R_t should therefore be treated as an abstraction for the combined effects of multiple neural signals. It is not proof that the brain contains one single signal identical to the mathematical quantity R_t.

Common Classification Errors

• Treating the reward signal as the physical reward itself.

  R_t is a numerical evaluation, while the physical object or event is what the animal may approach and work to obtain.

  Fix: Describe the physical object or event separately from the numerical signal assigned to it.

• Defining a reward only as something pleasant.

  The behavioral definition asks whether the animal approaches the object or event and works to obtain it.

  Fix: Use the animal's approach and effort as the relevant behavioral evidence.

• Calling every positive number a reinforcement signal.

  The defining property of a reinforcement signal is its instructional role in directing updates, not whether its sign is positive, negative, or zero.

  Fix: Ask which quantity most directly modulates the learning update.

• Calling money primary because people value it.

  The source identifies money as secondary because its rewarding quality is learned through prediction.

  Fix: Classify money as secondary and state that its value is acquired through a predictive relationship with primary or other secondary rewards.

• Treating the TD error as only the immediate reward.

  The TD error combines R_t with γV(S_t) − V(S_{t−1}).

  Fix: Interpret the TD error as a mixture of immediate and prediction-based contributions.

Practice Classification

Key Takeaways

1. A reward is an object or event that an animal approaches and works to obtain; a reward signal is a numerical evaluation.
2. Primary rewards are innate because their rewarding quality comes from nervous-system machinery shaped by evolution.
3. Secondary rewards acquire value through individual learning because they predict primary rewards or other learned rewards.
4. A reinforcement signal is the quantity that most directly drives changes to a policy or value function.
5. The TD error combines immediate reward with a change in predicted value and can therefore act as a mixed reinforcement signal.