Instrumental Conditioning
Classify conditioning by asking whether reinforcement depends on behavior.
Start with the contingency
The most reliable first question in a conditioning experiment is not what the learner is thinking. It is whether the reinforcing stimulus depends on what the learner does. If the apparatus provides reinforcement without making it contingent on the animal's behavior, the setup is classified as classical conditioning. If reinforcement depends on a particular behavior, the setup is classified as instrumental conditioning.
The distinction is a diagnosis of the experiment's arrangement. Inspect the relationship between behavior and reinforcement before drawing conclusions about the learner's internal process.
Trace the learner's action
In an instrumental-conditioning setup, the learner's behavior is part of the route to reinforcement. The important trace is therefore: the learner acts, the experiment checks whether that action meets the arranged contingency, and reinforcement is delivered or withheld according to that arrangement. In a classical-conditioning setup, the apparatus provides the reinforcing stimulus without making its delivery contingent on the animal's behavior.
Classifying two experimental arrangements
Two hypothetical experiments both present a cue before a reinforcing stimulus. In one, the apparatus provides the reinforcing stimulus whether or not the learner responds. In the other, the apparatus provides it only after the learner performs the arranged behavior. How should the experiments be classified?
Inspect the first arrangement: The reinforcing stimulus does not depend on the learner's behavior. This is the setup-based pattern called classical conditioning.
Inspect the second arrangement: The reinforcing stimulus depends on the learner's behavior. This is the setup-based pattern called instrumental conditioning.
Separate classification from mechanism: The different labels identify different experimental arrangements. They do not, by themselves, prove that the learner uses completely different mechanisms.
The first experiment is classified as classical conditioning and the second as instrumental conditioning. The classification comes from the reinforcement contingency.
Do not overread the labels
A common first interpretation is that classical conditioning and instrumental conditioning must use completely different learning mechanisms. That conclusion goes beyond the distinction itself. The distinction tells us how the experiment is arranged: whether reinforcement is behavior-independent or behavior-dependent. It does not by itself establish a clean separation between mental or computational mechanisms.
Treating the two labels as proof of two completely separate learning mechanisms.
The labels describe how reinforcement depends on behavior. They do not, by themselves, establish a clean separation between underlying learning processes.
Fix:
First classify the experimental setup. Then treat questions about learning mechanisms as a separate question.Classifying an experiment from the presence of a cue alone.
The reliable diagnostic is the relationship between the learner's behavior and reinforcement.
Fix:
Ask whether the reinforcing stimulus is delivered independently of behavior or only after an arranged behavior.
Connect cues to prediction
Classical conditioning is especially relevant to temporal-difference prediction. The important connection is that a learner encounters cues and reinforcing events over time, making the experiment useful for studying how an agent learns to predict future reinforcing events. This connection concerns prediction: estimating what reinforcing event is likely to occur, rather than directly deciding which action will cause it.
Reading a classical-conditioning sequence as prediction
Consider a generated sequence in which a cue is repeatedly followed by a reinforcing event. What kind of learning question does this sequence make useful?
Identify the cue: The cue occurs before the reinforcing event and can therefore be considered in relation to what happens later.
Identify the outcome: The reinforcing event is the future event that the learner may come to predict.
State the prediction question: Ask what future reinforcing event the cue now signals. This is the prediction perspective connected to temporal-difference learning.
The sequence is useful for studying how an agent learns to predict reinforcing events from earlier cues.
The classical-conditioning connection to temporal-difference prediction does not require treating the conditioning label as a complete theory of the learner. It identifies a useful setting for studying prediction of reinforcing events.
Separate prediction from control
Reinforcement learning treats prediction and control as distinct problems. Prediction asks what is likely to happen, such as whether a cue signals a future reinforcing event. Control asks how an agent can choose behavior to influence what happens. The problems are distinct because estimating an outcome is not the same as selecting an action to produce a desired outcome. They can nevertheless be combined in one learning system.
| Question | Problem | Connection to conditioning |
|---|---|---|
| What is likely to happen? | Prediction | Classical-conditioning experiments are closely related to temporal-difference prediction. |
| What behavior should be selected to influence what happens? | Control | Instrumental arrangements make behavior relevant to the delivery of reinforcement. |
| Can both be used together? | Yes | Prediction and control are distinct problems that can be combined. |
Check your diagnosis
A generated experiment delivers a reinforcing stimulus at scheduled times, regardless of whether the learner performs a response. A second generated experiment delivers the reinforcing stimulus only after the learner performs the arranged response. Classify each setup, then state whether the classifications alone prove that the experiments use completely different learning mechanisms.
Hints
- Focus on whether reinforcement depends on behavior.
- Keep the experimental label separate from the question about underlying mechanisms.
What do you think happens?
What is the correct classification of the two generated setups?
Reveal answer
Answer: The first is classical conditioning and the second is instrumental conditioning.
The first setup provides reinforcement independently of behavior. The second makes reinforcement depend on the learner's behavior. These labels describe the experimental arrangements; they do not by themselves prove completely different learning mechanisms.
When diagnosing a conditioning experiment, write down the contingency in plain language: reinforcement occurs independently of behavior, or reinforcement occurs only after the arranged behavior. This makes the classification explicit and avoids guessing the learner's internal process from the label.
Key takeaways
- Instrumental conditioning is identified by behavior-dependent reinforcement.
- Classical conditioning is identified, in this setup-based distinction, by reinforcement that is not contingent on the animal's behavior.
- The distinction classifies an experimental arrangement; it does not prove that the two arrangements use completely separate learning mechanisms.
- Classical conditioning is closely related to temporal-difference prediction because it provides a useful setting for studying prediction of reinforcing events.
- Prediction and control are distinct reinforcement-learning problems, although they can be combined.
Key Takeaways
- Classify conditioning by asking whether reinforcement depends on behavior.
- Treat classical and instrumental conditioning as descriptions of experimental setups, not automatic proof of separate learning mechanisms.
- Use classical-conditioning experiments to understand the connection between cues and temporal-difference prediction.
- Keep prediction and control conceptually distinct while recognizing that reinforcement-learning systems can combine them.