Spike-Timing-Dependent Plasticity

Why Timing Matters

A synapse does not necessarily change simply because neural activity occurred. In spike-timing-dependent plasticity, or STDP, the relationship between two events matters: an incoming presynaptic spike and a later postsynaptic spike. The order of those events helps determine whether the synapse becomes stronger or weaker.

Reading the STDP Rule

Suppose a synapse carries an incoming spike toward a postsynaptic neuron. In the most studied form of STDP described in the source, the synapse is strengthened when the presynaptic spike arrives shortly before the postsynaptic neuron fires. The interpretation is that the incoming activity has the right temporal relationship to the postsynaptic event. If the order is reversed, so that the postsynaptic neuron fires first and the presynaptic spike arrives shortly afterward, the synapse is weakened. The size of the change also depends on the relative timing, not only on which event came first.

Adding the Third Factor

Reward-modulated STDP extends the timing rule with a third factor: neuromodulatory input. Presynaptic and postsynaptic spikes can create a potential synaptic change, but that potential does not become a lasting modification unless a suitable neuromodulatory signal arrives within a relevant time window afterward.

The three factors are presynaptic activity, postsynaptic activity, and neuromodulation. The first two establish the timing relationship. The third determines whether that potential change is consolidated as a lasting synaptic modification. In the dorsal striatum, the neuromodulatory factor described in the source is dopamine, and the relevant synaptic sites are the spines of medium spiny neurons.

From Spikes to Eligibility

The timing sequence can be understood as a temporary record followed by a later evaluation. First, presynaptic and postsynaptic spikes occur in a relationship that STDP considers relevant. That relationship creates a potential synaptic modification, often described in the source as an eligibility trace. A later neuromodulatory signal can then convert that temporary eligibility into a lasting synaptic change if it arrives within the relevant time window.

Actor and Critic Connections

The actor in an actor-critic system changes the tendencies that produce actions. STDP supplies one ingredient for actor-like change because it can select synapses according to their recent contribution to a postsynaptic firing event. Reward-modulated STDP adds outcome sensitivity: a later neuromodulatory signal can determine whether a timing-related synaptic change becomes lasting.

The connection to reinforcement learning is especially close in the actor unit of an actor-critic network. The source states that the actor unit's learning rule closely corresponds to reward-modulated STDP, and that the relevant biological discussion includes corticostriatal synapses. This is a correspondence between a learning rule and a biological mechanism; ordinary STDP and the actor unit are not identical.

Evidence and Its Limits

The historical connection begins with A. H. Klopf's hedonistic neuron hypothesis, which suggested that neurons behave in a hedonistic manner by seeking to maximize their activity. The hypothesis influenced the actor-critic network of Barto and colleagues in 1983, whose neuron-like actor unit used a Law-of-Effect-style learning rule.

• Treating STDP as a rule that strengthens every active synapse.

  STDP depends on the relative timing and order of presynaptic and postsynaptic activity. Reversing the order can weaken the synapse.

  Fix: Ask which spike occurred first and how closely the two spikes were timed.

• Calling ordinary STDP a complete reward-learning rule.

  The source states that actor-like outcome-sensitive learning also requires a neuromodulatory factor such as dopamine.

  Fix: Distinguish two-factor timing conditions from reward-modulated STDP's additional neuromodulatory factor.

• Assuming a suitable spike pattern guarantees lasting change under reward-modulated STDP.

  The source reports that lasting corticostriatal changes required a neuromodulatory pulse within a relevant later time window.

  Fix: Separate the temporary eligibility created by spike timing from the later signal that can make the change lasting.

• Treating the actor-unit correspondence as proof of a biological actor-critic algorithm.

  The source describes the evidence as supporting biological plausibility, not establishing a complete algorithmic implementation.

  Fix: Use cautious language such as supports, corresponds to, or makes plausible.

• Assuming every detail of the hedonistic neuron hypothesis remains accepted.

  The source states that not all details of the proposal match what has since been learned.

  Fix: Treat the hypothesis as an important historical influence while separating it from modern evidence.

Apply the Distinction

Key Takeaways

1. STDP makes synaptic change depend on the order and relative timing of presynaptic and postsynaptic spikes.
2. In the most studied STDP form described here, presynaptic activity shortly before postsynaptic firing strengthens a synapse, while the reversed order weakens it.
3. Reward-modulated STDP is a three-factor process: presynaptic activity, postsynaptic activity, and later neuromodulation such as dopamine.
4. Spike timing can create an eligibility trace, while a later neuromodulatory signal can help convert that potential change into lasting synaptic modification.
5. Reward-modulated STDP makes actor-like, outcome-sensitive synaptic learning biologically plausible, but it does not prove that the brain implements a complete actor-critic algorithm.