CoursesMachine Learning
Machine LearningIntermediateSelf-paced
Reinforcement Learning
This course covers the fundamentals and advanced topics in Reinforcement Learning, including tabular and approximate solution methods, and applications. It provides a comprehensive understanding of the Reinforcement Learning problem and various techniques to solve it. Students will learn about key concepts such as Markov Decision Processes, Dynamic Programming, and Policy Gradient Methods.
0chapters
0modules
0lessons
0quizzes
Syllabus
133 chapters, 350 modules, 797 lessons and quizzes.
01Getting Started1 modules · 2 lessons
02Understanding Reinforcement Learning Limitations and Scope3 modules · 6 lessons
Alternative Approaches to Reinforcement Learning
Optimization and Its Challenges
Responsible Deployment of Reinforcement Learning
03The k-Armed Bandit Problem: Balancing Exploration and Exploitation2 modules · 5 lessons
Understanding the k-Armed Bandit Problem
Exploration vs Exploitation
04Efficient Action-Value Estimation2 modules · 4 lessons
05Handling Nonstationary Problems in Reinforcement Learning3 modules · 6 lessons
Nonstationary Problems and Weighted Averages
Step-Size Parameters and Convergence
Sample-Average Method and Nonstationarity
06Optimistic Initial Values for Exploration2 modules · 5 lessons
Understanding Optimistic Initial Values
07Balancing Exploration and Exploitation with Upper-Confidence-Bound Action Selection2 modules · 5 lessons
Understanding Upper-Confidence-Bound Action Selection
Performance and Limitations of UCB Action Selection
08Gradient Bandit Algorithms for Action Selection2 modules · 6 lessons
Introduction to Gradient Bandit Algorithms
Theoretical Foundations of Gradient Bandit Algorithms
09Defining Goals and Rewards in Reinforcement Learning2 modules · 4 lessons
Understanding Reward Signals
The Agent-Environment Interface for Rewards
10Understanding Returns in Reinforcement Learning4 modules · 7 lessons
Defining Returns
Episodic and Continuing Tasks
Discounting in Continuing Tasks
Applying Returns to Real-World Tasks
11Unified Notation for Episodic and Continuing Tasks2 modules · 4 lessons
Unified Notation for Episodic and Continuing Tasks
12Markov Decision Processes and Value Functions2 modules · 7 lessons
Markov Decision Processes
13Understanding Notation in Reinforcement Learning3 modules · 7 lessons
Notation Conventions
Notation in Bandit Problems
Notation in Markov Decision Processes
14Optimal Value Functions in Reinforcement Learning4 modules · 9 lessons
Defining Optimal Policies and Value Functions
Bellman Optimality Equations
Solving for Optimal Policies
Practical Considerations and Examples
15Optimality and Approximation in Reinforcement Learning2 modules · 4 lessons
Challenges of Optimality
16Reinforcement Learning Essentials4 modules · 9 lessons
Reinforcement Learning Problem Overview
Value Functions and Optimality
Knowledge and Approximation
Bibliographical and Historical Remarks
17Evaluating Policies through Iterative Computation3 modules · 7 lessons
Understanding Policy Evaluation
Iterative Policy Evaluation Algorithm
18Generalized Policy Iteration: A Unified Framework2 modules · 4 lessons
Understanding Generalized Policy Iteration
GPI: Competing and Cooperating Processes
19Dynamic Programming Essentials2 modules · 8 lessons
Key Concepts in Dynamic Programming
20Estimating Action Values with Monte Carlo Methods2 modules · 5 lessons
Understanding Action-Value Estimation
Maintaining Exploration in Monte Carlo Methods
21Monte Carlo Control Methods3 modules · 6 lessons
Introduction to Monte Carlo Control
Practical Considerations for Monte Carlo Control
Application of Monte Carlo ES to Blackjack
22Monte Carlo Methods Summary3 modules · 7 lessons
Advantages and Characteristics of Monte Carlo Methods
Exploration and Off-Policy Learning
Bibliographical and Historical Remarks
23Advantages of Temporal Difference Prediction Methods2 modules · 6 lessons
Advantages Over Monte Carlo and DP Methods
24Understanding TD(0) Optimality3 modules · 5 lessons
Example: You are the Predictor
Batch Updating and Optimality
Certainty-Equivalence Estimate
25Expected Sarsa: Improving TD Control2 modules · 5 lessons
Understanding Expected Sarsa
Expected Sarsa in Practice
26Special Cases in Reinforcement Learning: Afterstates and Games2 modules · 5 lessons
Understanding Afterstates
Applying Afterstates in Various Tasks
27Temporal-Difference Learning Methods: n-step and Off-policy Learning2 modules · 5 lessons
Summary of n-step Methods
Bibliographical and Historical Remarks
28Understanding Models and Planning in Reinforcement Learning2 modules · 7 lessons
Models and Planning Fundamentals
State-Space Planning and Random-Sample Q-Planning
29Integrating Planning and Learning with Dyna3 modules · 5 lessons
Introduction to Dyna Architecture
Components and Processes of Dyna
Example and Benefits of Dyna
30Dealing with Modeling Errors in Planning3 modules · 5 lessons
Consequences of Modeling Errors
Examples of Modeling Errors and Recovery
Addressing Exploration-Exploitation Trade-off
31Prioritized Sweeping for Efficient Planning3 modules · 9 lessons
Introduction to Prioritized Sweeping
Prioritized Sweeping Algorithm
32Planning in Action Selection2 modules · 4 lessons
Understanding Planning in Action Selection
Trade-offs and Applications
33Heuristic Search in Reinforcement Learning2 modules · 5 lessons
Introduction to Heuristic Search
Benefits and Trade-offs of Heuristic Search
34Monte Carlo Tree Search for Planning3 modules · 8 lessons
Introduction to Monte Carlo Tree Search
How Monte Carlo Tree Search Works
MCTS Algorithm Details
35Integrating Planning and Learning2 modules · 6 lessons
Planning and Learning Integration
Bibliographical and Historical Remarks
36Approximate Solutions for Large State Spaces3 modules · 6 lessons
Introduction to Approximate Solution Methods
Function Approximation and Reinforcement Learning
Overview of Part II Chapters
37Introduction to Value-Function Approximation2 modules · 4 lessons
Understanding Backups in Value Prediction
Requirements for Function Approximation in Reinforcement Learning
38Defining the Prediction Objective in Reinforcement Learning3 modules · 7 lessons
Understanding the Mean Squared Value Error (MSVE)
On-Policy Distribution in Episodic Tasks
Optimizing MSVE and Function Approximation Methods
39Feature Construction for Linear Function Approximation5 modules · 15 lessons
Polynomial Basis Functions
Fourier Basis Functions
Coarse Coding
Tile Coding
40Nonlinear Function Approximation with Artificial Neural Networks4 modules · 9 lessons
Introduction to Artificial Neural Networks
Properties and Capabilities of ANNs
Training ANNs
41Generalization and Function Approximation in Reinforcement Learning3 modules · 8 lessons
Function Approximation in Reinforcement Learning
Methods for Function Approximation
Bibliographical and Historical Remarks
42Stochastic and Semi-gradient Methods for Value Prediction3 modules · 7 lessons
Stochastic Gradient Descent for Value Prediction
Gradient Monte Carlo Algorithm
Semi-gradient TD(0) for Value Prediction
43Episodic Semi-gradient Control Methods2 modules · 5 lessons
Introduction to Episodic Semi-gradient Control
44Limitations of Discounted Setting in Continuing Problems2 modules · 3 lessons
Futility of Discounting in Continuing Problems
Alternative Objectives in Continuing Problems
45Extending Off-Policy Algorithms with Semi-Gradient Methods2 modules · 6 lessons
Introduction to Semi-Gradient Methods
46Understanding Instability in Semi-Gradient Methods2 modules · 5 lessons
Baird's Counterexample to Semi-Gradient TD(0)
Stability Issues with Other Methods
47Understanding the Deadly Triad in Reinforcement Learning1 modules · 3 lessons
48Online Forward View for λ-Return Algorithm3 modules · 6 lessons
Online λ-Return Algorithm Implementation
Introduction to Online λ-Return Algorithm
Computational Complexity and Performance
49Dutch Traces in Monte Carlo Learning2 modules · 5 lessons
Introduction to Dutch Traces in Monte Carlo Learning
Linear MC Algorithm and Dutch Traces
50Understanding Policy Gradient Theorem2 modules · 6 lessons
Policy Gradient Theorem Explained
Proof of Policy Gradient Theorem
51REINFORCE: A Monte Carlo Policy Gradient Method2 modules · 5 lessons
REINFORCE Algorithm and Properties
52Actor-Critic Methods for Reinforcement Learning3 modules · 4 lessons
Introduction to Actor-Critic Methods
One-Step Actor-Critic Methods
Actor-Critic with Eligibility Traces
53REINFORCE with Baseline: Reducing Variance in Policy Gradient Methods2 modules · 5 lessons
Understanding REINFORCE with Baseline
REINFORCE with Baseline Algorithm
54Policy Gradient Methods for Continuing Problems3 modules · 5 lessons
Policy Gradient for Continuing Problems
Actor-Critic Methods with Eligibility Traces
Proof of the Policy Gradient Theorem
55Policy Parameterization for Continuous Actions1 modules · 4 lessons
Representing Policies for Continuous Actions
56Beyond the Basics: Exploring Reinforcement Learning Frontiers3 modules · 6 lessons
Introduction to Advanced Reinforcement Learning Topics
Reinforcement Learning and Interdisciplinary Connections
57Understanding Reinforcement Learning Terminology1 modules · 6 lessons
Key Terms in Reinforcement Learning
- LessonReward Signals: What the Agent Receives and What Learning UsesEnroll to open
- LessonPrimary and Secondary RewardsEnroll to open
- LessonReinforcement Signal and its ComponentsEnroll to open
- LessonTerminology Differences between Psychology and Reinforcement LearningEnroll to open
- LessonClassical and Instrumental ConditioningEnroll to open
- QuizReinforcement Learning Terminology QuizEnroll to open
58Evolution of Reinforcement Learning3 modules · 6 lessons
Foundations of Reinforcement Learning
Trial-and-Error Learning
Temporal-Difference Learning
59Introduction to Reinforcement Learning5 modules · 10 lessons
What is Reinforcement Learning?
Reinforcement Learning vs Other Machine Learning Paradigms
The Exploration-Exploitation Dilemma
Characteristics of Reinforcement Learning Agents
60Reinforcement Learning Through Real-World Examples3 modules · 8 lessons
Understanding Agent-Environment Interaction
Exploring Goal-Oriented Behavior
61Key Components of Reinforcement Learning1 modules · 5 lessons
Understanding Reinforcement Learning Components
62Tic-Tac-Toe: A Reinforcement Learning Example3 modules · 7 lessons
Introduction to the Tic-Tac-Toe Problem
Value Function Approach to Tic-Tac-Toe
Comparing Methods and Generalizing to Other Problems
63Temporal Difference Prediction Methods3 modules · 5 lessons
Introduction to TD Prediction
Tabular TD(0) for Estimating Value Functions
Advantages and Example of TD Methods
64Reward Prediction Error Hypothesis Explained2 modules · 5 lessons
Understanding the Reward Prediction Error Hypothesis
65Reinforcement Learning Summary1 modules · 4 lessons
66Bibliographical Remarks and Further Reading2 modules · 6 lessons
Further Reading on Reinforcement Learning
67Associative Search and Contextual Bandits1 modules · 3 lessons
Introduction to Associative Search
68Understanding the Agent-Environment Interaction2 modules · 5 lessons
Agent-Environment Interaction Basics
Agent-Environment Boundary and Policy
69Asynchronous Dynamic Programming for Efficient Reinforcement Learning2 modules · 5 lessons
Introduction to Asynchronous Dynamic Programming
Asynchronous Value Iteration and Real-time Interaction
70Efficiency of Dynamic Programming Methods1 modules · 4 lessons
71Temporal-Difference Learning Summary4 modules · 7 lessons
Introduction to TD Learning
TD Learning for Prediction and Control
TD Learning Algorithms
Bibliographical and Historical Remarks
72Q-learning: Mastering Off-Policy TD Control3 modules · 5 lessons
Introduction to Q-learning
Q-learning Algorithm and Backup Diagram
Comparing Sarsa and Q-learning
73Understanding Maximization Bias and Double Learning2 modules · 5 lessons
Maximization Bias
Double Learning and Double Q-Learning
74n-Step Sarsa for Control and Prediction3 modules · 4 lessons
Introduction to n-Step Sarsa
n-Step Sarsa Algorithm
n-Step Sarsa for Control and Prediction
75Off-policy Learning with n-step Tree Backup3 modules · 6 lessons
Introduction to n-step Tree Backup Algorithm
Mathematical Formulation of n-step Tree Backup
n-step Tree Backup for Estimating Q
76n-Step Differential Semi-gradient Sarsa2 modules · 4 lessons
Generalizing to n-Step Bootstrapping
Differential Semi-gradient n-Step Sarsa Algorithm
77n-Step Semi-Gradient Sarsa for Control3 modules · 5 lessons
n-Step Semi-Gradient Sarsa
Performance and Bootstrapping
Episodic Semi-Gradient n-Step Sarsa
78Average Reward Setting for Continuing Tasks2 modules · 6 lessons
Understanding Average Reward Setting
Differential Semi-gradient Sarsa for Control
79Average Reward Formulation and Semi-gradient Methods2 modules · 5 lessons
Summary of Key Concepts
Bibliographical and Historical Remarks
80Prediction and Control in Reinforcement Learning2 modules · 5 lessons
Introduction to Prediction and Control
81Classical Conditioning: Learning Predictive Relationships4 modules · 7 lessons
Foundations of Classical Conditioning
The Rescorla-Wagner Model
The TD Model of Classical Conditioning
TD Model Simulations and Applications
82Instrumental Conditioning and Reinforcement Learning4 modules · 9 lessons
Introduction to Instrumental Conditioning
Instrumental Conditioning and Reinforcement Learning
Shaping and Motivation
Delayed Reinforcement
83Understanding Delayed Reinforcement in Learning3 modules · 7 lessons
The Problem of Delayed Reinforcement
Stimulus Traces and Secondary Reinforcement
Actor-Critic Architecture and Delayed Reinforcement
84n-Step TD Prediction Methods3 modules · 6 lessons
Introduction to n-Step TD Methods
n-Step TD Algorithm and Error Reduction Property
Performance of n-Step TD Methods
85Unifying n-step Action-Value Backups with Q(σ)2 modules · 5 lessons
Introduction to n-step Q(σ)
86Linear Methods for Function Approximation3 modules · 6 lessons
Linear Function Approximation Basics
Convergence of Linear TD(0)
n-step Semi-Gradient TD Algorithm
87Understanding the λ-return in Reinforcement Learning2 modules · 5 lessons
Introduction to λ-return
Properties and Applications of λ-return
88Tabular Solution Methods in Reinforcement Learning4 modules · 8 lessons
Introduction to Tabular Solution Methods
Problem Formulations in Tabular Solution Methods
Methods for Solving Finite Markov Decision Processes
Combining Solution Methods
89Action-Value Methods for Decision Making3 modules · 5 lessons
Estimating Action Values
Action Selection Methods
Comparative Performance of Action Selection Methods
90Balancing Exploration and Exploitation3 modules · 8 lessons
Methods for Balancing Exploration and Exploitation
Comparison of Bandit Algorithms
Advanced Topics and Historical Remarks
91Introduction to Monte Carlo Prediction4 modules · 9 lessons
Understanding Monte Carlo Prediction
First-Visit MC Policy Evaluation
Example Applications
Backup Diagrams and Independence
92Monte Carlo Control without Exploring Starts2 modules · 4 lessons
On-policy Monte Carlo Control Methods
On-policy First-visit MC Control
93Policy Improvement Techniques3 modules · 6 lessons
Understanding Policy Improvement Theorem
Greedy Policy Improvement
Policy Improvement for Stochastic Policies
94Policy Iteration for Optimal Decision Making3 modules · 8 lessons
Policy Iteration Fundamentals
Policy Iteration Algorithm and Example
95Value Iteration for Optimal Policies3 modules · 6 lessons
Introduction to Value Iteration
Value Iteration Algorithm and Example
Relationship to Bellman Optimality Equation
96Off-policy Prediction with Importance Sampling3 modules · 6 lessons
Introduction to Off-policy Learning
Importance Sampling for Off-policy Prediction
Examples and Challenges in Off-policy Learning
97Incremental Monte Carlo Methods2 modules · 5 lessons
Incremental Implementation of Monte Carlo Methods
Off-policy Every-Visit Monte Carlo Policy Evaluation
98Off-Policy Monte Carlo Control Methods3 modules · 5 lessons
Introduction to Off-Policy Monte Carlo Control
Off-Policy Every-Visit Monte Carlo Control
Return-Specific Importance Sampling
99Sarsa: On-Policy TD Control for Reinforcement Learning3 modules · 6 lessons
Introduction to Sarsa
Sarsa Control Algorithm
100Off-policy Learning with n-step Methods2 modules · 5 lessons
Introduction to Off-policy n-step Learning
101Least-Squares TD: Efficient Linear TD(0)3 modules · 6 lessons
Introduction to Least-Squares TD
LSTD Algorithm and Complexity
Comparison with Semi-Gradient TD
102TD(λ) Algorithm and Eligibility Traces4 modules · 6 lessons
Introduction to TD(λ)
Eligibility Traces in TD(λ)
TD(λ) Update Rule and Special Cases
Performance Comparison
103True Online TD(λ) for Efficient Learning3 modules · 5 lessons
Introduction to True Online TD(λ)
True Online TD(λ) Algorithm
Eligibility Traces in True Online TD(λ)
104Policy Approximation and its Benefits2 modules · 4 lessons
105Understanding Cognitive Maps in Reinforcement Learning3 modules · 5 lessons
Introduction to Cognitive Maps
Latent Learning and Cognitive Maps
Learning Cognitive Maps
106Habitual vs Goal-Directed Behavior in Reinforcement Learning3 modules · 6 lessons
Understanding Habitual and Goal-Directed Behavior
Outcome-Devaluation Experiments and Behavioral Control
From Goal-Directed to Habitual Behavior
107Reinforcement Learning and Animal Learning5 modules · 9 lessons
Reinforcement Learning and Conditioning
Mechanisms for Delayed Reinforcement
Cognitive Maps and Environment Models
Model-Free and Model-Based Algorithms
Shaping and Training
108Reinforcement Learning and Animal Learning1 modules · 2 lessons
Correspondence between Reinforcement Learning and Animal Learning
109Historical Context and Influences on Reinforcement Learning3 modules · 7 lessons
Early Influences and Models
Temporal Difference Model and Shaping
Model-Free and Model-Based Reinforcement Learning
110Foundations of Neuroscience for Reinforcement Learning3 modules · 7 lessons
Neuron Structure and Function
Synaptic Transmission and Plasticity
Neuronal Activity Patterns
111Understanding Reward and Reinforcement Signals in Reinforcement Learning3 modules · 6 lessons
Reward and Reinforcement Signals
Values and Prediction Errors
Neuroscience and Reinforcement Learning
112Dopamine's Role in Reward Processing and Reinforcement Learning3 modules · 7 lessons
Introduction to Dopamine and its Functions
Dopamine and Reinforcement Learning
Dopamine Neuron Anatomy and Function
113Experimental Evidence for Reward Prediction Error Hypothesis2 modules · 4 lessons
Dopamine Neuron Activity and Reward Prediction
TD Learning and Dopamine Neuron Activity
114TD Error and Dopamine Neuron Activity Correspondence1 modules · 6 lessons
TD Error and Dopamine Neuron Activity
- LessonHow TD Error Relates to Dopamine Neuron ActivityEnroll to open
- LessonSimplifying the Task Before Studying TD ErrorEnroll to open
- LessonHow TD Error Changes as Learning ProgressesEnroll to open
- LessonHow TD Error Mirrors Dopamine ActivityEnroll to open
- LessonImplications and Limitations of TD Error/Dopamine CorrespondenceEnroll to open
- QuizTD Error and Dopamine Activity QuizEnroll to open
115Neural Actor-Critic Algorithms3 modules · 6 lessons
Introduction to Actor-Critic Algorithms
Neural Network Implementation
Hypothetical Neural Implementation in the Brain
116Actor and Critic Learning Rules Explained1 modules · 7 lessons
Policy-Gradient Actor-Critic Method
- LessonActor-Critic Algorithm OverviewEnroll to open
- LessonHow the Actor and Critic Learn TogetherEnroll to open
- LessonHow the Critic Learns from Past StatesEnroll to open
- LessonHow the Actor Updates Its PolicyEnroll to open
- LessonHow Eligibility Traces Assign Credit in Actor-Critic LearningEnroll to open
- LessonWhy Actor-Critic Learning Could Be Biologically PlausibleEnroll to open
- QuizActor-Critic Learning Rules QuizEnroll to open
117Hedonistic Neurons and Reinforcement Learning3 modules · 5 lessons
Introduction to Hedonistic Neurons
Mechanisms of Hedonistic Neurons
118Collective Reinforcement Learning in Multi-Agent Systems4 modules · 7 lessons
Introduction to Collective Reinforcement Learning
Team Problems in Multi-Agent Reinforcement Learning
Mechanisms for Collective Learning
119Model-Based Reinforcement Learning in the Brain3 modules · 6 lessons
Brain Structures and Model-Based Learning
Role of Hippocampus in Planning
120Understanding Addiction through Reinforcement Learning2 modules · 7 lessons
The Neural Basis of Addiction
121Reinforcement Learning and Neuroscience3 modules · 8 lessons
Reinforcement Learning and Brain Function
Neural Mechanisms of Reinforcement Learning
122Reinforcement Learning and Neuroscience Convergence1 modules · 4 lessons
Correspondence Between Reinforcement Learning and Neuroscience
123Bibliographical and Historical Remarks on Reinforcement Learning3 modules · 8 lessons
Neuroscience and Reinforcement Learning
Historical Development of Reinforcement Learning Algorithms
Multi-Agent Reinforcement Learning and Neuroscience
124TD-Gammon: A Reinforcement Learning Application to Backgammon3 modules · 9 lessons
Introduction to TD-Gammon and Backgammon
TD-Gammon's Architecture and Learning
125Samuel's Checkers Player: A Pioneer in Reinforcement Learning3 modules · 9 lessons
Background and Motivation
Key Components of Samuel's Checkers Player
126Acrobot Control with Reinforcement Learning3 modules · 5 lessons
Introduction to the Acrobot
Reinforcement Learning for Acrobot Control
Results and Analysis
127Watson's Daily-Double Wagering Strategy3 modules · 8 lessons
Daily-Double Wagering Strategy Overview
Action Value Computation and Risk Abatement
128Optimizing Memory Control with Reinforcement Learning4 modules · 6 lessons
DRAM Memory Controller Basics
Reinforcement Learning for Memory Control
Modeling DRAM Access as an MDP
Performance Evaluation and Results
129Achieving Human-Level Performance in Video Games with Deep Reinforcement Learning4 modules · 9 lessons
Introduction to Deep Q-Networks (DQN)
DQN Architecture and Performance
DQN Algorithm and Modifications
Conclusion and Future Directions
130Mastering Go: AlphaGo's Breakthrough3 modules · 9 lessons
Introduction to AlphaGo and Go
AlphaGo's Architecture and Algorithms
Reinforcement Learning in AlphaGo
131Personalizing Web Services with Reinforcement Learning5 modules · 8 lessons
Introduction to Personalized Web Services
Contextual Bandit Problems in Personalization
Markov Decision Problems for Long-Term Personalization
Algorithms for Personalized Recommendation
132Thermal Soaring: Exploiting Rising Air Currents5 modules · 8 lessons
Introduction to Thermal Soaring
Modeling Thermal Soaring as an MDP
Reinforcement Learning for Thermal Soaring
Control Variables in Thermal Soaring
133Exploring the Frontiers of Reinforcement Learning1 modules · 5 lessons
The Unified View of Reinforcement Learning
Start Reinforcement Learning today.
Free for you right now.
Reinforcement Learning