Sample Models in Blackjack

Introduction to Environment Models

In reinforcement learning, an agent needs to anticipate the environment's response to its actions. This is where a model of the environment comes into play. A model helps predict the next state and reward given the current state and action.

Distribution Models vs Sample Models

There are two types of models: distribution models and sample models. The key difference lies in how they represent the probability of future outcomes.

A distribution model provides the complete probability distribution of possible outcomes, while a sample model returns a single outcome sampled from that distribution.

The Dozen-Dice Example

Consider rolling a dozen dice and recording the sum. This example illustrates the difference between distribution and sample models.

The distribution model would return all possible sums and their probabilities, whereas the sample model would return just one sum.

Capabilities and Availability

Distribution models are more capable because they contain enough information to generate samples. However, sample models may be easier to construct in many applications.

For instance, simulating the roll of a dozen dice and returning their sum is straightforward, but determining every possible sum and its probability can be complex.

Common Mistakes

• Confusing distribution models with sample models

  Sample models only return a single outcome, not the entire distribution.

  Fix: Understand that distribution models provide complete probability information, while sample models give one possible outcome.

Practice

Summary

1. A model of the environment is crucial in reinforcement learning for predicting future states and rewards.
2. Distribution models provide the full probability distribution of outcomes, while sample models return a single outcome.
3. Distribution models are more capable but may be harder to obtain than sample models.
4. The dozen-dice example illustrates the difference between distribution and sample models.