Text Generation with LSTM
A language model learns the statistical structure of language by modeling possible next tokens.
From Prediction to Generation
A language model learns the statistical structure of language by modeling possible next tokens. Its basic task is local: given tokens that have already appeared, it assigns probabilities to tokens that could appear next. Text generation extends this task through repetition. The model predicts one token, that prediction is appended to the sequence, and the enlarged sequence is used for another prediction.
The central idea is a repeated feedback cycle: predict, append, and predict again.
Tracing the First Prediction
A Short Character Sequence
Suppose a character-level model receives the characters c, a, and t as its current sequence. What happens when the model generates more text?
Start with the current sequence: The model receives c, a, and t as the conditioning sequence. In a character-level setup, these individual characters are the tokens.
Produce the next-character distribution: The LSTM character model produces a probability distribution over possible next characters. That distribution represents how plausible each possible next character is after c, a, and t.
Select a next character: A next character is selected using the model's probabilities. The selected character becomes part of the generated sequence.
Feed the enlarged sequence back: The current sequence now contains c, a, t, and the selected character. This enlarged sequence supplies more context for the next prediction.
The model does not need to produce the entire continuation in one step. It produces one next character, attaches it to the sequence, and repeats the process.
Tokens and Language Models
The word token does not necessarily mean a whole word. In language modeling, a token can be a word or a character. More generally, the model uses the preceding sequence to assign probabilities to possible next tokens. This makes language modeling a task of representing the statistical structure of language through next-token possibilities.
| Language modeling in general | Character-level language modeling |
|---|---|
| Models possible next tokens from preceding tokens. | Uses individual characters as the tokens. |
| The token may refer to a word or a character. | Receives a string containing N characters and predicts character N + 1. |
| The output provides probabilities for possible next tokens. | The output is a softmax over possible characters, forming a probability distribution for the next character. |
Character-level modeling is one particular choice of prediction unit within language modeling.
Character-level modeling changes the prediction unit, not the overall generation mechanism. The model still predicts, appends, and predicts again.
Following the Feedback Cycle
Consider a generated sequence that begins with a short conditioning string. At the first step, the LSTM uses that string to produce probabilities for possible next characters or words. After one output is selected and attached, the model receives a longer sequence at the next step. The second prediction therefore uses the original conditioning data together with the newly appended token. The same relationship continues at every later step.
What do you think happens?
After the model predicts and appends one token, what sequence is used for the next prediction?
Reveal answer
Answer: The enlarged sequence containing the earlier context and the appended token
Generation uses repeated feedback. The output from one step is added to the sequence, giving the next prediction more context than the previous prediction.
Practice the Generation Logic
Explain the generation process for a character-level LSTM when its current sequence contains N characters. Your explanation should name the model output, describe what happens to the selected character, and state what sequence is used at the next step.
Hints
- Begin with the model's prediction unit.
- Describe the output as a probability distribution over possible next characters.
- Explain how appending the selected character changes the sequence used for the next prediction.
Checking a Generation Trace
A model starts with conditioning data and generates three additional tokens. What should appear in a correct trace?
First step: The conditioning data is used to produce a distribution over possible next tokens. One token is selected and appended.
Second step: The model uses the enlarged sequence, which now includes the first selected token, to produce the next distribution.
Third step: The second selected token is appended, and the resulting sequence is used to produce the third prediction.
A correct trace shows three prediction steps and two feedback updates before the third prediction is made, with every later step using the sequence enlarged by earlier outputs.
Key Takeaways
- A language model represents the statistical structure of language by assigning probabilities to possible next tokens based on preceding tokens.
- Text generation is iterative: the model predicts a token, appends it to the sequence, and predicts again.
- The initial sequence is conditioning data, and each generated token becomes part of the context for later predictions.
- A character-level model uses individual characters as tokens and produces a probability distribution over possible next characters.
- The same feedback mechanism can continue repeatedly to produce sequences of arbitrary length.
Key Takeaways
- A language model predicts possible next tokens from previous tokens.
- An LSTM sequence generator creates text one token at a time through repeated feedback.
- Conditioning data determines the context for the first prediction.
- Each appended output enlarges the context used for the next prediction.
- Character-level language modeling uses individual characters as the prediction units.