LSTM Networks and Sequence Memory
Character-level generation learns the relationship between a fixed-length character context and the character that follows it.
One Character at a Time
Character-level generation asks a narrow question repeatedly: given the characters available so far, which character should come next? The model does not produce an entire word or sentence in one operation. It predicts one character, appends that character to the text, and uses the enlarged context for the next prediction.
Building Training Windows
Training begins by dividing the text into overlapping character sequences. Each sequence has a fixed length, often described as maxlen in the source. The character immediately following a sequence becomes that sequence's target. In other words, the input supplies the context and the target identifies what the model should learn to predict next.
Aligning context and target
Suppose a toy text is divided into fixed-length windows of four characters. Show how overlapping windows are paired with the character that follows each window.
First window: The first four-character context is paired with the fifth character as its next-character target.
Advance one position: The next context begins one character later, so it overlaps most of the first context. Its target is the character immediately after this new window.
Repeat: Moving the window through the text creates many context-target pairs. Every pair preserves the same contract: the input is the context, and the target is the following character.
Overlapping windows turn one text string into many supervised learning examples, each teaching the relationship between a fixed-length context and the character after it.
The source describes the input array x as three-dimensional. Its dimensions correspond to the number of sequences, the sequence length maxlen, and the number of unique characters. The target array y contains the one-hot-encoded character that follows each input sequence. This alignment is the central data contract for the training data.
From Context to Probabilities
The generation model combines an LSTM layer with a Dense softmax classifier. The LSTM processes the character sequence and uses sequence memory to help represent information from earlier characters while considering the current sequence. The Dense softmax classifier converts the model's result into a probability distribution over possible next characters.
The Dense softmax output must represent the full character vocabulary. If the vocabulary contains the unique characters used by the training text, the classifier needs one output position for each of those possible characters. The target's one-hot encoding and the classifier's output positions must therefore refer to the same character set.
The Autoregressive Loop
After training, generation starts with a seed snippet. The routine sends the current fixed-length context through the model, obtains a distribution for the next character, adjusts that distribution using a temperature, and samples one character. The sampled character is appended to the generated text. The oldest character is then removed from the fixed-length context so that the next prediction uses the newest available window.
Tracing one generation step
A seed snippet is already available as the current context. Describe one complete generation step without assuming a particular predicted character.
Read the context: The current fixed-length character window is presented to the trained model.
Predict: The LSTM and Dense softmax classifier produce a probability distribution over the character vocabulary.
Adjust and sample: The distribution is adjusted using the selected temperature, and one character is sampled from it.
Append: The sampled character is added to the generated text.
Advance: The oldest character is removed from the fixed-length context, leaving a new context for the next iteration.
One iteration both extends the generated passage and prepares the next context. Repeating the iteration creates the passage character by character.
Temperature and Variety
Temperature changes the probability distribution used for sampling. A lower temperature makes the sampling distribution more concentrated, so choices tend to favor the model's more likely characters. A higher temperature makes the distribution more varied, allowing less likely characters to receive more opportunity during sampling. Temperature changes how the same model predictions are sampled; it does not create a separate model.
The source generation loop compares temperatures of 0.2, 0.5, 1.0, and 1.2 while training progresses. These values are alternative sampling choices applied to predictions from the same model, making it possible to compare more concentrated and more varied generated text.
Diagnosing Generation Failures
A poor-looking passage does not identify its own cause. Debug the system in stages: data preparation, model construction, and sampling. Checking these stages separately prevents a sampling adjustment from hiding a data or model problem.
Pairing a context with the wrong target character.
The training example no longer teaches the intended next-character relationship.
Fix:
Check that every input sequence is aligned with the character directly after it.Using an output layer that does not cover the complete character vocabulary.
The model cannot represent a probability for every possible target character.
Fix:
Check that the Dense softmax output size matches the character vocabulary.Debugging only the temperature.
Temperature changes sampling, but it does not correct a broken probability-to-character conversion.
Fix:
Verify the conversion from predicted probabilities to a sampled index and then back to a character.
Practice the Trace
Describe the next generation step after a model has received its current fixed-length context. Include the model prediction, temperature adjustment, character sampling, text append, and context update.
Hints
- The model first produces a distribution over the character vocabulary.
- The sampled character must be added to the generated text.
- The next context removes the oldest character so the window remains fixed-length.
A generation result looks wrong. Explain which stage you would inspect first if the target is not the character immediately following each input sequence, and which stage you would inspect if the sampled probability index is converted back to the wrong character.
Hints
- The first issue concerns the relationship between input windows and targets.
- The second issue occurs after the model has produced probabilities.
Key Takeaways
- Character-level generation learns a mapping from a fixed-length character context to the character that follows it.
- Overlapping input windows and aligned one-hot next-character targets form the central training data contract.
- The LSTM processes sequence information, while the Dense softmax classifier produces probabilities across the character vocabulary.
- Generation is autoregressive: each sampled character is appended and used as part of the next context.
- Temperature changes sampling variety, while data alignment, vocabulary coverage, and probability-to-character conversion are separate debugging concerns.
Key Takeaways
- Character-level generation predicts one next character from a fixed-length context.
- Training uses overlapping contexts paired with the character immediately following each context.
- An LSTM supplies sequence processing and memory, and a Dense softmax classifier supplies vocabulary-wide next-character probabilities.
- The generation loop samples, appends, advances the context, and repeats.
- Temperature controls how concentrated or varied sampling is, but it cannot repair errors in data preparation or sampling logic.