Timeseries Forecasting
Sequence data includes text, timeseries, and other ordered inputs.
From Ordered Data to Model Input
Sequence data is data whose elements occur in an order. Text, timeseries, and other ordered inputs all fit this description. A neural network cannot receive raw text directly. Before sequence data can be processed, it must be represented as numeric tensors. This gives sequence processing two broad stages: create a numerical representation, then apply a model designed for sequences.
Tracing a Sequence Representation
A Short Text Sequence
Trace what happens to the ordered text sequence "red sky" before it can be given to a neural network.
Start with ordered text: The input is text, and its elements have an order. In this generated example, the units are words: "red" followed by "sky".
Tokenize: Tokenization identifies the units in the sequence. Here, the generated tokens are "red" and "sky".
Vectorize: Text vectorization supplies each generated token with a numeric representation. The exact numbers are not specified here; the important change is from tokens to numeric vectors.
Pack the sequence: The vectors remain arranged in sequence order and are packed into a sequence tensor.
The neural network receives an ordered numeric tensor rather than the original raw text.
The same reasoning applies to a timeseries. Its values occupy ordered positions, and the resulting numerical representation must preserve that sequence organization. The model therefore receives numeric sequence data, not an unprocessed description of the original input.
Tokenization and Vectorization
Tokenization and vectorization solve different problems. Tokenization chooses or identifies the units in a sequence. For text, those units can be words or characters. Vectorization then gives each generated token a numeric vector. Only after this second step can the vectors be packed into sequence tensors for a deep neural network.
| Stage | Main question | Result |
|---|---|---|
| Tokenization | What are the units in this sequence? | Tokens such as words or characters |
| Vectorization | How is each token represented numerically? | A numeric vector for each token |
| Sequence tensor formation | How are the representations supplied to the model? | Vectors packed in sequence order |
Two major ways to associate vectors with tokens are one-hot encoding and token embedding. Token embedding is typically used for words and is also called word embedding in that setting. These are ways of supplying numeric representations; they do not replace the need to identify the sequence units first.
Choosing a Sequence Model
Once sequence data has become numeric tensor data, a model designed for sequences can process it. The source identifies two fundamental deep-learning approaches for sequence processing: one-dimensional convolutional networks, or 1D convnets, and recurrent neural networks. A 1D convnet applies the one-dimensional convolutional approach to a sequence. At a broad level, it is useful when the task depends on detecting patterns across nearby positions in an ordered input. An RNN provides the contrasting sequential approach by carrying information through sequential states as it processes the sequence.
This distinction is about the broad processing role, not a rule that one model is always correct. A 1D convnet is a natural candidate when useful evidence appears as patterns across nearby sequence positions. A recurrent model is the contrasting choice when the explanation emphasizes information being carried through sequential states. Both still require an appropriate numeric sequence representation as input.
Where 1D Convnets Fit
For a text sequence, a 1D convnet can be considered when nearby words or characters form useful patterns. For a timeseries, it can be considered when nearby positions contain a useful temporal pattern. The same reasoning extends to other ordered inputs. In every case, the raw input must first be represented as numeric sequence data.
Mistakes Beginners Make
Treating raw text as if it were already a neural-network input.
A neural network needs a numeric tensor rather than raw text.
Fix:
Separate the preparation stages: tokenize the text, vectorize the tokens, and pack the resulting vectors into a sequence tensor.Using tokenization and vectorization as if they meant the same thing.
Tokenization identifies the units; vectorization supplies their numeric representations.
Fix:
Describe tokenization first and vectorization second.Ignoring sequence order after creating numeric vectors.
Text and timeseries are ordered inputs, so their positions are part of the sequence representation.
Fix:
Keep the generated vectors arranged as a sequence tensor.Describing a 1D convnet and an RNN as identical sequence models.
The approaches have different broad roles: a 1D convnet applies one-dimensional convolution, while an RNN carries information through sequential states.
Fix:
Use local pattern detection as the broad 1D-convnet intuition and sequential state as the broad RNN intuition.
Check Your Understanding
A dataset contains ordered text and you want to examine whether a 1D convnet is a suitable sequence-processing approach. Describe the preparation and model-selection reasoning in order. Your answer should name the units chosen during tokenization, explain what vectorization contributes, state what the neural network receives, and identify the kind of sequence pattern that would make a 1D convnet a sensible candidate.
Hints
- Start by separating tokenization from vectorization.
- Remember that the model receives a numeric tensor, not raw text.
- Use the broad role of a 1D convnet rather than claiming that it is suitable for every sequence task.
A Complete Reasoning Chain
Choose the correct description for a text sequence that will be processed by a sequence model.
Identify the data type: Text is ordered sequence data. It may be treated as a sequence of words or characters.
Prepare the representation: Tokenization identifies the chosen units. Vectorization gives each token a numeric vector, and those vectors are packed into a sequence tensor.
Select a broad model role: A 1D convnet is a candidate when useful patterns occur across nearby sequence positions. An RNN represents the contrasting sequential-state approach.
The correct reasoning keeps representation and model selection separate: first create ordered numeric input, then choose a sequence-processing approach based on the pattern the task requires.
Key Takeaways
- Sequence data includes text, timeseries, and other ordered inputs.
- Neural networks receive numeric tensors, so raw sequence data must first be represented numerically.
- Tokenization identifies sequence units; vectorization assigns those units numeric vectors.
- One-hot encoding and token embedding are two major ways to associate vectors with tokens.
- 1D convnets and recurrent neural networks are two fundamental approaches to sequence processing, with different broad processing roles.
Key Takeaways
- Sequence data is ordered data, including text and timeseries.
- A neural network needs numeric sequence tensors rather than raw text.
- Tokenization identifies units, while vectorization represents those units numerically.
- 1D convnets apply one-dimensional convolution to sequence processing and are useful to consider when nearby positions form meaningful patterns.
- RNNs provide a contrasting sequential-state approach to processing sequence data.