Working with Recurrent Neural Networks
A CNN and an RNN can be viewed as successive sequence-processing stages, but the source material does not prescribe a particular implementation.
From Text to Sequence Data
A neural network does not receive raw words as words. Before a 1D convnet or an RNN can process text, the text must become a numeric representation of an ordered sequence. The central idea in this article is that both model families can process the same kind of sequence tensor, but they process that sequence in different ways. A conceptual pipeline may place a 1D convnet stage before an RNN stage, although the source material does not prescribe one particular implementation.
The first question is not whether the model should be recurrent or convolutional. The first question is whether the input has already been converted into a numeric sequence tensor.
Tokenization and Vectorization
Text vectorization is the process of converting text into numeric tensors. Tokenization chooses the units that will make up the sequence. Those units may be words, characters, or overlapping n-grams of words or characters. Vectorization then associates numeric vectors with the selected tokens. The vectors are packed in their sequence order, producing the numeric sequence tensor that a neural network can process.
A Conceptual Text Trace
Trace the sentence "The service improved" from text to the form used by a sequence-processing model.
Choose tokens: Using words as the tokenization units produces an ordered token sequence: The, service, improved. This is an illustrative choice; tokenization could instead use characters or overlapping n-grams.
Associate vectors: Each token is associated with a numeric vector. Represent these vectors symbolically as v1, v2, and v3 rather than assigning particular numeric values.
Pack the sequence: The vectors are kept in their original order and packed into a sequence tensor: [v1, v2, v3].
Prepare for processing: The resulting object is numeric sequence data, so it can be supplied to a sequence-processing stage such as a 1D convnet or an RNN.
The model receives an ordered sequence of numeric vectors, not the original words as words.
Two Processing Stages
A 1D convnet and an RNN are different model families that can both process sequence data. The 1D convnet contributes a convolution-based way to process the sequence. The RNN contributes a recurrent way to process the sequence, using the fact that the elements have an order. In a conceptual combined pipeline, the convnet stage first transforms the sequence, and the RNN stage then processes the resulting sequence representation. This describes the roles of the stages, not a required layer arrangement or implementation.
Recurrent Processing Across Positions
An RNN is the recurrent component in this comparison. Its purpose is to process data whose elements have an order, such as words in text or measurements in a timeseries. As the sequence is processed, the RNN provides a way to use the sequence as a sequence rather than treating its items as an unordered collection of separate inputs. A useful conceptual trace is to follow a changing recurrent state from one sequence position to the next. The source pack identifies this recurrent role but does not prescribe the internal behavior of a particular recurrent cell.
The important distinction is ordering. An RNN is used to process sequence elements as an ordered sequence, not as an unordered collection of independent items.
Convolutional and Recurrent Roles
| Model family | Sequence-processing role | Associated task examples |
|---|---|---|
| 1D convnet | Convolution-based processing of one-dimensional sequence data | Document classification and timeseries classification |
| RNN | Recurrent processing that uses the order of sequence elements | Text and timeseries sequence processing |
The two families are therefore not unrelated choices. Both can operate on numeric sequence data, but they contribute different processing approaches. A 1D convnet is the one-dimensional counterpart of a 2D convnet and is suited to sequence data rather than image-like two-dimensional data. An RNN is designed around recurrent sequence processing. Using them together means combining those approaches in one pipeline; it does not mean that one model family has stopped being useful.
What the Models Receive
Assuming that the network receives words directly.
Before either model can process text, tokenization and vectorization convert the text into numeric sequence tensors.
Fix:
Trace the input as text, tokens, numeric vectors, and finally an ordered sequence tensor.Treating tokenization and vectorization as the same operation.
Tokenization chooses the units, while vectorization associates numeric vectors with those units.
Fix:
Describe tokenization first and numeric vector association second.Assuming that a 1D convnet-then-RNN pipeline is a required implementation.
The source presents the combination as a conceptual architecture example and does not prescribe a particular implementation.
Fix:
Use the pipeline to explain complementary processing roles, not as a fixed construction rule.Describing the model as if it had human-like understanding of text.
The documented process is conversion into numeric representations followed by sequence processing. The source does not claim human-like understanding.
Fix:
Say that the model processes numeric representations of an ordered sequence.Treating sequence items as an unordered collection.
The RNN's defining role here is to process data whose elements have an order.
Fix:
Keep the sequence order explicit when describing recurrent processing.
Practice the Trace
A sentiment-analysis system receives the sentence "The service improved". Describe the conceptual path from the original text to a possible combined 1D convnet-and-RNN pipeline. Name the tokenization step, the vectorization step, the sequence tensor, the role of the 1D convnet stage, and the role of the RNN stage.
Hints
- Start by choosing a tokenization unit, such as words.
- Explain that each token is associated with a numeric vector and that the vectors remain ordered.
- Describe the 1D convnet and RNN as different sequence-processing stages, not as two names for the same operation.
- State that the combined ordering is conceptual because the source does not prescribe a particular implementation.
What do you think happens?
Before reading the explanation, predict what an RNN should receive when it processes a sentence.
Reveal answer
Answer: An ordered numeric sequence tensor
Text must first be tokenized, associated with numeric vectors, and packed into sequence tensors. The RNN then processes the ordered numeric representation.
Key Takeaways
- Text must be tokenized and vectorized before a 1D convnet or RNN can process it.
- Tokenization chooses units such as words, characters, or overlapping n-grams; vectorization associates numeric vectors with those units.
- A 1D convnet contributes convolution-based processing for sequence data, while an RNN contributes recurrent processing that uses sequence order.
- A CNN-then-RNN pipeline is a conceptual combination of sequence-processing approaches, not a universally required implementation.
- These models process numeric representations of ordered sequences; that description should not be confused with human-like understanding of text.
Key Takeaways
- Neural sequence models receive numeric sequence tensors rather than raw words.
- Tokenization selects sequence units, and vectorization maps those units to numeric vectors.
- A 1D convnet and an RNN provide different ways to process ordered sequence data.
- A conceptual convnet-then-RNN pipeline combines convolution-based and recurrent processing without prescribing exact implementation details.
- Numeric text representations support sequence processing but should not be described as human-like understanding.