Text Classification Basics
A recurrent neural network is a powerful neural network that can produce output that is again text, unlike a classification model that only assigns class labels.
Two Shifts in Text Processing
Text processing has changed quickly in recent years, and much of that change comes from neural networks. Two developments stand out. The first is the use of continuous vector representations of words, which are also called word vectors. The second is the use of recurrent neural networks, which can produce text as their output. This article explains what each idea means, how the two relate, and how to decide which kind of model fits a given text task.
Keep one question in mind as you read: when a model finishes its work, does it hand back one choice from a fixed list, or does it write something new? The answer to that question determines which kind of model a task needs.
Words as Continuous Vectors
A word vector is a continuous vector representation of a word. It is also called a distributed word representation, and the technique behind it is implemented in the word2vec library.
A vector is a list of numbers. A word vector is therefore a list of numbers that stands in for a word. The important word in the name is continuous. The numbers in a word vector can take many values, not just a fixed set of labels. A word is not represented by picking one entry from a menu. It is represented by a pattern of numbers that can vary in many small steps.
Because the meaning of a word is spread across several numbers, it is distributed across the whole vector rather than sitting in a single place. This is the reason the same representation is called distributed. The name describes where the information lives: across the vector, not in one isolated value.
Tools for this kind of work are available in common natural language processing libraries. The source notes that spaCy and gensim both offer tools for the methods associated with word vectors and their follow-up work.
A Single Label Versus a Spread Pattern
The diagram below contrasts the two kinds of answer a text model might hold. On one side is a single class label, which is one value chosen from a fixed set of possibilities. On the other side is a word vector, where many numbers together carry the representation.
Read the diagram column by column. In the label column, the meaning is carried by which category was picked. In the vector column, the meaning is carried by the whole pattern of numbers. That pattern is what makes the representation distributed.
What a Classification Model Returns
Many text models only decide which category a piece of text belongs to. A classification model assigns class labels. Given a piece of text, it selects one label from the categories it knows about. The output is short and fixed in form.
A recurrent neural network is a powerful neural network whose defining feature is what it produces. An RNN can generate new text as its output. Its output is itself text, not a choice from a fixed list of labels.
Tracing an RNN From Input to Output
Follow a text task from start to finish to see the RNN's role. The text enters the model, and the model's output is text. The diagram below shows that flow and names the two tasks the source highlights as well suited to this design.
Notice what the RNN does not do in this trace. It does not stop at picking one label for the input. Because it writes text, it is well suited for automatic translation and summarization, since both tasks require writing text rather than choosing one label from a fixed list.
Matching Tasks to Output Types
Before you pick a model, ask what the finished output must look like. Use the table below to match each common text task to the kind of output it requires.
| Text task | Output the model must produce | Output kind |
|---|---|---|
| Classification | One class label chosen from a fixed set | Class label |
| Translation | A new text in another language | Text |
| Summarization | A new, shorter text that captures the content | Text |
Each task and the output it requires
Deciding the output type for a review
A team has a collection of customer reviews. Task one: decide whether each review is positive or negative. Task two: produce a short summary of each review. Which task needs a class label, and which needs generated text?
- 1Identify the output of task one
Deciding positive or negative means choosing one category from a fixed set, which is what a classification model does. The output is a class label.
- 2Identify the output of task two
A summary is a new piece of text. The model must write it rather than pick from a list, which is the defining ability of an RNN.
- 3Match each task to a model type
Task one fits a classification model. Task two fits a text-producing RNN, because summarization requires writing text.
Task one needs a class label. Task two needs generated text, so it calls for an RNN.
What do you think happens?
A model must translate a sentence into French. Does it output one label chosen from a fixed list?
Reveal answer
No, it must produce new text
Translation requires writing a new text in another language. The source states that translation and summarization require writing text rather than picking one label from a fixed list.
Mistakes Beginners Make With These Ideas
Common mistake Treating a word vector as a single class label
What it looks likeThinking that a word's vector is just one value picked from a list of categories.
Why it failsThe numbers in a word vector can take many values. The meaning is spread across the whole vector, not held in one choice.
Do this insteadDescribe a word vector as a continuous pattern of numbers, and reserve the term class label for a single chosen category.
Common mistake Assuming every text task needs an RNN
What it looks likeUsing a text-generating model to decide whether a review is positive or negative.
Why it failsA classification model assigns class labels, and a task that needs only one label from a fixed list does not require the text-writing ability of an RNN.
Do this insteadCheck the required output first. If the answer is one label, a classification model fits. If the answer is new text, an RNN fits.
Common mistake Believing that translation outputs a label
What it looks likeExpecting a translation model to return a category such as French or English.
Why it failsTranslation requires writing a new text. The output is the translated sentence itself.
Do this insteadExpect text as the output for translation and summarization.
Practice Matching Tasks
For each task, state whether the model must output a class label or new text: (a) labeling an email as spam or not spam, (b) translating a paragraph into Spanish, (c) shortening a news article into three sentences.
Hints
- Ask whether the answer is one choice from a fixed list or a new piece of text.
- Translation and summarization both require writing text, according to the source.
Explain in two or three sentences why a word vector is called distributed, using the idea that its numbers can take many values.
Hints
- Think about where the meaning is located: in one value, or across the whole vector?
Key Takeaways
- A word vector is a continuous vector representation of a word, also called a distributed word representation, because its meaning is spread across many numbers.
- A classification model assigns class labels, while a recurrent neural network can produce output that is itself text.
- Translation and summarization require writing text rather than picking one label from a fixed list, which makes RNNs well suited to them.
- Decide which model fits a task by first asking whether the required output is a class label or new text.