Text Classification with Machine Learning

Much of the machine learning work you have seen so far ends with a single decision: this email is spam, this image shows a cat, this customer will leave. The model's answer is a label chosen from a fixed set. Many language problems do not fit that pattern. If you want a sentence rewritten in another language, or a long article shortened to a few lines, the answer you need is itself a passage of text, and its length is not fixed in advance.

This article looks at a family of neural networks designed for exactly that situation. Before reading on, keep one question in mind: for a given task, does the model need to choose a category, or does it need to write new text?

There are two kinds of model to keep apart. One kind takes an input and returns one of several predefined categories. The other kind takes an input and produces a new piece of text as its output. A classification model belongs to the first kind: it can only assign class labels. A recurrent neural network, usually abbreviated RNN, belongs to the second kind. RNNs are described as a particularly powerful type of neural network, and their defining feature for this topic is the kind of output they can produce: the output can be text again.

The contrast is about the shape of the answer. In the classifier's case the answer is one entry from a list decided in advance. In the RNN's case the answer is text, which means a sequence of words. Because the desired answer is a sequence of words in tasks such as automatic translation and automatic summarization, an RNN is well suited to them.

A label-only model returns its whole answer in one decision. A text-producing model returns an answer that is a sequence, and the source notes that the length of that sequence is not fixed in advance. To see what this means, trace a short output one word at a time. The example below is an illustration of the general idea, not a description of any particular trained model.

Read the timeline as a sequence of decisions about what comes next. Each word is a piece of the answer, and the answer keeps growing until the text is complete. A classifier never has this stage: it makes one choice and stops. Keeping this difference in view explains why the RNN, rather than a classifier, is the family the source connects to translation and summarization.

Translation and summarization are the two text-processing tasks the source names as well suited to RNNs. Both take text as input and return text as output. The diagram below shows the flow for each. In both chains the model sits in the middle, and what changes is the meaning of the output: a sentence in another language in the first case, and a shorter version of a longer text in the second.

The worked example did not require knowing how the internal computation of an RNN works. It only required checking the shape of the input and the shape of the desired output. The same check applies to summarization: a long article goes in, and a shorter text comes out, so the task calls for text as output.

Learners new to this topic tend to slip into a few predictable errors. The items below describe each one and how to correct it.

The source offers two directions for further study. Read the paper for the underlying ideas and the tutorial for hands-on implementation.

A sensible order is to first confirm the output-shape reasoning from this article, then read the paper for the ideas behind text-producing networks, and then work through the TensorFlow tutorial to build one yourself.