Chapter 7 Working with Text Data

Most of the features you have worked with so far are either numbers that measure a quantity or items chosen from a fixed list. Many real applications, however, contain a third kind of information: text. Consider a system that must decide whether an email is legitimate or spam. The content of the email will almost certainly carry information that matters for that decision. The same is true when someone wants to learn a politician's opinion on immigration from speeches or tweets, or when a customer service system must decide whether a message is a complaint or an inquiry from its subject line and content.

This article focuses on the step that comes before learning. It asks two questions: how is text data characterized as a feature, and why does it have to be prepared before a machine learning algorithm can use it? Along the way you will see how text differs from the two feature types you already know, and what the source does and does not tell you about the kinds of string data that appear in practice.

Before looking at text in detail, recall the two feature types covered so far. A continuous feature describes a quantity. A categorical feature is an item from a fixed list. Each kind of feature asks a different question about the data, and the difference determines how the data must be handled.

The key distinction lies in the shape of each value. Continuous and categorical features each have a fixed kind of value per example: a quantity in one case, a label from a fixed list in the other. Text features are strings, and those strings can differ widely in length. That variable shape is the reason text cannot simply be dropped into an algorithm the way a number can.

Read the diagram column by column. Each feature type has its own shape of value. The continuous and categorical columns both have one fixed kind of value per example, while the text column is defined by strings whose length varies. This is the structural difference that makes text a separate case rather than a special type of number or category.

Machine learning algorithms cannot directly handle raw text. Preprocessing is the work of converting raw text into a structured format suitable for analysis. The source describes this as a necessary step: the text has to be processed before any algorithm is applied to it.

The pipeline shows the order of the stages at the level the source supports. Raw strings enter preprocessing, which produces a structured format, and only that structured format is passed to an algorithm. The source does not name the individual sub-steps inside preprocessing, so this article does not list them.

When you are handed a column and must decide how to treat it, you can ask the same questions the definitions answer. The flowchart below turns those questions into an order to check them in.

Follow the flowchart from the top. The first question separates quantities from everything else. The second separates fixed-list labels from strings. The third identifies text. Work through the questions in this order because a quantity can look like a string when it is stored as characters, and the definitions are what decide the feature type.

The material introduces four kinds of string data that you may encounter in practice. It announces these four kinds but does not list or describe them, so this article does not name them or state how they differ. What can be said with confidence is that string data is the general form of the text features discussed above, and that the four kinds are a further classification developed by the surrounding material. Their details should be checked in a fuller source.