Natural Language Processing (NLP) Basics

NLP is the processing and analysis of natural language data, and this chapter's example application is classifying movie reviews.

Programming

Why a Model Cannot Read a Review

A learning algorithm works with numbers. It does not read a sentence the way a person does. Text also arrives in an awkward form for this purpose: each piece of text has its own number of characters, and it does not come with a fixed number of features per example. A short review and a long review can be completely different in size, and their vocabularies can differ too.

This chapter uses one running application: classifying movie reviews. Given the text of a review, decide which category it belongs to. Before reading further, pause and ask yourself what you would need to do to hand a stack of reviews to a learning algorithm. Each review needs to be converted into the same kind of structured description, so that every review ends up looking like every other review to the model. This article follows that conversion step by step, and then looks at what you can learn by examining the result.

What NLP Means

Definition

Natural language processing, usually abbreviated NLP, is the name for the work of processing and analyzing natural language data so that it can be used in computational methods.

The word natural in this definition refers to human language, such as the English sentences in a movie review, as opposed to structured data that already sits in fixed columns. NLP is the set of steps that takes that language and prepares it for computation and analysis. Classifying movie reviews is the typical application used in this chapter: the text of a review is the input, and the category it belongs to is the output.

Turning Text into Token Counts

The central idea is a representation called bag of words. It breaks a text into tokens, which are the pieces the representation works with, and it keeps the count of each token. The source describes this as a simple and powerful solution for text classification tasks. It is simple because the idea needs only counting. It is powerful because, for many tasks, the counts of the right words carry much of the signal a classifier needs.

The result of the bag-of-words step is that every document gets the same kind of structured description: a set of token-to-count pairs. A review of twelve words and a review of two hundred words both become a collection of tokens with counts, so both can be given to the same model. The pipeline below shows the path a raw review takes from text to a predicted label.

Notice what the model receives at the end of this chain. It never receives the sentence itself. It receives the counts derived from the sentence, and those counts are the only description of the review it will learn from.

Tracing One Review Into Counts

Worked example

Counting the tokens of one review

Turn the review 'The movie was great, great acting and a great story' into token counts.

1
Normalize the text
Lowercase the review and remove the comma so that 'The' and 'the' are treated as the same token and the comma does not attach to a word.
2
Split into tokens
Split on whitespace. The tokens in order are the, movie, was, great, great, acting, and, a, great, story.
3
Count each token
Walk through the tokens and add one to the count for each token seen. The word great appears three times, and every other token appears once.
4
Keep the pairs
The review is now described by eight token-to-count pairs. The order of the sentence is no longer part of this description.

Result

the: 1, movie: 1, was: 1, great: 3, acting: 1, and: 1, a: 1, story: 1

The source notes that the example in this chapter is illustrative and written for the article, not taken from the source. It uses a tiny made-up review and plain Python to count tokens. Read it as a model of the idea, not as the exact pipeline a production system would use.

python

review = 'The movie was great, great acting and a great story'
tokens = review.lower().replace(',', '').split()
counts = {}
for token in tokens:
    counts[token] = counts.get(token, 0) + 1
print(tokens)
print(counts)

review = 'The movie was great, great acting and a great story'
tokens = review.lower().replace(',', '').split()
counts = {}
for token in tokens:
    counts[token] = counts.get(token, 0) + 1
print(tokens)
print(counts)

Output (expected)

['the', 'movie', 'was', 'great', 'great', 'acting', 'and', 'a', 'great', 'story']
{'the': 1, 'movie': 1, 'was': 1, 'great': 3, 'acting': 1, 'and': 1, 'a': 1, 'story': 1}

Notes on the code

lower() makes The and the the same token
replace removes the comma so it does not stick to great
split() breaks the review on whitespace into tokens
counts.get(token, 0) returns zero for a new token, so the first sighting starts at one

Reading the Token Count Map

The counts from the previous step can be read as a map. Each token is a key, and its count is the value. Looking at the map shows what the representation kept: the word great stands out with a count of three, while the other tokens each contribute a single count. A classifier built on this representation can only respond to what appears in this map.

The diagram above is the state of the representation after the counting step. Each token keeps its count, and nothing about the sentence's ordering survives in the map.

What Word Order Throws Away

Because the representation keeps counts rather than positions, two texts with the same words in a different order can produce identical descriptions. The example below is generated for this article to show that effect. Both sentences contain the same five tokens, so their counts match exactly.

python

a = 'the movie was not great'.split()
b = 'great was not the movie'.split()
print(sorted(a) == sorted(b))
print(sorted(set(a)) == sorted(set(b)))

a = 'the movie was not great'.split()
b = 'great was not the movie'.split()
print(sorted(a) == sorted(b))
print(sorted(set(a)) == sorted(set(b)))

Output (expected)

True
True

Notes on the code

sorted compares the multiset of tokens, which is what counting keeps
sorting a and b shows they hold the same tokens with the same counts

Tokens and N-grams

A single token is not the only unit you can extract. The source defines an n-gram as a run of n consecutive tokens. A one-token run is a unigram, a two-token run is a bigram, and a three-token run is a trigram. Bigrams and trigrams keep a little of the local order that bag-of-words discards, because the tokens in each run remain next to one another.

Think of a window of fixed width that slides across the token list one position at a time. For a trigram window over the review 'the movie was great', the first window covers the movie was, and after one slide it covers movie was great. The diagram shows that sliding process.

The same sliding idea produces unigrams when the window is one token wide and bigrams when it is two tokens wide. Each run is a feature you can count, just like a single token. Looking at which tokens and n-grams were extracted is one of the ways to check what the representation actually contains.

Why Representation Matters for Classification Tasks

The source names spam and fraud detection and sentiment analysis as text classification tasks. In each case the input is text and the output is a category. A message is either spam or not, a transaction description is flagged or not, and a review expresses a sentiment that can be assigned to a class. The movie review example belongs to the same family: the text is the input and the sentiment category is the output.

Task	Input	Output category
Spam detection	A message	Spam or not spam
Fraud detection	A transaction description	Flagged or not flagged
Sentiment analysis	A review	A sentiment class
Movie review classification	The text of a review	The category the review belongs to

Text classification tasks share one structure: text in, category out

The source makes a point that goes beyond building the representation: as is often the case in machine learning, the representation of the data is key in NLP applications. The same raw texts can look very different under different representations, and that difference determines what a classifier is able to learn. If the representation drops the features that separate spam from legitimate mail, no amount of model tuning will recover them. For this reason, choosing and checking the representation is part of the classification work itself.

Inspecting Tokens and Models

The source states that inspecting the tokens and n-grams that are extracted can give powerful insights into the modeling process. Inspection lets you check whether the representation contains the features you expected. It also helps reveal whether the model may be relying on things you did not intend, such as a token that appears in every review regardless of its label.

The source also states that in text-processing applications it is often possible to introspect models in a meaningful way, for both supervised and unsupervised tasks. Supervised tasks, such as classifying reviews with known labels, and unsupervised tasks, where labels are not given, can both be examined this way. The practical advice in the source is to take full advantage of that ability when using NLP-based methods in practice.

Best practice

Before trusting a text model, print the tokens and n-grams it was built from, check that the counts match what you expect for a few example texts, and look at which features the model is using. This turns the model from a black box into something you can question.

Common Mistakes With Text Representations

Common mistake
Assuming the bag-of-words counts keep the order of the words
What it looks like
The sentences 'the movie was not great' and 'great was not the movie' produce identical token counts.
Why it fails
The representation stores how often each token occurs, not where it occurs.
Do this instead
Treat word order as lost, and use n-grams if local order matters for the task.
Common mistake
Forgetting to normalize text before counting
What it looks like
Counting 'The' and 'the' as different tokens splits one word's count across two keys.
Why it fails
The counts then no longer reflect how often the word actually appears.
Do this instead
Apply the same normalization, such as lowercasing and removing punctuation, to every review before counting.
Common mistake
Reading a high count as proof of meaning
Why it fails
A count only says how often a token occurred. The source does not claim that word counts capture everything about language.
Common mistake
Skipping inspection and trusting the model's accuracy alone

Practice Tracing a Representation

EASY

Take the review 'the plot was dull and the acting was dull'. Write out its tokens in order, then list the count for each distinct token. Next, list the trigrams produced by sliding a three-token window across the tokens.

Hints

Lowercase the text and split on whitespace before counting.
dull appears twice and the appears twice, so those two keys should have a count of two.
A review with n tokens produces n minus two trigrams when n is at least three.

Your attempt (not saved)

MEDIUM

Write a short Python function that takes a review string and returns a dictionary of bigram counts, where each bigram is a tuple of two consecutive tokens. Then explain in one sentence what the bigram counts capture that the unigram counts do not.