Tokens and N-grams in Text Processing

Imagine you are asked to build a filter that flags spam emails, or one that decides whether a movie review is positive or negative. The learning algorithm you plan to use only accepts numbers. It has no direct way to take in a sentence such as 'Great film, I loved the ending.' Something has to turn that sentence into numbers first. That step is called data representation.

The choice of representation sits between the raw text and the model, so it decides what information the model is allowed to see. The course material makes a general point that applies here: as is often the case in machine learning, the representation of the data is key in NLP applications. The same review can be turned into different sets of numbers, and a model that learns from one set may succeed or fail because of the representation rather than the learning algorithm itself.

A bag-of-words representation starts by splitting text into tokens, which are the units the representation counts. In this article, a token is a lowercase word with punctuation removed. The representation then records each distinct token and how many times it appears. The result is a bag: a collection of words with counts, and nothing that says where each word sat in the sentence.

Follow the steps by hand on one short review: 'Boring plot, boring acting, great ending'. Each step changes the text, and the next step works on the result of the previous one.

Notice that the six tokens collapse into five keys. The repeated token boring is stored once with a count of 2. This is what it means for the representation to count distinct tokens: the model learns a fixed table of words and how often each one occurred in the review.

A bag-of-words representation keeps which tokens occur and how often. It does not keep the order of the tokens. This is the trade-off behind its simplicity. Two texts with the same words in different orders end up with identical counts, so the representation cannot tell them apart.

Whether that loss matters depends on the task. For many classification problems, a strong signal such as repeated words like boring may be enough. This is consistent with the source's description of bag-of-words as a simple and powerful solution for text classification tasks such as spam and fraud detection and sentiment analysis.

An n-gram is a run of n consecutive tokens. A window of size n slides across the token sequence one position at a time, and each position produces one n-gram. A bigram, with n equal to 2, keeps a small piece of local order that a single-token count throws away.

Notice what this list reveals. The bigrams 'boring plot' and 'boring acting' record that boring is adjacent to other words, while a single-token bag only knows that boring appeared twice. Inspecting a list like this is a direct way to check that the extracted pieces match your expectations for the text.

As n grows, each unit covers more of the sentence and captures more local order, but the number of distinct units also tends to grow. The representation becomes more specific, which can help when a short phrase carries the signal.

The course material makes a point that goes beyond choosing a representation: inspecting the tokens and n-grams that are extracted can give powerful insights into the modeling process. Looking at these pieces directly lets you see what the model is actually being given, which is often different from what you assumed when you wrote the preprocessing code.

Consider the review 'Not boring at all'. Lowercasing and removing punctuation gives the tokens not, boring, at, all. A unigram bag records boring once and not once, which looks like a negative cue. The bigram 'not boring' keeps the adjacency that tells the model the negation applies to boring.