Tokenization in Text Processing

Machine learning models work with numbers, but much of the data we care about is text: emails, product reviews, news articles. A model cannot read a sentence the way a person does, so the text has to be converted into a numeric form first.

Bag-of-Words processing is a method for converting a collection of text documents into numeric form. It does this in a fixed sequence of steps, and the output of the whole sequence is a numeric vector for every document. In this article you will follow a document from the moment it enters the process until it leaves as a list of numbers, and you will see exactly which information is kept and which is thrown away along the way.

Bag-of-Words processing has three stages, and each one depends on the output of the stage before it. The first stage is tokenization, which breaks each document into individual words. The second stage is vocabulary building, which looks across all of the documents and gathers every distinct word into one shared, numbered list. The third stage is encoding, which uses that shared list to count how often each vocabulary word appears in each document. The final output is one vector of counts per document.

The order matters. Vocabulary building cannot start until tokenization has produced token lists for every document, and encoding cannot count anything until the shared vocabulary exists. Each stage consumes exactly what the previous stage produced.

Tokenization splits each document into tokens, usually by splitting on whitespace and punctuation. A token is the unit that the rest of the process works with. Tokenization is the first step in Bag-of-Words processing, and everything that follows works with these tokens rather than with the original sentence.

Notice what the tokens are no longer doing. After this stage, the document is a collection of words rather than a sequence, which is the reason the method is called a bag of words. A bag holds its items without any arrangement, and that is exactly the state the document is in once tokenized.

Vocabulary building takes the token lists from every document and collects every distinct word into one numbered list. This single list is shared by all of the documents. Each distinct word gets exactly one entry, no matter how many documents contain it.

The sorted list gives each vocabulary word a fixed position. Position 0 is cat, position 1 is dog, position 2 is down, position 3 is sat, and position 4 is the. Those positions are what the next stage will use to lay out the counts.

Encoding counts how often each vocabulary word appears in a document. The result is one count vector per document, with one feature for each unique word in the shared vocabulary. To encode a document, walk through the vocabulary in its fixed order and write down how many times each word occurs in that document. A word that does not appear gets a count of zero.

Each position in a document's vector is a feature, and each feature corresponds to one unique word from the whole dataset. The feature for a given word therefore means the same thing in every document. The first feature always refers to the same vocabulary word, and so does the last. This shared meaning is what lets a model compare documents, because every document is measured on the same set of word features.

This is also why the vocabulary must be built once, from all of the documents, rather than separately for each one. A document's vector has length equal to the number of unique words in the whole dataset, not the number of unique words in that document alone. In the two-document example, both vectors have five features, even though 'the cat sat' contains only three distinct words.

Look again at the second document. Its word order, dog then sat then down, is gone from the vector. The vector only records that dog, sat, and down each appear once. This is the intended behavior of the representation, and it is also its main limit: two sentences that differ only in word order produce identical vectors.

In the comparison, 'cat sat' and 'sat cat' both produce the vector with cat at 1 and sat at 1, so the two are indistinguishable to the model. The document 'cat cat sat' differs because the count for cat changes to 2, so its vector is different. Word order is discarded, but word counts are kept.