Vocabulary Building for Text Features
Bag-of-Words processing runs three stages in order: tokenization, vocabulary building, and encoding.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
Why Text Must Become Numbers
Machine learning models work with numbers, but much of the data people care about is text: emails, product reviews, and news articles. Bag-of-Words processing is a method for converting a collection of text documents into numeric form. It does this in a fixed sequence of steps, and the output of the whole sequence is a numeric vector for every document.
What do you think happens?
Consider the two sentences 'the cat sat' and 'sat the cat'. Before reading on, predict whether Bag-of-Words processing will give them the same numeric vector or different ones.
Reveal answer
The same vector, because they use the same words
Bag-of-Words records how often each word appears, not the order the words appear in. Both sentences contain one cat, one sat, and one the, so their count vectors are identical. Keep this result in mind; the later sections explain exactly why it happens.
Three Stages in Fixed Order
Bag-of-Words processing has three stages, and each one depends on the output of the stage before it. The first stage, tokenization, breaks each document into individual words. The second stage, vocabulary building, looks across all of the documents and gathers every distinct word into one shared, numbered list. The third stage, encoding, uses that shared list to count how often each vocabulary word appears in each document. The final output is one vector of counts per document.
Tracing Two Documents Through the Pipeline
We will follow two short documents through every stage: 'the cat sat' and 'the dog sat down'. Keep both in view as you read, because each stage's output is the input to the next.
Why Every Vector Has the Same Shape
Each position in a document's vector is a feature, and each feature corresponds to one unique word from the whole dataset. This is why the vectors above both have five entries even though the first document contains only three words. The length of every vector is set by the size of the shared vocabulary, not by the length of the individual document.
Because the feature at each position always refers to the same vocabulary word, the meaning of a feature is the same in every document. In the example, the first feature always refers to cat and the last always refers to the. This shared meaning is what lets a model compare documents, since every document is measured on the same set of word features. A document that never uses a word still gets a 0 in that word's position, which keeps the positions aligned.
What Word Order Disappears
Notice what happens to the original sentences. The second document's word order, dog then sat then down, is gone from the vector. The vector records only that dog, sat, and down each appear once. This is the intended behavior of the representation, and it is also its main limit: two sentences that differ only in word order produce identical vectors.
Mistakes That Corrupt Vectors
Each stage depends on the one before it, so an error in an early stage spreads into every later vector. The table below lists three common errors and how each one changes the result compared with the correct procedure.
| Mistake | Effect on the vectors | Correct handling |
|---|---|---|
| Building a separate vocabulary for each document | Each vector gets a different length and the positions no longer refer to the same word, so documents cannot be compared. | Build one vocabulary from all documents and reuse it for every encoding step. |
| Leaving punctuation attached to words | 'sat.' and 'sat' are counted as two different words, so the count for sat is split and the vocabulary grows with spurious entries. | Split on punctuation during tokenization so that only the word itself is counted. |
| Treating the same word in different casing as different words | 'The' and 'the' take up separate positions, so the count for the is split between them and the vector no longer reflects the true frequency. | Apply one consistent casing rule during tokenization so that every occurrence maps to the same vocabulary entry. |
Common errors in Bag-of-Words processing and their effect on the count vectors
Common mistake Assuming word order is stored in the vector
What it looks likeBelieving 'the cat sat' and 'sat the cat' produce different vectors.
Why it failsEncoding only counts occurrences, so order is discarded during the process.
Do this insteadExpect identical vectors for any two documents with the same word counts.
Common mistake Using different vocabularies for different documents
What it looks likeGiving document 1 a vector of length 3 and document 2 a vector of length 4.
Why it failsEach document would have its own feature list, so position 0 could mean cat in one document and dog in another.
Do this insteadBuild a single shared vocabulary from all documents, giving every vector the same length.
Practice on a Third Document
Using the shared vocabulary cat, dog, down, sat, the from the two-document example, write the count vector for the third document 'the cat sat down'. Then state the vector for 'sat down the cat' without recounting from scratch, and explain your answer.
Hints
- Tokenize the third document first, then check each vocabulary word against its tokens.
- The vocabulary is fixed, so every position still means cat, dog, down, sat, and the in that order.
- Compare the two documents: they contain the same words, so their vectors must match.
What do you think happens?
What vector does 'the cat sat down' produce, and why does 'sat down the cat' produce the same one?
Reveal answer
[1, 0, 1, 1, 1]; the second sentence contains the same words with the same counts, so its vector is identical.
The counts are cat 1, dog 0, down 1, sat 1, and the 1. Word order is discarded during encoding, so rearranging the words does not change the vector.
Key Takeaways
- Bag-of-Words processing runs three stages in a fixed order: tokenization, vocabulary building, and encoding.
- Tokenization splits each document into tokens, usually on whitespace and punctuation, and later stages work only with these tokens.
- The vocabulary is one shared numbered list of every distinct word, so every document's count vector has one feature per unique word in the whole dataset.
- Encoding counts vocabulary words in each document, and word order is discarded, so sentences differing only in order produce identical vectors.
- Errors such as building per-document vocabularies, leaving punctuation attached, or inconsistent casing corrupt the vectors.