Representing Text Data for Classification
Machine learning models need text converted into numbers, and the bag-of-words representation is a simple and effective way to do so.
A Spam Filter That Cannot Read
Machine learning models work with numbers. A spam filter, a sentiment classifier, or a topic tagger cannot read a review the way a person does. Before a model can learn anything from a text, every text has to be turned into a numeric form. The central question is which parts of the text to keep when making that conversion.
One of the simplest and most widely used answers is the bag-of-words representation. It is simple enough to compute by hand and effective enough to appear in many practical systems. As you read, keep one question in mind: if we deliberately ignore most of what makes a text readable, what is left, and is it still useful?
What do you think happens?
Do the sentences 'dog bites man' and 'man bites dog' produce the same numeric representation under bag-of-words?
Reveal answer
Yes, they produce identical counts.
Bag-of-words discards word order. Both sentences contain one 'dog', one 'bites', and one 'man', so their tallies match exactly. This is the main limitation of the representation.
What the Bag Keeps and What It Throws Away
The name comes from a mental image. Take a paragraph, cut out every word, and drop the words into a bag. The sentences, the paragraph breaks, the chapter headings, and the formatting are all gone. What remains is a bag of individual words with no record of which word came first or which sentence it belonged to.
Discarding order and layout is a deliberate trade. The representation does not try to preserve meaning carried by grammar or word order. It asks only how often each word shows up. A bag can be described completely by a tally of its contents, and that tally is exactly what the representation stores.
- Discarded: word order, sentence boundaries, paragraph breaks, headings, and formatting
- Discarded: which words were close to one another in the text
- Kept: a count of how often each word appears in each text of the corpus
- Kept: a separate tally for every text, so each text gets its own set of numbers
Reading One Count Vector
The representation is applied to a corpus, which is a collection of documents. First, the words that appear across the corpus are listed in a fixed order. This list is the vocabulary. Each position in the vocabulary corresponds to one word. For each text, the representation then records how many times each vocabulary word appears in that text. The resulting row of numbers is the count vector for that text.
The source illustrates this with the single text 'the cat sat on the mat'. The word 'the' appears twice, and every other word appears once. The tally therefore has a 2 in the position for 'the' and a 1 in each of the other positions for that text. Nothing in this tally says that 'cat' came before 'mat' or that 'the' opened the sentence. That missing information is the cost of the representation, and the count is the benefit.
Building the Count Table by Hand
Work through a small corpus of three texts. Each text is written in lowercase, and each word is counted separately. Follow the steps in order: first list the vocabulary, then count each text against that list, then arrange the results as a table with one row per text.
Counting a Three-Text Corpus
Corpus: T1 'the dog bites the man', T2 'the man bites the dog', T3 'the cat sat on the mat'. Compute the count vector for each text, with vocabulary in alphabetical order.
- 1Step 1: List the vocabulary
Collect every distinct word across the corpus: bites, cat, dog, man, mat, on, sat, the. That gives 8 positions.
- 2Step 2: Count T1
T1 contains bites once, dog once, man once, and the twice. It has no cat, mat, on, or sat. Its vector is [1, 0, 1, 1, 0, 0, 0, 2].
- 3Step 3: Count T2
T2 uses the same words as T1 with different order: man, bites, the, dog, the. Its counts are bites once, dog once, man once, and the twice. Its vector is [1, 0, 1, 1, 0, 0, 0, 2].
- 4Step 4: Count T3
T3 contains cat once, mat once, on once, sat once, and the twice. It has no bites, dog, or man. Its vector is [0, 1, 0, 0, 1, 1, 1, 2].
- 5Step 5: Arrange the table
Place each vector as one row. Columns are the vocabulary words; rows are the texts. Rows T1 and T2 are identical even though the texts differ in order.
Count table: T1 [1, 0, 1, 1, 0, 0, 0, 2]; T2 [1, 0, 1, 1, 0, 0, 0, 2]; T3 [0, 1, 0, 0, 1, 1, 1, 2].
Vocabulary (8 positions): bites, cat, dog, man, mat, on, sat, the
T1 'the dog bites the man' -> [1, 0, 1, 1, 0, 0, 0, 2]
T2 'the man bites the dog' -> [1, 0, 1, 1, 0, 0, 0, 2]
T3 'the cat sat on the mat' -> [0, 1, 0, 0, 1, 1, 1, 2]Mistakes When Reading Bag-of-Words
The representation is easy to describe and easy to misremember. The following misreadings come up often when learners first meet it. Each one is checked against what the representation actually stores.
Common mistake Assuming the representation understands what a text means
What it looks likeBelieving that a count vector knows that a review is positive because it contains praise.
Why it failsThe representation does not try to preserve meaning carried by grammar or word order. It only records how often each word appears.
Do this insteadTreat the vector as a tally of words, not a summary of meaning. Any meaning that depends on arrangement is not present in the input.
Common mistake Believing that word order is kept in some hidden way
What it looks likeExpecting 'dog bites man' and 'man bites dog' to receive different vectors.
Why it failsOrder is discarded, so texts that differ only in arrangement receive identical counts.
Do this insteadWhen two texts have different meanings but the same words, expect identical vectors and plan for that limitation.
Common mistake Assuming that words with similar meanings share a position
What it looks likeExpecting a synonym of 'cat' to be counted in the same column as 'cat'.
Why it failsEach distinct word in the vocabulary has its own position. The representation counts how often each word appears; it does not merge words based on meaning.
Do this insteadRead each column as one exact word. Two different words always occupy two different positions.
Common mistake Thinking only whether a word appears matters, not how often
What it looks likeReading the 2 in the 'the' position of T3 as if it were just a 1.
Why it failsThe representation stores how often each word shows up in each text, so repeated words carry a larger number.
Do this insteadRead each entry as a count. A value of 2 means the word occurred twice in that text.
Common mistake Forgetting that the vocabulary order must be fixed across all texts, so that each column means the same word in every row
Practice Counting a New Corpus
Use the vocabulary bites, cat, dog, man, mat, on, sat, the (in that order). Compute the count vector for the text 'the cat bites the dog' and for the text 'dog sat on the mat'. Then state whether these two vectors are identical and explain your answer in terms of what bag-of-words discards.
Hints
- Go through the vocabulary one word at a time and count how many times that word appears in the text.
- The word 'the' appears twice in the first text; check the second text carefully before you assign counts.
- Identical vectors would mean the representation cannot tell the two texts apart; consider whether they share the same words.
What do you think happens?
Before computing, predict: will 'dog bites the man' and 'the man bites dog' share the same count vector?
Reveal answer
Yes. Both texts contain bites once, dog once, man once, and the twice, so their vectors match.
The only difference between the texts is word order, which bag-of-words discards.
Key Takeaways
- Machine learning models need text converted into numbers, and bag-of-words is a simple, effective way to do so.
- Bag-of-words discards order, sentences, paragraphs, and formatting, keeping only how often each word appears in each text.
- Each text in a corpus becomes a count vector aligned to a shared vocabulary, so every row has one position per vocabulary word.
- Texts that differ only in word arrangement receive identical counts, which is the main limitation to remember when evaluating a model.