Loading Text Data into Python
A token is not the same thing as a line, a sentence, or a whole file. Tokenization works on the text and produces smaller pieces, which are words and special characters.
Raw Text Before Tokenization
A large language model does not read a book the way a person does. Before any learning can happen, the text has to be broken into smaller pieces that a model can work with. This article focuses on the first of those pieces of work: tokenization, the process of splitting input text into individual tokens. The source describes tokenization as a required preprocessing step for creating embeddings for an LLM.
What do you think happens?
You read a text file into a Python variable. Is the content now already a list of tokens?
Reveal answer
No, the variable holds raw text and tokenization is a separate step.
Reading a file into a variable gives you raw text, not tokens. Tokenization is a separate step that happens after the text is loaded. Keep this distinction in mind for the rest of the article, because most confusion about tokenization comes from mixing up the raw text you start with and the tokens that come out of splitting it.
Loading the File as Raw Text
The text used in the source is The Verdict, a short story by Edith Wharton, which has been released into the public domain. The source notes that you can copy the text into a plain text file, and the file used there is named the-verdict.txt. Python's standard file reading utilities can load that file into a variable.
open()opens the plain text file for readingf.read()returns the entire contents of the file as one stringraw_textnow holds raw text, not tokens- The slice
raw_text[:100] shows only the first 100 characters
Run code like this and study the output before going further. Look at what the printed slice shows. You are looking at the material you will tokenize, not yet at tokens. The loaded variable is the starting point of the pipeline, and it is the thing that tokenization will split.
What Counts as a Token
Tokenization is the process of splitting input text into individual tokens. A token is either an individual word or a special character, including punctuation characters.
The definition has two parts that matter. The first is that a token can be a word. The second is that a token can be a special character, and punctuation marks such as commas, periods, and question marks count as special characters. This means a sentence's punctuation does not disappear during tokenization. Each punctuation mark is one of the pieces the text is broken into.
A token is not the same thing as a line, a sentence, or a whole file. Tokenization works on the text and produces smaller pieces, which are words and special characters.
Where Tokenization Sits in the Pipeline
The source presents tokenization as one stage in a sequence of text processing steps that surround an LLM. The sequence starts with the input text. Tokenization happens directly after the input text and before the token IDs are produced. Token embeddings come after the token IDs, and the text reaches the transformer only after these steps. Read the diagram from left to right to follow the order.
Splitting a Sentence by Hand
The best way to see what tokens are is to split a short sentence yourself. Use this sentence, which is an original example written for this article: Hello, world! Apply the definition from earlier. Each word is a token, and each punctuation mark is a token too.
Tokenizing Hello, world! by hand
Split the sentence Hello, world! into tokens, and note the position of each token in the sequence.
- 1Find the words
Hello and world are individual words, so each one is a token.
- 2Find the special characters
The comma after Hello and the exclamation mark after world are punctuation characters, so each one is also a token.
- 3Order the tokens
Reading left to right, the tokens are Hello, the comma, world, and the exclamation mark. Numbering from zero gives positions 0, 1, 2, and 3.
Four tokens: Hello (position 0), a comma (position 1), world (position 2), and an exclamation mark (position 3).
Notice that the comma and the exclamation mark keep their own positions. Punctuation is not attached to the word beside it in this split. Keeping track of position is useful because a model works with an ordered sequence of pieces, and the position of each token in that sequence is part of the picture.
Where a Hand Split Diverges
A hand split is a useful model, but it can disagree with what a real tokenizer produces. Try this original sentence: It's late, isn't it? A simple split on spaces and punctuation gives It, an apostrophe, s, late, a comma, isn, an apostrophe, t, it, and a question mark. Whether a tokenizer keeps a contraction such as It's or isn't together, or divides it at the apostrophe, is a decision the source does not describe. You should not assume your hand split matches a tokenizer's output without checking that tokenizer's behavior.
- Contractions such as isn't, where the apostrophe could split one word into several pieces or keep it whole
- Compound words, where a space-and-punctuation split may not match how the text is actually divided
- Rare or unusual words, where your split may produce pieces you did not expect
Mistakes Beginners Make
Common mistake Treating the loaded variable as already tokenized
What it looks likeAfter reading the-verdict.txt into raw_text, assuming raw_text is a list of words and punctuation marks.
Why it failsReading a file into a variable gives you raw text. Tokenization is a separate step that happens after loading.
Do this insteadKeep the raw text and the tokens as two different stages, and only treat the output of the tokenization step as tokens.
Common mistake Dropping punctuation when listing tokens
What it looks likeListing the tokens of Hello, world! as only Hello and world.
Why it failsThe source defines a token as either a word or a special character, including punctuation characters. The comma and exclamation mark are tokens too.
Do this insteadCount every punctuation mark as its own token and give it a position in the sequence.
Common mistake Assuming a hand split always matches the tokenizer
What it looks likeConcluding that It's must be three tokens because the apostrophe is a special character.
Why it failsThe source does not describe how a tokenizer divides contractions, so a simple rule may not match real output.
Do this insteadCheck the behavior of the specific tokenizer you are using before relying on your own split.
Practice Splitting a Sentence
Split the sentence Wait, is it done? into tokens by hand. List each token in order, give each one a position starting from zero, and state which tokens are punctuation. Then identify one place where your split might differ from a real tokenizer and explain why you cannot be sure from the source alone.
Hints
- A token is either a word or a special character, including punctuation.
- Count the comma and the question mark as separate tokens.
- The source does not describe how a real tokenizer divides every word, so think about which words in your sentence could be divided in more than one way.
Key Takeaways
- Tokenization is the process of splitting input text into individual tokens, and it is a required preprocessing step for creating embeddings for an LLM.
- A token is either an individual word or a special character, including punctuation characters.
- Reading a file into a variable gives you raw text, not tokens. Tokenization is a separate step that happens after loading and before token IDs and token embeddings are produced.
- A hand split of a sentence is a useful model, but the source does not describe how a real tokenizer divides contractions, compound words, or rare words, so verify before assuming a match.