Tokenization Scheme

A tokenization scheme is a method for splitting text into individual words or tokens. The simplest version splits on spaces. That works until the first comma appears. Then the word before the comma carries the punctuation with it, and the model sees a different unit than the one you intended. Consider the sentence Is it well-known, really? A space-only splitter returns the word well-known, with its comma still attached, as if the comma were part of the word.

The tokenizer in this article takes a different approach. It does not look for words directly. It looks for the characters that separate words, and it asks the splitting function to keep those characters in the output. That single decision is what turns punctuation into tokens of its own. Commas, periods, question marks, quotation marks, apostrophes, double dashes, and spaces are all treated as separators.

The pattern is a single string of symbols. A bracketed group lists the characters that count as separators on their own. Because the list includes the period, the comma, the question mark, and the apostrophe, each of these is matched as its own separator rather than fused onto the neighboring word. The pattern also contains two more alternatives: a double dash, written as a two-character sequence so it is matched as a unit, and a single space. Both alternatives are separators.

The key detail is the parentheses around the whole pattern. They form a capturing group. When re.split is given a pattern with a capturing group, it keeps the text that matched the separators in the output list, interleaved with the words. Without the group, the separators are thrown away. Here is the same pattern run both ways, with the parentheses present and then removed.

Read the first output from left to right. The words and separators alternate, and the order matches the sentence. The space after Is is the separator between Is and it. The comma sits directly after well-known, and an empty string sits between the comma and the space that follows it. That empty string is a side effect of two adjacent separators, and it has to be removed later. The period and question mark follow the same rule: each one becomes its own item in the list, in its original position.

Because the capturing group keeps the separators, the raw list also contains the spaces and the empty strings. Removing whitespace requires a cleanup step that strips each item and drops the ones that are empty. Without this step, spaces and blanks remain in the output and the token list is noisy. Stripping turns a space item into an empty string, and the filter then removes it along with the existing empties.

The whitespace decision is not a technical requirement. It depends on the application. The simple tokenizer in this section removes whitespace, and the source says it does so for simplicity and brevity of the tokenized output. Removing whitespace reduces memory and computing requirements. Keeping whitespace can matter when a model is sensitive to the exact layout of the text, and the source names Python code as an example. A later scheme in the course will include whitespace, so the current code should be treated as a deliberate simplification rather than a universal rule.

The source notes that the scheme works on a simple sample and is then modified to handle other punctuation and special characters. Extending the scheme means adding symbols to the bracketed list. Before you do that, predict what will happen to a character that is not yet in the list. A hyphen is a good test case. It is not in the list, even though the double dash is. A single hyphen therefore stays inside its word, and anything you want treated as a separator must be added explicitly.

This part of the module does not teach a new procedure. It points you to original research papers. The section lists three references. Two describe tokenization schemes, and one describes an attention technique. The table below records the facts the source gives for each: the name, the year, the authors, and the kind of contribution. The last column is what separates the tokenizers from the attention method.

A subword tokenizer does not split only at spaces and punctuation. It breaks words into smaller pieces that can be reused across the vocabulary. SentencePiece is described in the source as a simple, language-independent subword tokenizer and detokenizer for neural text processing, used by some popular LLMs. The phrase language-independent means that its description is not tied to one particular language. The source does not give the internal method, so this article does not describe it further.

WordPiece is also described as a tokenization scheme used by some LLMs. The 2020 paper Fast WordPiece Tokenization by Song et al. focuses on making WordPiece tokenization faster. The source confirms that the work presents a fast tokenization method but does not give its details.

Bahdanau attention is a different kind of thing. It is an attention technique for RNN-based neural machine translation, and it is not a tokenizer. In the tokenization pipeline, the tokenizer decides what the units are, and the attention technique later decides how strongly the model weighs those units against each other. The source lists neural machine translation as a prerequisite, so this paper is best read after you have a working picture of translation models.

Before you open any of the three papers, fill in the table of facts for each one: name, year, authors, and type of contribution. Then ask what question you should bring to the paper. For a tokenizer, ask how it decides where to split and how it handles whitespace and punctuation. For the attention paper, ask how it relates the tokens to one another.