Understanding String Data Types in Machine Learning
Freeform text data consists of phrases or sentences.
Text That Does Not Fit in a Table Cell
Many datasets are neat tables, with one column for each measurement and one row for each record. Text data is different. A tweet, a chat message, or a hotel review is written in ordinary language, and its length and shape change from one example to the next. This kind of data is often called freeform text. Before a model can learn anything from it, you need a clear way to talk about what the data is and how it is organized. This article introduces two terms used in text analysis for that purpose: corpus and document.
What do you think happens?
A file contains 5,000 hotel reviews, one review per line. Is the whole file best described as a single document or as a corpus?
Reveal answer
A corpus
The source describes a dataset of text in text analysis as a corpus. A collection of reviews such as hotel reviews can be discussed as a corpus, while one review used as one data point is a document.
What Freeform Text Is Made Of
Freeform text data consists of phrases or sentences. Unlike a fixed set of short categories or numbers, each piece of freeform text is a stretch of natural language written as phrases and sentences.
The source names several everyday examples of freeform text data: tweets, chat logs, and hotel reviews. Each of these is a piece of writing that a person produced in natural language. A single tweet may be one short sentence or a few phrases, while a hotel review is usually a longer block of several sentences. Both are freeform text, even though they differ in length.
Tracing a Text Collection Down to One Text
The most useful habit when you meet a text dataset is to ask a single question first: are we talking about the whole dataset, or about one text within it? Follow the trace below, starting at the top level and moving down one level at a time.
- 1State 1: You are given a folder of hotel reviews. This is the whole dataset, so the level you are talking about is the collection. In text analysis this collection is called a corpus.
- 2State 2: You open the folder and pick out one review, such as a single guest's comment about a room. This one text used as one data point is a document.
- 3State 3: You notice that the same vocabulary applies at another scale. A collection of tweets, a collection of chat logs, or a collection of hotel reviews can each be discussed as a corpus, and each individual tweet, chat log entry, or review is a document.
- 4State 4: You consider a much larger collection, such as Wikipedia or Project Gutenberg's collection of 50,000 ebooks. These can also be corpora. A single article or a single ebook used as one data point is again a document.
Corpus as the Whole Dataset
In text analysis, the dataset is often called a corpus. A corpus is the whole collection of text that is being analyzed. It can be small, such as a set of reviews, or very large, such as a Wikipedia-scale collection or Project Gutenberg's collection of 50,000 ebooks.
The word corpus describes the level of the whole. When someone says they are working with a corpus of chat logs, they mean the entire set of chat logs taken together, not any one conversation within it.
Document as One Data Point
Each data point represented as a single text is called a document. A document is one text treated as one item in the dataset.
The key word is single. A document is not defined by how long it is or what it is about. It is defined by the role it plays: when one text is used as one data point, it is a document. That is why a single hotel review, a single tweet, or a single chat log entry can each be called a document.
Corpus Versus Document Side by Side
The two terms describe the same kind of material at two different levels. The table and diagram below set them against each other so you can see which level a given description refers to.
| Question | Corpus | Document |
|---|---|---|
| What does it refer to? | The whole dataset of text | One text used as one data point |
| How many texts does it contain? | Many texts | One text |
| Example from the source | A collection of tweets, chat logs, or hotel reviews | A single tweet, a single chat log, or a single hotel review |
| Can it be a large collection? | Yes, such as Wikipedia or Project Gutenberg's 50,000 ebooks | Yes, a single ebook can be one document |
Corpus and document describe the same text at different levels of the dataset.
Classifying Text Examples
Is it a corpus or a document?
Classify each of these as a corpus or a document: (a) a set of hotel reviews collected from a booking site, (b) one tweet, (c) Project Gutenberg's collection of 50,000 ebooks, (d) one chat log entry.
- 1Step 1: Find the level being described
For item (a), the description is about a set of reviews, which is the whole collection. This is a corpus.
- 2Step 2: Check for a single text
For item (b), the description is one tweet used as one data point. This is a document.
- 3Step 3: Apply the same test to a large collection
For item (c), the description is the collection of 50,000 ebooks as a whole. The source notes that larger collections such as this can also be corpora, so this is a corpus.
- 4Step 4: Apply the test to a single entry
For item (d), the description is one chat log entry used as one data point. This is a document.
(a) corpus, (b) document, (c) corpus, (d) document.
Mistakes Beginners Make With These Terms
Common mistake Calling a whole collection of reviews a document.
What it looks likeSaying 'this document has 5,000 reviews in it.'
Why it failsA document is one text used as one data point. A set of 5,000 reviews is the whole dataset, which is a corpus.
Do this insteadSay 'this corpus contains 5,000 reviews,' and reserve document for a single review.
Common mistake Judging whether something is a document by its length.
What it looks likeThinking a long ebook cannot be a document because it is too big.
Why it failsThe source defines a document by its role as a single data point, not by size. A large ebook used as one data point is still a document.
Do this insteadAsk whether one text is being used as one data point. If it is, call it a document, whatever its length.
Common mistake Assuming a corpus must be very large.
What it looks likeBelieving that a set of ten chat logs cannot be called a corpus.
Why it failsThe source states that a collection of text can be large or small, such as a set of reviews or a chat log collection.
Do this insteadAny collection of texts analyzed together can be discussed as a corpus, regardless of its size.
Common mistake Confusing freeform text with a fixed set of categories. Freeform text consists of phrases or sentences, so the content is natural language rather than a short list of labels.
Practice Sorting Real Text Datasets
A research team has a folder with 2,000 tweets about a product launch. For each item below, say whether it is a corpus or a document: (1) the folder of 2,000 tweets, (2) one tweet from that folder, (3) a single chat log collection of many conversations, (4) one hotel review taken from a larger review set. Explain your reasoning using the question: am I looking at the whole dataset, or one text within it?
Hints
- Start by asking whether the description covers the whole collection or one text.
- The source says a document is each data point represented as a single text.
- A collection of texts, even a small one, can be discussed as a corpus.
Key Takeaways
- Freeform text data consists of phrases or sentences, and it can take forms such as tweets, chat logs, and hotel reviews.
- In text analysis, the dataset is often called a corpus, and it can be large, such as Wikipedia, or small, such as a set of reviews.
- Each data point represented as a single text is called a document, regardless of how long that text is.
- When you meet a text dataset, first ask whether you are talking about the whole collection (corpus) or one text within it (document).