Sentiment Labels from Folder Names
The movie review dataset is provided as text files in two separate folders, one for training data and one for test data.
Why the Folder Layout Matters
Before you can analyze any movie reviews, you need to know where they are and how they are arranged on disk. A sentiment analysis project depends on loading the right files from the right places, so the layout of the dataset matters as much as the text inside it. In this article you will look at the folder structure of the movie review dataset and learn to read it from a directory listing.
What do you think happens?
The pos subfolder holds positive reviews and the neg subfolder holds negative reviews. Which folder would you open to find the reviews labeled positive?
Reveal answer
pos
The names describe the kind of review stored inside. Because pos holds positive reviews, the label of a review is recoverable from the name of the folder it sits in.
Tracing the Dataset Layout
The dataset is organized as a tree of folders. At the top is the folder data/aclImdb. Inside it are two data folders, one named test and one named train. Each of these two data folders in turn contains two subfolders, one called neg and one called pos. Read from the top down, the structure is always the same: one top-level folder, two data folders, and a pair of label folders inside each data folder.
Notice that the pos and neg pair appears inside both data folders. The train folder and the test folder each hold a neg subfolder and a pos subfolder, so the same naming pattern applies no matter which data folder you are looking at.
Keeping Training and Test Data Apart
After the data is unpacked, the reviews are provided as text files in two separate folders. One folder holds the training data and the other holds the test data. The two folders are kept apart so that data used to learn from is never confused with data used to check results afterward.
From Folder Name to Sentiment Label
Each review does not need a separate label written next to it. The name of the folder the review sits in carries the information. A review inside pos is a positive review, and a review inside neg is a negative review. The folder name is therefore the source of the sentiment label.
Labeling a review from its path
A review file sits in the folder data/aclImdb/train/neg. What sentiment label does this review carry, and which role does it play in the project?
- 1Identify the parent folder
The review's immediate parent folder is neg.
- 2Map the folder name to a label
neg holds negative reviews, so the review is labeled negative.
- 3Identify the data role
The path passes through train, so this review is part of the training data, kept apart from the test data.
The review is a negative review and belongs to the training data.
Reading the tree -L 2 Output
The source uses the tree command to display the directory structure. When you run it in a notebook, the command is written with a leading exclamation mark. That mark tells the notebook the line is a shell command rather than Python code. The -L 2 option limits the display to two levels deep, so the listing shows the data folders and their pos and neg subfolders, and then stops.
data/aclImdb
├── test
│ ├── neg
│ └── pos
└── train
├── neg
└── pos
6 directories, 0 files- The exclamation mark marks this line as a shell command.
- data/aclImdb is the root of the tree and is not counted in the total.
- The -L 2 option stops the display after the second level: test and train are level one, and neg and pos are level two.
The exact spacing in the sketch above is illustrative, but the shape and the final line are the part to read carefully. The final line confirms the layout: 6 directories and 0 files. The two data folders, test and train, account for 2 of those directories. The four subfolders, neg and pos inside each data folder, account for the other 4. The top-level folder data/aclImdb is the root and is not part of the count. The output lists folders only, which matches the 0 files in the final line.
Mistakes When Reading the Layout
Common mistake Counting the top-level folder in the directory total
What it looks likeAdding data/aclImdb to the 4 subfolders and reporting 5 directories.
Why it failsThe top-level folder is shown as the root of the tree and is not included in the 6 directories count.
Do this insteadCount only the test and train folders plus the four neg and pos subfolders, which gives 6.
Common mistake Assuming pos and neg exist in only one data folder
What it looks likeLooking for pos only under train and concluding the test data has no positive reviews.
Why it failsEvery data folder uses the same pair of subfolders, so both train and test contain pos and neg.
Do this insteadCheck each data folder separately, and expect the same pos and neg pair inside each one.
Common mistake Expecting the -L 2 listing to show individual review files
What it looks likeSearching the tree output for a file name and finding none.
Why it failsThe output shown here contains only folders and no files, and the -L 2 limit stops the display at the second level.
Do this insteadUse the listing to find folders. The folder structure is what this output reveals.
Common mistake Dropping the exclamation mark in a notebook cell
What it looks likeTyping tree -L 2 data/aclImdb as a line of Python code.
Why it failsThe leading exclamation mark marks the line as a shell command rather than Python code.
Do this insteadWrite the command as !tree -L 2 data/aclImdb in the notebook cell.
Practice Locating Folders
Using the tree -L 2 output from this article, give the full path, starting at data/aclImdb, of the folder that holds the training reviews labeled negative.
Hints
- Find the train folder at the first level under the root.
- Then look for the neg folder at the second level under train.
Key Takeaways
- The movie review dataset is provided as text files in two separate folders, one for training data and one for test data, kept apart so learning data is never confused with checking data.
- Each data folder contains two subfolders: pos for positive reviews and neg for negative reviews, and the same pair appears in both the train and test folders.
- A review's sentiment label can be read from the name of the folder it sits in, so the folder layout carries the labels.
- The tree command with the -L 2 option displays the structure two levels deep; the output in this article reports 6 directories and 0 files, with the top-level data/aclImdb folder not included in the count.