Loading Movie Review Text Files
A sentiment analysis project depends on loading the right files from the right places, so the layout of the dataset matters as much as the text inside it.
Why the Folder Layout Matters
Before you can analyze any movie reviews, you need to know where they are and how they are arranged on disk. A sentiment analysis project depends on loading the right files from the right places, so the layout of the dataset matters as much as the text inside it. In this article you will look at the folder structure of the movie review dataset and learn to read it from a directory listing.
Think of the dataset as a set of labeled boxes. If you open the wrong box, or open the right box but take files from the wrong one, your analysis will be built on the wrong reviews. Before writing any loading code, it helps to trace the layout from the top of the dataset down to the individual review files. The sections below do exactly that, starting with the outermost folder and working inward.
Tracing the Dataset Layout Top-Down
The movie review dataset is provided as text files in two separate folders, one for training data and one for test data.
- 1Start at the top-level folder, data/aclImdb. It is the root of the tree and holds the two data folders.
- 2Look one level down. You will find two folders, one named train and one named test.
- 3Look one level below each of those. Each data folder contains two subfolders, one named pos and one named neg.
- 4Stop there. The pos and neg folders are the lowest level this listing needs, and they are where the review files for each label are stored.
The two data folders are kept apart on purpose. After the data is unpacked, one folder holds the training data and the other holds the test data. Keeping them separate means the data used to learn from is never confused with the data used to check results afterward.
What the pos and neg Folders Hold
Each of the two data folders has the same pair of subfolders, named pos and neg. The names describe the kind of review stored inside: pos holds positive reviews and neg holds negative reviews. Because every data folder uses the same pattern, the same naming applies whether you are looking at training data or test data. The train folder's pos subfolder and the test folder's pos subfolder both hold positive reviews; only the data folder they sit inside changes.
| Folder | Purpose | Contains |
|---|---|---|
| train | Data used to learn from | pos and neg subfolders of reviews |
| test | Data used to check results afterward | pos and neg subfolders of reviews |
| pos | Positive reviews | Review text files labeled positive |
| neg | Negative reviews | Review text files labeled negative |
How the two data folders and their subfolders relate
The folder name carries the label. To load positive examples from the training data, you point your code at the train folder's pos subfolder. To load negative examples from the test data, you point it at the test folder's neg subfolder.
Reading the tree -L 2 Listing
The source uses the tree command to display the directory structure. When you run it in a notebook, the command is written with a leading exclamation mark. That mark tells the notebook the line is a shell command rather than Python code. The -L 2 option limits the display to two levels deep, so the listing shows the data folders and their pos and neg subfolders, and then stops.
data/aclImdb
├── test
│ ├── neg
│ └── pos
└── train
├── neg
└── pos
6 directories, 0 files- The leading exclamation mark sends the line to the shell instead of Python.
- tree draws the folder structure as an indented listing.
- -L 2 limits the display to two levels below the root folder.
- data/aclImdb is the root of the tree and is not counted in the total.
The output contains only folders and no files, which the final line confirms. The count is 6 directories and 0 files. Check it against the layout: the two data folders, test and train, account for 2 directories, and the four subfolders neg and pos inside them account for the other 4. The top-level folder data/aclImdb is the root of the tree and is not included in the total.
Locating the pos folder inside train
Using the tree -L 2 listing, where is the pos folder that holds positive training reviews?
- 1Find the data folder that holds training data
The listing shows train as one of two folders one level below data/aclImdb.
- 2Look one level further down inside train
Inside train there are two subfolders, neg and pos.
- 3Select the pos entry
The pos folder under train is the location of the positive training reviews, reached as train then pos.
The positive training reviews are in the pos folder inside train, which sits inside data/aclImdb.
Mistakes Beginners Make With the Layout
Common mistake Counting data/aclImdb as one of the directories in the total
What it looks likeAdding the root folder to the 2 data folders and 4 subfolders and expecting a total of 7.
Why it failsThe tree output reports 6 directories, and the root is shown as the top of the tree and is not included in that count.
Do this insteadCount only the folders below the root: train, test, and the four pos and neg subfolders.
Common mistake Assuming pos and neg mean different data folders
What it looks likeThinking the pos folder is a separate split from train and test.
Why it failspos and neg are subfolders inside each data folder, and every data folder uses the same pair.
Do this insteadRead the path as data folder first, then label: train/pos or test/neg.
Common mistake Expecting review files in the listing
What it looks likeLooking for individual review text files in the tree -L 2 output.
Why it failsThe output contains only folders, and the final line reports 0 files.
Do this insteadUse a deeper listing or open the pos and neg folders in code to find the individual review files.
Common mistake Forgetting the leading exclamation mark when running tree in a notebook, which sends Python an invalid line instead of a shell command.
Check Your Reading of the Layout
What do you think happens?
If the tree -L 2 listing of the dataset is read again, how many directories are counted in total, and does the root folder data/aclImdb count toward that number?
Reveal answer
6 directories, and the root data/aclImdb is not counted.
The two data folders account for 2 directories and the four pos and neg subfolders account for the other 4. The root is the top of the tree and is excluded from the total.
Without looking back, trace the path to the negative test reviews. Write the folder path from the dataset root down to the neg folder that holds negative reviews for testing.
Hints
- Start at data/aclImdb, then choose the data folder for testing rather than learning.
- The label folder for negative reviews is named neg.
Key Takeaways
- The movie review dataset is provided as text files in two separate folders: one for training data and one for test data, kept apart so learning data is never confused with checking data.
- Each data folder contains two subfolders, pos for positive reviews and neg for negative reviews, and the same pattern repeats in both data folders.
- The tree command with the -L 2 option displays the folder structure up to two levels deep; in a notebook it is run with a leading exclamation mark.
- The tree output for this dataset reports 6 directories and 0 files; the top-level data/aclImdb folder is the root and is not counted.
- To locate a folder, read the path from the root down: data folder first, then label subfolder, such as train/pos.