Training a Convnet from Scratch on a Small Dataset

From Archive to Training Set

Training a convolutional network from scratch begins before the model code runs. The image files must first be organized so the training process can find images by both subset and class. In the Dogs vs. Cats example, the original collection is reduced to a smaller working dataset and divided into training, validation, and test subsets.

The Dogs vs. Cats Collection

The original Dogs vs. Cats dataset contains 25,000 medium-resolution color JPEG images. It contains 12,500 cat images and 12,500 dog images. The dataset was made available by Kaggle for a computer-vision competition in 2013, and its compressed size is 543 MB.

The original collection is much larger than the smaller working dataset used in this exercise. The preparation process selects images and assigns them to separate subsets while preserving the two class categories: cats and dogs.

Three Separate Subsets

The working dataset is divided into training, validation, and test sets. Keeping these subsets separate gives the workflow distinct groups of images for the different stages of developing and assessing the convnet. Each subset must retain both categories, so every subset has a cats directory and a dogs directory.

Required Directory Tree

The directory structure mirrors the dataset split. There is a top-level working-dataset directory. Inside it are training, validation, and test directories. Inside each of those are two class directories: cats and dogs.

Moving Images into Place

Each selected image follows a simple path. First, the preparation process identifies whether the file belongs to the cat or dog class. Next, it uses the selected image range to determine whether the file belongs in the training, validation, or test subset. Finally, it copies the file into the matching class directory inside that subset.

Checking the File Counts

The first useful verification is a directory count. The source procedure uses os.listdir() to obtain the contents of a directory and len() to count those contents. Check all six class directories, not just one of them. A correct preparation should produce 1,000 training cats, 1,000 training dogs, 500 validation cats, 500 validation dogs, 500 test cats, and 500 test dogs.

import os

paths = [
    "train/cats",
    "train/dogs",
    "validation/cats",
    "validation/dogs",
    "test/cats",
    "test/dogs",
]

for path in paths:
    print(path, len(os.listdir(path)))

Preparation Mistakes

• Checking only the training directories

  The preparation is incomplete even though one subset looks correct.

  Fix: Count all six class directories: two inside train, two inside validation, and two inside test.

• Treating the original dataset count as the working-dataset count

  The working dataset is a smaller selection with the stated per-directory targets.

  Fix: Use the working-dataset targets: 1,000 images for each training class and 500 images for each validation and test class.

• Copying an image into a class directory without applying the subset selection

  The image may be placed in the wrong subset, making the directory structure inconsistent with the intended split.

  Fix: Apply both decisions: identify the class and use the selected image range to identify the subset.

• Starting model training before checking the folders

  A dataset-preparation failure can occur before any model code runs.

  Fix: Create the directory structure, copy the files, and verify the six counts first.

Check Your Dataset Plan

Preparation Checklist

1. The original Dogs vs. Cats dataset contains 25,000 medium-resolution color JPEG images: 12,500 cats and 12,500 dogs.
2. The smaller working dataset is divided into training, validation, and test subsets.
3. Every subset contains separate cats and dogs directories.
4. Selected image ranges determine which files are copied into each destination directory.
5. The expected counts are 1,000 training cats, 1,000 training dogs, 500 validation cats, 500 validation dogs, 500 test cats, and 500 test dogs.
6. os.listdir() and len() provide a direct way to inspect and count the contents of every class directory.