Sequence Processing with Convnets

Start with the Data Path

When studying a deep learning application, begin with the task's data path rather than its model name. Ask two questions: What kind of data enters the model, and what kind of data leaves it? The answers form an input-output modality mapping. This viewpoint is useful for organizing applications, including tasks involving sequences, without assuming that a particular architecture is required.

Trace an Application

1. Identify the data entering the model.
2. Identify the data produced by the model.
3. Write the input modality followed by the output modality.
4. Treat the resulting mapping as an organizational label, not as a complete description of the model.

The classification process has two ends. First, determine whether the input is vector data, image data, timeseries data, text, or a combination of modalities. Next, determine whether the output is vector data, image data, text, or another relevant modality. Only after both ends are known can the task be assigned a mapping category.

Compare the Mapping Categories

A vector-to-vector task connects vector data at both ends. An image can instead produce a vector, text, or another image, so the input label alone is insufficient. Timeseries data is listed as mapping to vector data. Text can map to text or images. Multimodal tasks differ because more than one input type participates, such as image and text or video and text before text is produced.

Follow a Sequence Mapping

This classification is the useful first step when discussing sequence processing with convolutional networks. The source material identifies the timeseries-to-vector relationship, but it does not define the internal operations of a convolutional network. Therefore, do not infer convolution steps, sequence positions, or architectural details from the mapping label alone.

Keep Generalization Separate

The mapping category describes the relationship between data types, not the full behavior of a model. Knowing that a model maps images to text does not establish how well it will perform on data that differs from its training data. The same caution applies to other mappings, including timeseries-to-vector and multimodal mappings.

Mistakes in Task Classification

• Classifying a task only by its input.

  An image can map to a vector, text, or another image. Those are different application categories.

  Fix: Always identify both the input modality and the output modality.

• Treating the mapping as the model architecture.

  The mapping labels the data relationship; it does not by itself specify the model architecture.

  Fix: Describe the input-output mapping first, then discuss architecture separately when the available information supports it.

• Assuming that a known mapping guarantees generalization.

  The mapping does not establish how well the model performs beyond its training data.

  Fix: Treat the mapping as an organizational tool rather than a complete description of model behavior.

• Ignoring multimodal inputs.

  More than one input type participates, so the task belongs to a multimodal mapping.

  Fix: Record every input modality before naming the output relationship.

Practice the Two-Ended Test

Key Takeaways

1. Classify a deep learning task by examining both what enters the model and what it produces.
2. Vector, image, timeseries, text, and multimodal data can participate in different input-output mappings.
3. A timeseries-to-vector task is identified from its data ends; the mapping alone does not specify the internal architecture.
4. Image-to-image, image-to-text, and image-to-vector tasks differ because their output modalities differ.
5. A modality mapping organizes an application but does not guarantee generalization beyond the training data.