Interpreting Deep Learning Models
Visualization makes convnet learning and classification decisions easier to inspect.
From Label to Evidence
A convolutional network can produce a class decision, but the class label alone does not show what the network learned or what visual evidence influenced that decision. Visualization makes the network's learning and classification process easier to inspect. The goal is not merely to display a result. It is to examine the route from the input image, through the network's convolutional layers, to the final class decision.
A useful interpretation asks two related questions: what visual patterns does the convnet learn, and how do those patterns relate to a particular classification decision?
The Route Through the Network
Begin with the complete route rather than with an isolated visualization. An image enters the convnet. Visual information is processed through convolutional layers. The network then produces a class decision. At every stage, ask not only what output the layer produces, but also what visual information that output represents for the next stage. This creates a state-by-state view of the path toward the decision.
Tracing One Classification
An illustrative convnet receives an image and produces a class decision. How should the interpretation be organized?
Inspect the input: Start with the image that entered the convnet. The image is the source of the visual information being processed.
Inspect layer outputs: Follow the visual information through convolutional layers. For each layer, ask what its output represents for the next stage.
Inspect the decision: Record the class selected by the network, then relate that outcome to the visual evidence observed along the route.
The interpretation is a trace from input image to convolutional representations to class decision, not a list of visualization names detached from the route.
Reading Feature Maps
Feature-map visualizations show which patterns activate across network layers. Early layers can be examined for simpler visual patterns such as edges. Deeper layers can be examined for more developed patterns, including object parts. Comparing layers helps you observe how the representation changes as the image moves through the convnet.
For an illustrative feature-map reading, an early-layer visualization might highlight edge-like patterns, while a deeper-layer visualization might highlight more developed patterns, including parts of an object. The important observation is the change in representation across layers, not the claim that one isolated map completely explains the final decision.
Choosing the Right View
Different visualizations answer different questions about a convnet decision. Saliency and attribution maps focus on which pixels or regions of the input contributed most to the decision. Activation maximization reverses the usual inspection direction: instead of starting with an image and asking what activates, it asks what input pattern would produce the strongest activation for a particular neuron or class. Comparative views examine why the convnet favors one class over another and whether the visual evidence differs between those predictions.
| View | Question it asks | What to inspect |
|---|---|---|
| Feature map | What patterns activate at this layer? | The changing representation across layers |
| Saliency or attribution | Which parts of the input contributed to the decision? | Highlighted pixels or regions |
| Activation maximization | What input pattern would produce the strongest response? | A pattern associated with a neuron or class |
| Comparison | Why is one class favored over another? | Differences in visual evidence between predictions |
The useful visualization depends on the question being asked about the convnet.
Name the question before choosing the visualization. If the question concerns changing internal representations, compare feature maps across layers. If it concerns input evidence, inspect saliency or attribution. If it concerns a preferred response pattern, use activation maximization. If it concerns competing classes, compare the explanations.
Decision and Explanation
The classification decision is the class selected by the convnet. An explanation is a visual inspection of information related to that decision, such as activated patterns, highlighted input regions, or differences between class predictions. The explanation helps examine the decision, but it is not the decision itself.
Practice the Trace
A convnet has selected a class for an image. You want to investigate both how its internal representation changes and which parts of the input are associated with the decision. Describe the order of your investigation and choose the visualization for each question.
Hints
- Begin with the complete route from input image to class decision.
- Use feature maps to compare activated patterns across layers.
- Use saliency or attribution when the question concerns input pixels or regions.
- Treat the selected class as the outcome and the visualizations as evidence used to inspect it.
A Strong Interpretation Plan
How can the investigation answer both what the convnet learns and how it reaches a decision?
Establish the outcome: Record the class selected by the convnet. This is the decision being examined.
Trace internal change: Compare feature-map visualizations across layers, looking for simpler patterns in early layers and more developed patterns, including object parts, in deeper layers.
Inspect input evidence: Use saliency or attribution to examine which pixels or regions are associated with the selected class.
Add contrast when needed: If the question is why one class is favored over another, use a comparative view to examine differences between the predictions.
The explanation connects the class decision with both the changing internal representation and the visual evidence associated with the input.
Summary
- Visualization helps inspect both what a convnet learns and how it makes classification decisions.
- A complete interpretation traces visual information from the input image through convolutional layers to the final class decision.
- Feature maps reveal how activated patterns change across network depth, from simpler patterns such as edges to more developed patterns including object parts.
- Saliency and attribution focus on input pixels or regions, activation maximization asks what pattern produces a strong response, and comparison views examine competing class explanations.
- The classification decision is the outcome; visual explanations are evidence used to inspect and interpret that outcome.
Key Takeaways
- Visualization turns a convnet classification from an isolated label into a traceable path from image input to class decision.
- Feature-map comparisons show how the patterns represented by the network change across layers.
- Saliency, attribution, activation maximization, and comparison views answer different questions and should be selected accordingly.
- An explanation provides visual evidence about a decision; it is not the decision itself.