Representation Learning
Deep learning extends machine learning by emphasizing successive learned representations.
From Data to Representations
Deep learning extends machine learning by emphasizing successive learned representations. Instead of asking one learned representation to carry an entire task, a deep-learning model learns a succession of representations from data. The central idea is therefore not simply that a model is large or complicated. The important change is how learning is organized across layers.
The word representation refers to how data is expressed at a particular learned layer. Deep learning emphasizes a sequence of these learned expressions rather than a single learned representation.
Tracing Layered Learning
An Abstract Layer Sequence
Trace what happens when abstract input data is processed by a model that learns several successive representations.
Start with input data: The model receives data that must be represented for a task.
Learn the first representation: A neural-network layer learns a representation of the data.
Pass the representation onward: The representation produced by one layer becomes the input to the next layer.
Continue through later layers: Additional layers learn further representations, creating a succession rather than a single representation.
The model contains an organized sequence of learned representations. The number of contributing layers gives the model its depth.
This example is intentionally abstract. The essential pattern does not depend on naming a particular application. What matters is that data is represented through successive learned layers, and that the number of those layers determines the model's depth.
What Deep Measures
Model depth is the number of contributing layers. In deep learning, deep refers to the presence of many layers in the succession of learned representations. It does not mean deeper understanding.
An approach that learns only one or two representation layers has a shorter representation sequence. Deep learning emphasizes many stacked layers. Modern deep-learning models often contain tens or even hundreds of such layers, and those layers are learned automatically from training data.
Neural Networks as Layered Structures
Neural networks provide the stacked-layer structure used to learn successive representations. A layer contributes to the sequence, and its learned representation can be passed to another layer. This arrangement lets the model organize learning across multiple representations instead of assigning the whole task to one learned representation.
Neural Does Not Mean Biological
A common first impression is that deep learning is a computer version of the brain. The name neural network can encourage that interpretation, but it is misleading. In this context, deep learning is a mathematical framework for learning representations from data.
Treating a neural network as a model of the brain.
Deep-learning models are not models of the brain, and their learning mechanisms should not be treated as the same as biological learning mechanisms.
Fix:
Understand a neural network here as the stacked-layer structure used to learn representations from data.Interpreting deep as deeper understanding.
In this context, depth refers to the number of contributing layers.
Fix:
Count the contributing layers when interpreting model depth.Calling a model deep merely because it uses a neural network.
Deep learning emphasizes a succession of representations and the number of contributing layers.
Fix:
Focus on the layered organization and the model's depth, not only on the term neural network.
Check the Representation Sequence
A model learns an input representation, passes it to a second learned representation, and then stops. Another model learns a succession of many stacked representations. Which model better matches the source definition of deep learning, and what does its depth refer to?
Hints
- Look for the number of contributing layers.
- Deep learning emphasizes successive learned representations.
- Do not use the word deep to mean deeper understanding.
Practice Answer
Compare the two abstract models.
Identify the representation sequence: The first model learns only a short sequence of representations. The second learns many stacked representations.
Apply the definition: The second model better matches deep learning because deep learning emphasizes successive learned representations.
Interpret depth: Its depth refers to the number of contributing layers.
The model with many stacked learned representation layers better matches the described deep-learning approach.
Key Takeaways
- Deep learning extends machine learning by emphasizing successive learned representations.
- Deep refers to the number of contributing layers, not to deeper understanding.
- Neural networks provide the stacked-layer structure used to learn these representations.
- Approaches with one or two representation layers differ from deep approaches that organize many layers in succession.
- Deep-learning models are mathematical frameworks for learning from data, not models of the biological brain.
Key Takeaways
- Deep learning is a subfield of machine learning focused on successive learned representations.
- A model's depth is the number of contributing layers.
- Neural networks arrange layers so representations can be learned successively.
- Deep learning should not be interpreted as a computer version of the brain.
- The difference between shallow and deep approaches is the organization and number of learned representation layers.