Deep Neural Networks
Data provides the material from which deep-learning systems learn.
Learning Needs Material
A deep-learning system begins with data rather than with intelligence already built in. Data provides the material from which the system learns. The system can improve only when training methods can make useful progress from that material.
The rise of deep learning depended on two improvements arriving together: much more available data and progress in the algorithms used to train neural networks.
The Internet’s Dataset Effect
Before internet-scale collection and distribution, the supply of material available for machine learning was more limited. The rise of the internet changed the scale of that material. Very large datasets became feasible to collect and distribute for machine learning.
Following the Data Story
Explain why internet-scale material mattered to the rise of deep learning.
Start with material: A learning system needs data as the material from which it can learn.
Increase the supply: The internet made very large collections of images, videos, and natural-language material feasible to collect and distribute.
Connect material to training: Those large datasets supplied more material for machine-learning systems, provided that training algorithms could use it effectively.
Internet-scale data addressed the supply side of deep learning, while training-algorithm progress addressed whether neural networks could improve from that supply.
Why Depth Became Difficult
A neural network with several layers must use feedback to improve. That feedback passes through the stack of layers. Adding layers made the path through which the feedback had to travel longer, creating a training difficulty: the feedback signal could fade before it had a useful effect throughout the network.
The important contrast is not simply that a deeper network contains more parts. Its feedback must cross more layers before it can affect the whole network. As the stack became deeper, that feedback could fade away before reaching layers where it needed to have a useful effect.
The Fading Feedback Signal
Think of training feedback as a message sent backward through the network. In a network with several layers, the message must pass through the stack so that the network can improve. If the message becomes weaker as it travels, it may no longer have a useful effect by the time it reaches parts of the network farther along the backward path.
Tracing One Training Attempt
Trace what happens when a deep network tries to use feedback to improve.
Feedback enters the stack: The network uses feedback to improve after processing data.
Feedback crosses layers: The feedback must pass through the network's layers rather than affecting every layer directly.
The path becomes longer: With more layers, the feedback has more of the stack to cross.
The signal fades: The source identifies fading feedback as the reason training very deep networks was difficult.
Improvement is limited: If the feedback fades before having a useful effect throughout the network, the network cannot reliably make useful progress across all those layers.
Increasing depth created a longer feedback path, and a fading signal limited the training effect across the network.
The central obstacle was a training problem, not merely a shortage of possible network designs: feedback could fade across many layers before it had a useful effect throughout the network.
Before Reliable Deep Training
Before a reliable way to train very deep neural networks was available, neural networks generally remained shallow, using only one or two layers of representations. This limitation mattered in comparison with more-refined shallow methods such as support vector machines and random forests.
| Situation | Network depth | Training consequence |
|---|---|---|
| Earlier neural-network practice | Generally one or two layers of representations | Very deep networks were difficult to train reliably |
| Increasing the layer count | A deeper stack of layers | Feedback had farther to travel and could fade |
| More-refined shallow methods | Shallow methods such as support vector machines and random forests | Neural networks did not consistently outperform them before reliable deep training |
Check Your Understanding
A learner says, “Deep learning became possible simply because computers could use more layers.” Correct this explanation using the roles of data and training algorithms.
Hints
- Identify what supplies the material for learning.
- Identify what determines whether a network can make useful progress from that material.
- Explain what happens to feedback when the layer stack becomes deeper.
Treating data as optional background information.
Data provides the material from which a deep-learning system learns.
Fix:
Explain both sides: data supplies the material, and training methods determine whether the network can improve from it.Assuming that more layers automatically make training better.
As the stack became deeper, feedback could fade before it had a useful effect throughout the network.
Fix:
Describe greater depth together with the longer feedback path and its training difficulty.Explaining the rise of deep learning through internet data alone.
The source identifies advances in training algorithms as the second condition that improved alongside data availability.
Fix:
Connect internet-scale datasets with progress in neural-network training methods.
Key Takeaways
- Data is the material from which deep-learning systems learn.
- The internet made very large datasets feasible to collect and distribute for machine learning.
- The rise of deep learning required both more available data and progress in training algorithms.
- Adding layers made the feedback path longer, and the feedback signal could fade before affecting the whole network usefully.
- Before reliable training for very deep networks was available, neural networks generally remained shallow and did not consistently outperform refined shallow methods.
Key Takeaways
- Data supplies the material from which deep-learning systems learn.
- Internet-scale collection and distribution expanded the supply of images, videos, and natural-language material available in large datasets.
- Deep learning advanced when data availability and training algorithms improved together.
- Feedback had to travel through every layer, and increasing depth could cause that signal to fade before it had a useful effect.