Machine Learning Datasets
Data provides the material from which deep-learning systems learn.
From Examples to Rules
A deep-learning system begins with data rather than with intelligence already built in. Data provides the material from which the system learns. In machine learning, people provide data together with expected answers, and the computer uses that material to produce rules. Those learned rules can then be applied to new data to generate answers.
Why Data Became Abundant
The rise of deep learning depended on improvements in two areas: the amount of available data and the algorithms used to train neural networks. The internet changed the scale of the material available for machine learning because internet-scale collection and distribution made very large datasets feasible. These datasets can contain images, videos, or natural-language material.
| Condition | Role in learning |
|---|---|
| More available data | Supplies more material from which a learning system can improve |
| Better training algorithms | Determine whether a network can make useful progress from that material |
Deep learning became more promising when data availability and training methods improved together.
When Depth Weakens Feedback
Training a neural network requires feedback so the network can improve. That feedback must pass through the network's layers. As the stack becomes deeper, the feedback signal can fade before it has a useful effect throughout the network. This made training deep networks difficult.
Before a reliable way to train very deep neural networks was available, neural networks generally remained shallow, using only one or two layers of representations. Because of this limitation, they did not consistently outperform more-refined shallow methods such as support vector machines and random forests.
Two Ways to Program
| Classical programming | Machine learning |
|---|---|
| Humans provide a program containing rules and data. | Humans provide data together with expected answers. |
| The computer follows the supplied rules. | The computer uses the paired data and answers to produce rules. |
| The computer produces answers from the human-designed path. | The learned rules are applied to new data to produce answers. |
In classical programming, the path from input to output is designed by the programmer before the computer runs the program. In machine learning, humans do not provide the complete set of rules in advance. They provide examples and expected answers, and the computer produces rules from that material. The rules can then be used with information that was not part of the original material.
A Message-Classification Trace
Learning from labeled messages
A human wants a computer to assign each message to one of two categories. How can machine learning approach this task?
1. Supply data: The human gives the computer many messages as data.
2. Supply expected answers: The human supplies the expected category for each message, creating paired examples.
3. Produce rules: The computer uses the paired messages and answers to learn rules.
4. Apply the rules: When a new message arrives, the learned rules are applied to that new case.
5. Generate an answer: The rules produce an answer for the new message.
The computer has moved from paired examples to learned rules and then used those rules to answer a case that was not part of the original material.
The important distinction is the direction of construction. In classical programming, a human writes the rules first. In this machine-learning scenario, the human supplies examples and expected answers, while the computer produces rules that can later be applied to new messages.
What do you think happens?
In the message scenario, what does the computer receive before it can produce learned rules?
Reveal answer
Answer: Messages together with expected answers
Machine learning begins with data and expected answers. The computer uses that paired material to produce rules, which can then be applied to new data.
Lovelace and Computer Learning
Machine learning connects to a historical question: can a computer learn rather than merely follow explicitly ordered instructions? In 1843, Ada Lovelace argued that Charles Babbage's Analytical Engine could perform whatever people knew how to order it to perform, but could not originate anything. Alan Turing later discussed this idea as Lady Lovelace's objection while considering whether general-purpose computers could learn and show originality.
Machine learning addresses the learning part of that question by changing where the rules come from. Instead of having a human explicitly order every rule, the computer examines data and expected answers, produces useful rules, and uses those rules on information it has not seen before.
Common Reasoning Mistakes
Treating machine learning as classical programming with a different name.
Classical programming starts with human-written rules, while machine learning uses data and expected answers to produce rules.
Fix:
Ask who supplies the rules. In classical programming, humans supply them; in machine learning, the computer produces them from paired examples.Describing data as optional training material.
Data provides the material from which deep-learning systems learn.
Fix:
Treat data as the material a learning system uses to improve.Explaining difficult deep-network training only by counting layers.
The key issue described here is that feedback must pass through the stack, and the signal can fade before it has a useful effect throughout the network.
Fix:
Trace the feedback path through the layers and ask whether the signal remains useful across the stack.Assuming that more data alone caused the rise of deep learning.
The rise of deep learning depended on both access to much more data and progress in training algorithms.
Fix:
Explain the two conditions together: data supplies material, while training methods determine whether the network can make useful progress.
Check Your Understanding
Describe the inputs and outputs in each approach. For classical programming, state what humans provide and what the computer produces. For machine learning, state what humans provide, what the computer produces during learning, and what happens when the learned rules meet new data.
Hints
- Use the words rules, data, expected answers, and new data.
- Remember that machine learning changes where the rules come from.
Explain why adding many layers made neural-network training difficult. Your answer should mention the feedback signal and what can happen as it passes through the stack.
Hints
- Feedback is needed for improvement.
- The source describes the signal as capable of fading before it has a useful effect throughout the network.
Key Takeaways
- Data provides the material from which deep-learning systems learn.
- The internet made very large datasets feasible to collect and distribute, including images, videos, and natural-language material.
- Training deep networks was difficult because feedback could fade as it passed through many layers.
- Classical programming gives the computer human-written rules and data; machine learning gives it data and expected answers so it can produce rules.
- Machine learning connects to the historical question of whether computers can learn rather than merely follow explicitly ordered instructions.
Key Takeaways
- Deep learning needs data as the material from which its systems learn.
- Internet-scale collection and distribution expanded the supply and variety of machine-learning datasets.
- Feedback can fade across many neural-network layers, limiting the training of deep networks.
- Machine learning uses data and expected answers to produce rules that can be applied to new data.
- This shift in the source of rules connects machine learning to Lady Lovelace's historical question about whether computers can learn or originate something new.