Why Deep Learning Now?

The Hardware Turning Point

The rise of deep learning was not caused only by better algorithms. Hardware changed the boundary between an idea that could be computed and an idea that was impractical to run. Between 1990 and 2010, off-the-shelf CPUs became approximately 5,000 times faster. That improvement made small deep-learning models runnable on an ordinary laptop, even though the same kind of task would have been intractable 25 years earlier.

From Runnable to Impractical

The CPU improvement did not make every deep-learning workload easy to run on a laptop. It made small deep-learning models possible on ordinary hardware. Typical computer-vision and speech-recognition models still require far more computational power than a laptop can provide. This creates an important distinction: a model can be small enough to run locally, while a typical model in an application area can still exceed the available computing capacity.

Why Neural Networks Fit Parallel Hardware

A deep neural network is well suited to a parallel processor because it consists mostly of many small matrix multiplications. The individual computational pieces can be handled in parallel instead of being treated as one long, purely sequential calculation. This property is called parallelizability. It connects the structure of the model with the design goal of a GPU: performing many operations at the same time.

From Graphics to General Computation

GPUs were not originally created for deep learning. They were developed to render video-game graphics, including increasingly photorealistic scenes. That investment produced fast, massively parallel chips. Although their original purpose was graphics, the hardware was also useful for other applications whose computations could be parallelized.

The transition was therefore a change in use, not a claim that GPUs were originally built for neural networks. Their graphics role encouraged the development of massively parallel hardware. Once researchers recognized that other workloads also contained computations that could be parallelized, the same kind of hardware became useful beyond graphics.

CPU and GPU Work Styles

The useful contrast is between a CPU's general-purpose role and a GPU's ability to perform many operations at the same time. Deep-learning computation benefits from the GPU style because the model contains many similar matrix-multiplication pieces. The GPU supplies a different kind of computational capacity rather than simply being described as a universally better replacement for a CPU.

CUDA Makes the Hardware Usable

The hardware alone was not the whole story. In 2007, NVIDIA launched CUDA, a programming interface for its GPUs. CUDA enabled researchers and developers to use those GPUs for applications beyond their original graphics role. The source describes an early consequence in scientific computing: a small number of GPUs began replacing massive clusters of CPUs in highly parallelizable applications, starting with physics modeling. Because deep neural networks are also highly parallelizable, researchers began writing CUDA implementations of neural networks around 2011.

Mistakes in the Hardware Story

• Assuming CPU improvement made typical deep-learning models easy to run on a laptop.

  Typical computer-vision and speech-recognition models still require far more computational power than a laptop can provide.

  Fix: Distinguish between small models that became practical and typical models that remained too computationally demanding.

• Saying that GPUs were originally created for deep learning.

  Their later usefulness for deep learning came from the fact that their massively parallel design also suited highly parallelizable workloads.

  Fix: Describe deep learning as a later use of graphics-oriented parallel hardware.

• Treating CUDA as the GPU itself.

  CUDA is software access to the hardware, not the hardware component.

  Fix: Explain the relationship as GPU hardware plus CUDA programming support.

• Ignoring the model structure when explaining GPU usefulness.

  Without the parallelizable structure of those calculations, the connection between neural networks and GPUs is incomplete.

  Fix: Always connect GPU usefulness to the many computational pieces that can be handled in parallel.

Check the Causal Chain

The Complete Explanation

1. Between 1990 and 2010, off-the-shelf CPUs became approximately 5,000 times faster, making small deep-learning models runnable on ordinary laptops.
2. Typical computer-vision and speech-recognition models still require far more computational power than a laptop can provide.
3. GPUs began as video-game graphics hardware, but their massively parallel design also suited scientific and deep-learning workloads.
4. CUDA, launched by NVIDIA in 2007, provided a programming interface that enabled applications to use NVIDIA GPUs beyond graphics.
5. Deep neural networks match GPUs because they consist mostly of many small matrix multiplications whose computational pieces can be handled in parallel.