Neural Networks
Deep learning is a subset of machine learning.
The Nested Definition
Begin with the relationship among three ideas. Machine learning is the broader category. Deep learning is a subset of machine learning, not a completely separate field. Deep learning involves using neural networks to analyze data. A complete definition therefore needs both parts: deep learning belongs inside machine learning, and neural networks are central to how it analyzes data.
A useful one-sentence definition is: Deep learning is a subset of machine learning that uses neural networks to analyze data.
Data as Tensors
Neural networks work with data by representing it as tensors, which are multi-dimensional arrays. The word tensor describes a range of representations. A scalar has no dimensions. A vector has one dimension. A matrix has two dimensions. A higher-dimensional tensor has more than two dimensions. These representations give neural-network data an organization that tensor operations can manipulate.
| Representation | Dimensions | Role in the tensor model |
|---|---|---|
| Scalar | No dimensions | A single value |
| Vector | One dimension | A one-dimensional arrangement of values |
| Matrix | Two dimensions | A two-dimensional arrangement of values |
| Higher-dimensional tensor | More than two dimensions | A multi-dimensional arrangement of values |
Tensor representations differ by the number of dimensions used to organize their values.
Dimension here describes organization, not simply the total number of values. A matrix and a vector could contain related values, but their arrangements differ. That organization matters because tensor operations can preserve an arrangement, combine values across dimensions, or change the arrangement.
Four Tensor Operations
Tensor operations form the computational machinery of a neural network. They manipulate values, combine values, support computation across dimensions, or change the organization of data. Element-wise operations work with corresponding values. Broadcasting supports an operation across dimensions when a smaller value arrangement is applied across a larger arrangement. A tensor dot operation combines tensor values through a dot-style computation. Reshaping changes how values are organized without being described as a new collection of values.
| Operation | Main role | What to watch |
|---|---|---|
| Element-wise operation | Manipulates corresponding values | The operation is applied across matching positions |
| Broadcasting | Extends an operation across dimensions | A smaller value arrangement participates across a larger arrangement |
| Tensor dot operation | Combines values through a dot-style computation | Values are combined rather than merely handled independently |
| Reshaping | Changes organization | The arrangement changes while the tensor representation is reorganized |
The operations are distinguished by whether they act on corresponding values, extend across dimensions, combine values, or reorganize data.
Following Values Through Operations
Consider a generated tensor example in which an operation first changes corresponding values, a second operation applies information across dimensions, a dot operation combines values, and reshaping reorganizes the result. What should you track at each stage?
Track the values: Ask whether the operation changes individual values, combines them, or leaves their numerical content to be reorganized.
Track the organization: Ask whether the tensor keeps its arrangement or is reorganized. Reshaping is the operation whose main role is changing organization.
Track the operation's role: Do not treat all tensor operations as interchangeable. Element-wise operations, broadcasting, tensor dot operations, and reshaping affect data in different ways.
Connect the result to computation: The transformed tensor becomes part of the computation performed by the neural network.
A reliable trace records both the tensor's values and its organization after every operation.
Tracing a Transformation
The most useful way to read a tensor computation is to trace two things at once: the values and the organization. An element-wise operation can alter values while preserving the pattern of corresponding positions. A dot operation combines values through a tensor computation. A reshape changes organization. These changes are not separate from learning; they create the tensor computation whose derivatives can later provide information for parameter updates.
What do you think happens?
A tensor passes through a reshape operation. Should you describe the main effect as changing the values or changing the organization?
Reveal answer
Answer: Changing the organization
Reshaping is identified by its role in changing how tensor values are organized. Element-wise operations manipulate corresponding values, while tensor dot operations combine values.
From Computation to Learning
Tensor computation alone does not explain learning. A neural network also has parameters that must be optimized so that the network's loss function is minimized. The gradient is described as the derivative of a tensor operation. It provides derivative information about the computation, allowing optimization to determine how parameters should be updated.
Backpropagation chains derivatives through the tensor operations. This connects the final loss to the operations and parameters that contributed to it. Stochastic gradient descent is one method used in the optimization process: it uses gradient information repeatedly to update parameters while seeking to minimize the loss function.
Reading One Optimization Cycle
Trace one generated cycle from a neural network's tensor computation to a parameter update.
Compute: The network performs tensor operations using its current parameters.
Measure loss: The computation contributes to the loss function, which indicates the result being optimized.
Chain derivatives: Backpropagation chains derivatives through the tensor operations so their effects can contribute to parameter updates.
Update: Stochastic gradient descent uses the gradient information to update parameters.
Repeat: The process is repeated as gradient-based optimization works toward minimizing the loss function.
The learning loop links tensor computation, derivatives, parameter updates, and loss minimization.
Mistakes to Avoid
Treating deep learning as completely separate from machine learning.
Deep learning is a subset of machine learning.
Fix:
Place machine learning as the broader category and deep learning as a more specific part within it.Defining deep learning without mentioning neural networks.
The definition is incomplete because it omits the role of neural networks in analyzing data.
Fix:
State both the subset relationship and the use of neural networks.Treating every tensor operation as if it performs the same job.
The operations differ in whether they manipulate corresponding values, operate across dimensions, combine values, or change organization.
Fix:
Identify the operation's role before describing its result.Explaining neural-network learning using tensor computation alone.
Learning also requires gradient-based optimization of parameters to minimize a loss function.
Fix:
Connect tensor computations to gradients, backpropagation, and stochastic gradient descent.Describing gradients as the final update rather than information for an update.
The gradient is derivative information used by an optimization method.
Fix:
Explain that stochastic gradient descent uses gradient information to update parameters.
Practice the Connections
Write a two-sentence explanation of deep learning. Your first sentence should identify its relationship to machine learning. Your second sentence should explain the role of neural networks.
Hints
- Use the words broader category and subset.
- Mention that neural networks analyze data.
Classify each operation by its main role: an operation on corresponding values, an operation applied across dimensions, a dot-style combination of values, and an operation that changes organization. Then explain why tracking both values and organization is useful.
Hints
- Match corresponding values with element-wise operations.
- Match changed organization with reshaping.
- Remember that tensor dot operations combine values.
Put these learning events in order: parameter update, tensor computation, gradient information, loss function, repeated optimization. Explain how backpropagation connects the tensor computation to the update.
Hints
- Start with the tensor computation using current parameters.
- The loss leads to derivative information.
- Backpropagation chains derivatives through tensor operations.
Key Takeaways
- Deep learning is a subset of machine learning.
- Deep learning uses neural networks to analyze data.
- Neural-network data is represented as tensors, including scalars, vectors, matrices, and higher-dimensional tensors.
- Element-wise operations, broadcasting, tensor dot operations, and reshaping manipulate tensor values or organization in different ways.
- Gradients and backpropagation provide derivative information, while stochastic gradient descent uses that information to update parameters and help minimize the loss function.
Key Takeaways
- Machine learning is the broad category; deep learning is a subset that uses neural networks.
- Tensors provide the multi-dimensional representations used by neural networks.
- Tensor operations can manipulate values, combine values, support computation across dimensions, or change organization.
- Learning connects tensor computation to gradients, backpropagation, stochastic gradient descent, and loss minimization.