Anatomy of a Neural Network

One Network, Several Jobs

A neural network is easier to understand when you separate the jobs performed by its parts. Data must be represented in a form the network can process. Tensor operations manipulate that data. Layers organize transformations, and a model connects layers into a larger network. During training, a loss evaluates a prediction, while derivatives, gradients, and stochastic gradient descent provide information used to improve the model's parameters.

From Scalars to Tensors

Neural network data is represented as tensors. A scalar is a 0D tensor, a vector is a 1D tensor, and a matrix is a 2D tensor. Data can also be represented with 3D tensors and tensors with still higher dimensionality. The number of dimensions describes the structural form in which the data is organized.

The representation is more than a container for numbers. Its dimensions organize the data for the neural network. Vector data, timeseries or sequence data, image data, and video data are discussed as examples of data tensors. A batch groups multiple samples so they can be processed together.

Changing Tensor Structure

Tensor reshaping changes a tensor's structure. The important question is not simply whether values are present, but how their dimensions organize those values for later processing. Reshaping therefore belongs to the group of operations that transform tensor structure rather than describing a new kind of data.

Operations on Tensor Values

Tensor operations manipulate values or tensor structure. The operations highlighted here are element-wise operations, broadcasting, tensor dot, and reshaping. Their names identify different ways of working with tensors: some operate across corresponding values, some allow differently shaped tensors to participate together, some combine tensor dimensions through a dot operation, and reshaping changes the tensor's organization.

These operations are not interchangeable labels. Element-wise operations focus on values in corresponding positions. Broadcasting concerns how tensor shapes can participate together. Tensor dot concerns a dot-based combination of tensors. Reshaping concerns structure. In a neural network, these operations can transform data before or within the network.

Layers and Model Organization

Layers are the building blocks of deep learning models. A model is a network of layers. This gives the architecture a two-level description: an individual layer is a component, while the model organizes layers into a network. Data moves through layers, and the sequence of layers constitutes the model's network.

Loss functions and optimizers configure the learning process around the model. The model organizes the layers and their transformations; the loss function evaluates the result of a prediction; and the optimizer participates in using training information to improve the model. Keras is introduced as a deep learning framework for developing with deep learning models.

The Training Cycle

Training is cyclical rather than one-time. The network produces a prediction. Training evaluates the result through a loss. Derivative information is represented by a gradient. Stochastic gradient descent uses that information in optimization, producing an updated model that can participate in another training step.

Backward Information and Updates

Gradient-based optimization is the training engine of a neural network. A derivative describes change, and the gradient is introduced as the derivative of a tensor operation. During training, derivative information is carried back through the connected computation so that stochastic gradient descent can use it to guide parameter improvement. This backward use of derivative information is the role associated with backpropagation in the training process.

The key idea is not that training performs one backward action and stops. The updated model enters another cycle. Repeated cycles connect predictions, losses, derivatives, gradients, and stochastic gradient descent into gradient-based optimization.

Mistakes in the Mental Model

• Treating a tensor as only a container for numbers

  The dimensions organize the data, and that organization is part of how the neural network represents and processes it.

  Fix: Always identify both the values and the tensor's dimensional structure.

• Confusing reshaping with an operation that creates a new set of values

  Reshaping is highlighted as an operation on tensor structure and organization.

  Fix: Describe reshaping in terms of changed structure and index organization.

• Using element-wise operation, broadcasting, and tensor dot as synonyms

  The topic distinguishes these operations as different ways of working with tensor values or structure.

  Fix: Ask whether the operation concerns corresponding values, shape participation, or a tensor dot combination.

• Calling a single layer a complete model

  Layers are components, while a model is a network of layers.

  Fix: Use layer for an individual building block and model for the organized network.

• Thinking training ends after a prediction

  Training continues through loss evaluation, derivative and gradient information, stochastic gradient descent, and a later training cycle.

  Fix: Trace the complete cycle from prediction to update and back to another step.

Trace the Architecture

Structural Takeaways

1. Tensors represent neural network data, from 0D scalars through vectors, matrices, and higher-dimensional forms.
2. Element-wise operations, broadcasting, tensor dot, and reshaping manipulate tensor values or structure in different ways.
3. Layers are building blocks, while a model is a network of layers; Keras is a framework for developing with deep learning models.
4. Training cycles through prediction, loss, derivatives, gradients, stochastic gradient descent, and parameter updates.
5. Backpropagation and gradient-based optimization connect information about change to improvements in the model during repeated training cycles.