Artificial Neural Networks

From Input to Training

An artificial neural network is not one isolated computational rule. It is a network of interconnected units whose computations work together to represent a relationship between inputs and outputs. During training, the network repeatedly computes activations in one direction and then computes information about its weights in the opposite direction. This alternating process is called backpropagation.

Interconnected Units

Each unit receives activations from units before it and contributes an activation that can be used by later units. Because the units are connected, the network represents a relationship through the combined activity of many units. This connected structure is important when the relationship being represented is not a simple, straight, easily described rule.

The Forward Pass

A forward pass begins with the current activations of the network's input units. Computation then proceeds toward later units. Each unit's activation is determined from the activations available before it. In this direction, information moves from the input side through the network.

The Backward Pass

After the forward pass, backpropagation performs a backward pass. Its task is not to recompute the forward activations. Instead, it efficiently computes a partial derivative for each weight in the network. The collected partial derivatives form an estimate of the true gradient.

This information is useful because a network with hidden layers contains weights whose effects are separated from the later parts of the computation. The backward pass organizes the calculation of weight-specific derivatives across the network, giving the training process a gradient estimate rather than only the activations produced by the forward pass.

Why the Directions Alternate

Training an ANN with hidden layers requires more than sending an input through the network once. A forward pass computes the activations produced by the current input and current network state. A backward pass then computes a partial derivative for every weight. Alternating these passes connects what the network currently does with information that the training process can use to adjust its parameters.

• Forward: compute the activation of each unit from the activations available before it.
• Backward: compute a partial derivative for each weight.
• Collect: treat the derivatives together as an estimate of the true gradient.
• Repeat during training: alternate the two directions as the network is trained.

Nonlinear Function Approximation

Many machine learning tasks require a system to represent a relationship that is not simply a straight, easily described rule. Artificial neural networks are widely used for nonlinear function approximation: the network of connected units serves as the tool for representing that relationship.

The important connection is between structure and purpose. A single isolated unit offers only one local computation. An interconnected network combines the activations of many units across the network, allowing the system to represent a more complex relationship. Hidden layers are part of this connected organization, and backpropagation provides a way to train the weights in such a network.

Deep Layered Networks

Artificial neural networks have a long history. More recent progress in training deeply layered ANNs has contributed to some of the most impressive abilities of machine learning systems. The basic idea remains a network of interconnected units; advances in training networks with many layers have helped those systems develop stronger capabilities.

Common Mistakes

• Treating an ANN as a single isolated unit.

  The definition emphasizes a network of interconnected units.

  Fix: Think about how activations move among connected units and combine across the network.

• Calling the forward pass the entire training process.

  A network with hidden layers requires a backward pass that computes a partial derivative for each weight.

  Fix: Describe training as alternating forward and backward passes.

• Saying that the backward pass recomputes the forward activations.

  Its task is instead to compute partial derivatives for the weights.

  Fix: Separate the forward-pass result, unit activations, from the backward-pass result, weight derivatives.

• Confusing individual derivatives with the gradient estimate.

  The collected derivatives form the gradient estimate.

  Fix: Distinguish each weight's partial derivative from the collection of derivatives.

Check Your Understanding

Key Takeaways

1. An artificial neural network is a network of interconnected units, not a single isolated unit.
2. A forward pass computes unit activations from the current input-unit activations toward later units.
3. A backward pass computes a partial derivative for each weight rather than recomputing the forward activations.
4. The collected derivatives form an estimate of the true gradient used by the training process.
5. ANNs are widely used for nonlinear function approximation, and advances in training deeply layered networks have expanded their abilities in machine learning systems.