Adding Tensors
Broadcasting makes tensors with different shapes compatible for operations such as addition when possible and unambiguous.
Why Shape Matters
Adding tensors is straightforward when both tensors have the same shape. The difficulty appears when their shapes differ. An addition operation that handles only two-dimensional tensors with identical shapes cannot directly handle every addition used in a neural-network layer. For example, a Dense layer can add a two-dimensional tensor to a vector. Broadcasting describes how tensors with different shapes can be made compatible for operations such as addition when that compatibility is possible and unambiguous.
Before addition can be performed on tensors with different shapes, broadcasting may be needed to make their shapes compatible.
What do you think happens?
A smaller tensor must be prepared to work with a larger tensor. What happens first in the broadcasting process?
Reveal answer
Answer: Broadcast axes are added to the smaller tensor.
Adding broadcast axes is the first of the two broadcasting steps described in the source. The smaller tensor begins with fewer axes, and this step gives it the same number of dimensions as the larger tensor.
Aligning the Tensor Shapes
Broadcasting changes the dimensional organization of the smaller tensor so that it can be considered toward the shape of the larger tensor. The larger tensor provides the reference shape. The smaller tensor is the one adjusted during broadcasting; the larger tensor is not described as being reduced to fit the smaller one.
The diagram shows the central direction of the adjustment: the smaller tensor is broadcast toward the larger tensor's shape. Broadcasting does not begin by shrinking the larger tensor. It begins by organizing the smaller tensor so that its dimensional structure can match the larger tensor's number of dimensions.
Adding Broadcast Axes
A tensor's ndim is its number of dimensions. When the smaller tensor has fewer dimensions than the larger tensor, broadcast axes are added to the smaller tensor to match the larger tensor's ndim. This is the first of the two broadcasting steps. The important state change is dimensional: the smaller tensor starts with fewer axes and then has additional axes in its organization.
Finding the First Broadcasting Change
A larger tensor has more dimensions than a smaller tensor. Identify the first change made to the smaller tensor before addition.
Compare dimensional organization: The smaller tensor begins with fewer axes than the larger tensor.
Add broadcast axes: Broadcast axes are added to the smaller tensor so that its number of dimensions matches the larger tensor's ndim.
Continue toward compatibility: The smaller tensor is now organized with the same number of dimensions as the larger tensor. This is the first broadcasting step; the source describes broadcasting as a two-step process.
The first step adjusts the smaller tensor by adding broadcast axes. The larger tensor remains the reference shape.
From Preparation to Addition
The sequence begins by comparing the tensors' dimensional organization. If one tensor has fewer axes, that smaller tensor is the candidate for broadcasting. Broadcast axes are added first, giving the smaller tensor the same number of dimensions as the larger tensor. Broadcasting can then make the tensors compatible for an operation such as addition when the result is possible and unambiguous.
A Dense layer can add a two-dimensional tensor to a vector. This is an example of why tensor addition cannot always be treated as the addition of two tensors that already have identical two-dimensional shapes. Broadcasting provides the description of how tensors with different shapes can be made compatible for this kind of operation.
Mistakes About Broadcasting
Assuming tensors must already have identical shapes before they can be added.
Broadcasting exists to make tensors with different shapes compatible for operations such as addition when possible and unambiguous.
Fix:
First consider whether broadcasting can make the different shapes compatible.Adjusting the larger tensor instead of the smaller tensor.
The smaller tensor is broadcast toward the shape of the larger tensor.
Fix:
Use the larger tensor as the reference shape and identify the smaller tensor as the one being adjusted.Forgetting that broadcast axes are added before later broadcasting work.
Adding broadcast axes is the first of the two broadcasting steps.
Fix:
Trace the first step explicitly: add axes to the smaller tensor until its number of dimensions matches the larger tensor's ndim.Treating a broadcast axis as a value operation rather than a dimensional change.
The important state change described here concerns the tensor's dimensional organization.
Fix:
Track the axes and ndim first; discuss addition only after the shapes are compatible.
When tracing tensor addition, write down three facts in order: which tensor has the larger shape, which tensor has fewer axes, and what broadcast axes must be added to the smaller tensor. This keeps the reference shape and the first broadcasting step distinct.
Check the First Step
A smaller tensor and a larger tensor have different numbers of dimensions. Explain which tensor is adjusted during broadcasting and describe the first change made to it. Then state what the larger tensor contributes to the process.
Hints
- The smaller tensor is the one broadcast toward the larger tensor.
- The first broadcasting step concerns axes and ndim.
- The larger tensor supplies the reference shape.
Practice Answer
Explain the first step when a smaller tensor must be made compatible with a larger tensor for addition.
Identify the adjusted tensor: The smaller tensor is adjusted; the larger tensor remains the reference shape.
Identify the dimensional change: Broadcast axes are added to the smaller tensor.
Identify the target: The added axes bring the smaller tensor to the same number of dimensions as the larger tensor.
The first step is adding broadcast axes to the smaller tensor so that its ndim matches the larger tensor's ndim.
Key Takeaways
- Tensors with identical shapes can be added straightforwardly, while tensors with different shapes may need broadcasting.
- Broadcasting makes different tensor shapes compatible for operations such as addition when compatibility is possible and unambiguous.
- The smaller tensor is broadcast toward the larger tensor's shape.
- Broadcast axes are added to the smaller tensor to match the larger tensor's ndim.
- Adding broadcast axes is the first of the two broadcasting steps.
Key Takeaways
- Broadcasting is needed when tensors with different shapes must be made compatible for addition.
- The smaller tensor is adjusted toward the larger tensor's reference shape.
- Broadcast axes are added to the smaller tensor to match the larger tensor's number of dimensions.
- Adding broadcast axes is the first broadcasting step.
- A Dense layer may add a two-dimensional tensor to a vector, illustrating why identical starting shapes are not always required.