Dense Layers

Broadcasting makes tensors with different shapes compatible for operations such as addition when possible and unambiguous.

Programming

Why Shape Matters

Adding tensors is straightforward when both tensors have the same shape. The situation becomes more involved when their shapes differ. A Dense layer can add a two-dimensional tensor to a vector, so the addition operation needs a defined way to make those tensors compatible. Broadcasting provides that way when compatibility is possible and unambiguous.

How are the dimensions of two differently shaped tensors compared before addition?

The First Shape Change

Broadcasting adjusts the smaller tensor toward the shape of the larger tensor. The first broadcasting step is dimensional: broadcast axes are added to the smaller tensor until it has the same number of dimensions, or ndim, as the larger tensor. This does not begin by changing the larger tensor. The smaller tensor is the tensor being adjusted.

What happens first when a smaller tensor is made compatible with a larger tensor?

A Shape-Alignment Example

Adding a Two-Dimensional Tensor and a Vector

A Dense layer needs to add a two-dimensional tensor to a vector. What is the first broadcasting change?

Identify the tensors: One tensor is two-dimensional, while the other is a vector with fewer dimensions.

Choose the tensor to adjust: The vector is the smaller tensor in terms of dimensional organization, so it is broadcasted toward the shape of the two-dimensional tensor.

Add broadcast axes: Broadcast axes are added to the vector so that its number of dimensions matches the larger tensor's ndim. This is the first broadcasting step.

Continue only when compatible: After the dimensional organization is aligned, broadcasting can proceed only if the shapes are compatible and the operation is unambiguous.

The smaller vector is adjusted first by adding broadcast axes. The larger two-dimensional tensor remains unchanged during this first step.

Which tensor changes shape during the first broadcasting step, and which tensor remains unchanged?

The important detail is not a numerical result but a change in dimensional organization. Before broadcasting, the vector has fewer axes. After the first step, broadcast axes have been added so that the smaller tensor has the same number of dimensions as the larger tensor. The larger tensor is the reference shape toward which the smaller tensor is broadcasted.

Avoiding Shape Confusion

Assuming that tensors with different shapes can always be added directly.
Broadcasting applies when compatibility is possible and unambiguous; different shapes alone do not guarantee that condition.
Fix: Check the tensor shapes and determine whether broadcasting can make them compatible.
Adding broadcast axes to the larger tensor.
The smaller tensor is broadcasted toward the shape of the larger tensor.
Fix: Identify the tensor with fewer dimensions and track the axes added to that tensor.
Confusing the first broadcasting step with the entire broadcasting process.
The source identifies adding broadcast axes as the first of two broadcasting steps and separately requires compatibility for the operation.
Fix: Describe axis addition as the first dimensional change, then remember that the overall shapes must still be compatible and unambiguous.
Thinking of broadcast axes as values from the larger tensor.
The source describes the first state change as dimensional organization: the smaller tensor gains axes until its ndim matches the larger tensor.
Fix: Use the language of axes, dimensions, ndim, and shape when describing this first step.

Check Your Understanding

EASY

A Dense layer is about to add a two-dimensional tensor and a vector. Without calculating any values, identify which tensor is adjusted first and describe the first broadcasting step.

Hints

Compare the number of dimensions rather than the numerical values.
The tensor with fewer dimensions is the smaller tensor for this step.
The first step adds broadcast axes until the smaller tensor has the same ndim as the larger tensor.

Your attempt (not saved)

What do you think happens?

A smaller tensor has fewer dimensions than a larger tensor. Which tensor receives broadcast axes in the first broadcasting step?

The smaller tensor
The larger tensor
Both tensors equally
Neither tensor

Reveal answer

Answer: The smaller tensor

Broadcasting moves the smaller tensor toward the shape of the larger tensor. Adding broadcast axes to match the larger tensor's ndim is the first broadcasting step.

Key Takeaways

Broadcasting makes tensors with different shapes compatible for operations such as addition when compatibility is possible and unambiguous.
The smaller tensor is broadcasted toward the shape of the larger tensor.
Broadcast axes are added to the smaller tensor to match the larger tensor's number of dimensions, or ndim.
Adding broadcast axes is the first of the two broadcasting steps.
The first state change is dimensional: the smaller tensor gains axes while the larger tensor remains unchanged during this step.

Key Takeaways

Broadcasting is needed when tensors with different shapes must participate in an operation such as addition.
The smaller tensor is adjusted toward the larger tensor's shape.
The first broadcasting step adds axes to the smaller tensor until its ndim matches the larger tensor's ndim.
Shape compatibility must still be possible and unambiguous before addition can proceed.