Concepts / Tensor Shapes

Tensor Shapes

Broadcasting makes tensors with different shapes compatible for operations such as addition when possible and unambiguous.

  • Programming

Why Shape Matters

Adding tensors is straightforward when both tensors have the same shape. The situation becomes more involved when their shapes differ. A neural-network layer may need to add a two-dimensional tensor to a vector, so the operation needs a method for making tensors with different shapes compatible when that compatibility is possible and unambiguous.

Broadcasting is the process of making tensors with different shapes compatible for operations such as addition when the operation is possible and unambiguous.

The Alignment First Step

Broadcasting begins with the tensor that has fewer dimensions. Before compatibility can be considered, the dimensions of the smaller tensor are aligned with the larger tensor's dimensional structure. The first broadcasting step is to add broadcast axes to the smaller tensor until it has the same number of dimensions as the larger tensor.

provides target dimensional structureis adjusted towardthenLarger tensorMore dimensionsSmaller tensorFewer dimensionsDimension alignmentAdd broadcast axesCompatibility checkPossible and unambiguous
How are dimensions handled before the tensors are checked for compatibility?

Broadcast Axes

A broadcast axis is an axis added to the smaller tensor so that its number of dimensions can match the larger tensor's number of dimensions. This is a change to dimensional organization: the smaller tensor begins with fewer axes, and the broadcasting process adds axes before the operation is treated as compatible.

containsreceivescombines withraises dimensional count toSmaller tensorFewer axesExisting axesOriginal dimensionalstructureBroadcast axesAdded dimensionsMatching ndimSame number of dimensions
Which axes are added so the smaller tensor can reach the larger tensor's number of dimensions?

A Dense Layer Example

Making a Vector Compatible

A Dense layer needs to add a two-dimensional tensor to a vector. Which tensor is adjusted in the first broadcasting step?

Identify the dimensional difference: The two-dimensional tensor has more axes than the vector, so the vector is the smaller tensor for this operation.

Choose the tensor to adjust: Broadcasting adjusts the smaller tensor toward the shape of the larger tensor. The two-dimensional tensor remains the reference for the target dimensional structure.

Add broadcast axes: Broadcast axes are added to the vector until it has the same number of dimensions as the two-dimensional tensor. This is the first of the two broadcasting steps.

Continue to compatibility: Once the dimensional counts match, the tensors can be considered for compatibility for addition. Broadcasting makes the operation possible only when that compatibility is unambiguous.

The vector is adjusted. The larger two-dimensional tensor keeps its original dimensional structure, while broadcast axes are added to the vector during the first broadcasting step.

broadcasted toward larger shapekeeps structureVectorFewer axesBroadcasted vectorBroadcast axes addedTwo-dimensionaltensorMore axesTwo-dimensionaltensorOriginal dimensionalstructure
Which tensor keeps its original shape, and which tensor is virtually expanded toward it?

What Broadcasting Does Not Guarantee

Different shapes do not automatically make an addition valid. Broadcasting describes how tensors can be made compatible when possible and unambiguous. Adding broadcast axes is the first step, not a guarantee that every pair of different shapes can be added.

may requiresupportswhen the result isDifferent shapesBroadcasting may be neededBroadcast axesAdded to smaller tensorCompatible shapesWhen possibleUnambiguous additionOperation can proceed
How does broadcasting distinguish a pair that can be made compatible from a pair whose addition is not established as valid?

Common Shape Mistakes

  • Assuming that any two tensors with different shapes can be added.

    Broadcasting makes tensors compatible only when the operation is possible and unambiguous.

    Fix: After identifying the smaller tensor and adding broadcast axes, still consider whether the resulting operation is unambiguous.

  • Adjusting the larger tensor instead of the smaller tensor.

    The source describes the smaller tensor as the tensor broadcasted toward the shape of the larger tensor.

    Fix: Use the larger tensor as the target dimensional structure and adjust the smaller tensor.

  • Treating broadcast axes as ordinary data values.

    The first broadcasting step concerns dimensional organization: axes are added so the smaller tensor has the same number of dimensions as the larger tensor.

    Fix: Describe the first step in terms of axes and dimensions rather than new tensor values.

  • Calling the first step the complete broadcasting process.

    Adding broadcast axes is identified as the first of two broadcasting steps.

    Fix: Recognize the added axes as the first step, then remember that compatibility for addition must still be possible and unambiguous.

Check Your Understanding

EASY

A larger tensor and a smaller tensor have different numbers of axes. Explain the first broadcasting step in three parts: identify which tensor is adjusted, state what is added to it, and describe the dimensional result.

Hints
  • The tensor with fewer axes is the smaller tensor for this step.
  • Broadcast axes are added to the smaller tensor.
  • The target is the same number of dimensions as the larger tensor.
MEDIUM

A Dense layer adds a two-dimensional tensor to a vector. Describe why broadcasting may be needed and explain why the vector, rather than the two-dimensional tensor, is adjusted first.

Hints
  • The tensors have different shapes.
  • The smaller tensor is broadcasted toward the larger tensor's shape.
  • The first step adds axes to match the larger tensor's number of dimensions.

The Shape-Checking Habit

  1. Broadcasting helps tensors with different shapes become compatible for operations such as addition when compatibility is possible and unambiguous.
  2. The smaller tensor is broadcasted toward the shape of the larger tensor.
  3. Broadcast axes are added to the smaller tensor so it can match the larger tensor's number of dimensions.
  4. Adding broadcast axes is the first of the two broadcasting steps.
  5. Matching dimensional counts does not by itself guarantee that an addition is valid; the operation must still be possible and unambiguous.

Key Takeaways

  • Broadcasting addresses shape differences that arise when tensors are used together in operations such as addition.
  • The smaller tensor is the one adjusted during broadcasting.
  • The first broadcasting step adds broadcast axes to the smaller tensor.
  • Those axes give the smaller tensor the same number of dimensions as the larger tensor.
  • The operation is valid only when the resulting compatibility is possible and unambiguous.