Matrix Multiplication with Tensors

This article follows a short PyTorch program in which one small tensor is created, inspected, reshaped, transposed, and multiplied by its own transpose. Each step is simple on its own. The habit this article builds is predicting the result of each line before reading the output, then checking the prediction against what actually printed. A single wrong assumption early in the program, such as a shape you did not expect, carries forward into every later line and can look like a failure somewhere else entirely.

Before reading any further, try this prediction. The starting tensor has two rows of three numbers. If you apply the transpose attribute to it, what shape do you expect the result to have? Commit to an answer, then keep reading.

The starting tensor is built from a list of two inner lists, each holding three numbers. PyTorch reads that nested structure and creates a two-dimensional tensor. The shape attribute reports how many entries lie along each dimension. It is the first thing to check whenever a later operation behaves strangely, because almost every error in this kind of program traces back to a shape that was not what you assumed.

The six values in this tensor form one fixed sequence: 1, 2, 3, 4, 5, 6. Every operation in the program either regroups that sequence, flips it, or combines it arithmetically. Reading the shape first tells you how those six values are currently grouped.

The reshape method takes a new shape and returns a tensor with that shape. The starting tensor holds six values, so any new shape must also account for exactly six entries. Asking for a shape with a different total count cannot succeed, because there is no way to place seven values into six slots or five values into six slots.

The important point is the order of the values. Reshaping does not shuffle or recalculate anything. The six values keep their sequence, and the new shape determines where each row begins and ends. Reading the same six values in order into three rows of two produces a different grouping of the same numbers. The diagram below shows this: each position in the flat sequence maps to exactly one cell in the new layout.

The transpose attribute, written .T with no parentheses, flips a two-dimensional tensor so that its rows become its columns and its columns become its rows. The first row of the original becomes the first column of the result, and the same pattern continues for every row. Because the two dimensions trade places, a tensor of shape (2, 3) becomes a tensor of shape (3, 2).

Notice how the earlier prediction is checked here. If you predicted (3, 2) and the printed shape is torch.Size([3, 2]), the prediction held. If the shape had come back as (2, 3), the transpose would not have run as expected, and the mistake would be in the input or in which attribute was called, not in the matrix arithmetic that comes next.

The matmul method multiplies two tensors as matrices. In this program the tensor is multiplied by its own transpose. The original is two by three and the transpose is three by two. The inner dimensions must agree: the three in the first tensor and the three in the second tensor. When they agree, they drop out of the result, and the outer dimensions form the shape of the product.

To check the top-left entry by hand, multiply the first row by the first column and add the products: 1 times 1, plus 2 times 2, plus 3 times 3, which equals 14. The entry at position (0, 1) is 1 times 4, plus 2 times 5, plus 3 times 6, which equals 32. Checking one entry this way is a fast test when a result looks wrong, because it isolates a single cell from the rest of the program.

When an output differs from what you expected, the most efficient approach is to move forward through the program one operation at a time and check the shape and values after each. The first line where the observed result departs from your prediction is the failure location. Everything after that point may be a consequence of that line rather than a separate cause.