Tensor Shapes and Reshaping

Matrix multiplication is a fundamental operation in linear algebra, and PyTorch gives you more than one way to carry it out. This article starts with a short program rather than a definition. Before you read the output, try to work out the answer yourself. The goal is not just to get the right numbers, but to be able to say exactly which numbers produced each value you see.

The .matmul method is the common way to multiply two matrices in PyTorch. You call it on one tensor and pass the other tensor as the argument. In the program below, the 2D tensor is multiplied by its transpose, written tensor2d.T, which flips rows and columns. The @ operator performs the same multiplication in a more compact form. Both spellings return the matrix product, so for this case you can treat them as interchangeable: one is a method call, the other is an infix operator.

The printed result has four entries: 14, 32, 32, and 77. Notice that the two off-diagonal entries are equal. That is not a coincidence. When you multiply a matrix by its own transpose, the result is always symmetric, so the entry in row i and column j matches the entry in row j and column i. You can check this by looking at the output rather than taking it on trust.

Every entry of a matrix product is built from one row of the first matrix and one column of the second matrix. You pair the numbers in that row with the numbers in that column position by position, multiply each pair, and then add the products together. The position of the entry in the output tells you which row and which column were used. In symbols, the entry in row i and column j of a product C = AB is $C_{ij} = \sum_{k} A_{ik} B_{kj}$, which says exactly the same thing: multiply matching pairs and sum them.

When you multiply a matrix by its own transpose, the columns of the second factor are simply the rows of the original matrix. That is why the calculation looks like pairing rows with rows. The top-right entry of the product, for example, comes from the first row of the first matrix paired with the second row of the transpose, which is the second column of the original matrix. Seeing the entry as a row-by-row pairing is the key to reading every result that follows.

The same procedure gives every other entry. The bottom-right entry pairs row 1 with row 1, giving 16 plus 25 plus 36, which equals 77. Because the two entries in the middle both pair row 0 with row 1, they both equal 32. Each position in the output is a self-contained calculation, which is why an error in one entry can be traced back to one specific row pair.

The transpose is what makes the product of a matrix and its own transpose possible. Transposing flips rows and columns, so a tensor with shape 2 by 3 becomes a tensor with shape 3 by 2, and each value moves to a new position. The value in row 0, column 2 of the original moves to row 2, column 0 of the transpose. Keeping track of where values move is the main way to check that the shapes of your operands are what you think they are.

Shapes and order determine whether a product is defined and what shape it has, so you should check the shape of each operand before multiplying. For two matrices, the inner dimensions must match. A tensor with shape m by n can be multiplied by a tensor with shape n by p, and the result has shape m by p. The inner n cancels out in the sense that it is used for pairing, while the outer m and p survive to become the shape of the output.