Tensor Operations in PyTorch

PyTorch supports matrix multiplication through the .matmul method and the @ operator, and the source shows both producing the same result for multiplying a 2D tensor by its transpose.

Programming

Why Start with a Program

Matrix multiplication is a fundamental operation in linear algebra, and PyTorch gives you more than one way to carry it out. Instead of opening with a definition, this article starts with a short program that multiplies a 2D tensor by its own transpose. Run it, read the output carefully, and aim to say exactly which numbers produced each value you see.

As you work through the trace, keep one question in mind: if the printed result ever looked wrong, where would you look first? Learning to answer that question is the real skill here. Each entry of a product is produced by a small, repeatable calculation, so a wrong entry always points to one specific row and one specific column.

Transposing a Tensor

The example multiplies a tensor by tensor2d.T, its transpose. The .T attribute flips rows and columns: the first row of the original becomes the first column of the transpose, the second row becomes the second column, and so on. If the original has shape 2 by 3, its transpose has shape 3 by 2.

This matters for the product. When you multiply a matrix by its own transpose, the columns of the second factor are simply the rows of the original matrix. That is why the calculation looks like pairing rows with rows. Keep this in mind, because it explains the symmetric pattern you will see in the output.

Running the Product with .matmul and @

The source introduces the .matmul method as the common way to multiply two matrices in PyTorch. You call it on one tensor and pass the other tensor as the argument. The @ operator performs the same multiplication in a more compact, infix form. The following program uses a 2 by 3 tensor so that the transpose can be multiplied against it. The input values here are an illustration chosen for this article; the printed result matches the values 14, 32, 32, and 77 reported in the source.

python

import torch

tensor2d = torch.tensor([[1, 2, 3],
                        [4, 5, 6]])

print(tensor2d.shape)              # torch.Size([2, 3])
print(tensor2d.T.shape)            # torch.Size([3, 2])

result_method = tensor2d.matmul(tensor2d.T)
result_operator = tensor2d @ tensor2d.T

print(result_method)
print(result_operator)
print(torch.equal(result_method, result_operator))

import torch

tensor2d = torch.tensor([[1, 2, 3],
                        [4, 5, 6]])

print(tensor2d.shape)              # torch.Size([2, 3])
print(tensor2d.T.shape)            # torch.Size([3, 2])

result_method = tensor2d.matmul(tensor2d.T)
result_operator = tensor2d @ tensor2d.T

print(result_method)
print(result_operator)
print(torch.equal(result_method, result_operator))

Output (expected)

torch.Size([2, 3])
torch.Size([3, 2])
tensor([[14, 32],
        [32, 77]])
tensor([[14, 32],
        [32, 77]])
True

Notes on the code

tensor2d.T flips rows and columns, so a 2 by 3 input gives a 3 by 2 transpose
The .matmul call and the @ operator are two spellings of the same product
torch.equal confirms that both results hold identical values

The source states that tensor2d @ tensor2d.T accomplishes the same thing as the .matmul call and prints the same tensor. For this case you can treat the two spellings as interchangeable. One is a method call and the other is an infix operator, but both return the matrix product.

Does calling .matmul follow the same path as using the @ operator, and do they return the same output for the same inputs?

How Each Entry Is Built

Each entry of a matrix product is built from one row of the first matrix and one column of the second matrix. You pair the numbers in that row with the numbers in that column position by position, multiply each pair, and add the results together. The position of the entry in the output tells you which row and which column were used. The top-right entry, for example, comes from the first row of the first matrix and the second column of the second matrix.

Which row of the first tensor and which column of the transpose combine to produce the entry at row 0, column 1, and how are their values multiplied and summed?

Worked example

Multiplying a 2 by 2 matrix by its transpose

Let X have first row 2, 1 and second row 0, 3. Find X times its transpose, and predict the shape before computing it.

1
Predict the shape
X is 2 by 2 and its transpose is 2 by 2, so the product is 2 by 2.
2
Find the transpose
Rows become columns, so Xᵀ has first column 2, 1 and second column 0, 3. Its rows are 2, 0 and 1, 3.
3
Entry (0, 0)
Row 0 of X is 2, 1. Column 0 of Xᵀ is 2, 1. Products: 2×2 = 4 and 1×1 = 1. Sum: 5.
4
Entry (0, 1)
Row 0 of X is 2, 1. Column 1 of Xᵀ is 0, 3. Products: 2×0 = 0 and 1×3 = 3. Sum: 3.
5
Entry (1, 1)
Row 1 of X is 0, 3. Column 1 of Xᵀ is 0, 3. Products: 0 and 9. Sum: 9.

Result

The product is [[5, 3], [3, 9]]. The off-diagonal entries match, which reflects the symmetric structure of a matrix multiplied by its own transpose.

The entry in row i and column j is the sum over k of the product of the entry in row i, column k of the first matrix and the entry in row k, column j of the second matrix. In compact notation this reads $C_{ij} = \sum_{k} A_{ik} B_{kj}$, where the index k runs across the shared inner dimension. Because a product of a matrix with its transpose uses the rows of the original on both sides, the entry in row i and column j equals the dot product of row i and row j.

Predicting the Shape of a Product

What do you think happens?

The input tensor2d in the program has shape 2 by 3. Before running anything, what shape will tensor2d.matmul(tensor2d.T) have?

2 by 2
3 by 3
2 by 3
3 by 2

Reveal answer

Answer:

2 by 2

A 2 by 3 matrix times a 3 by 2 matrix produces a 2 by 2 result. The inner dimensions (3 and 3) match and are consumed by the sum, and the outer dimensions (2 and 2) become the shape of the result.

Shapes and order determine whether a product is defined and what shape it has, so check the shape of each operand before you multiply. For an m by n tensor A and an n by p tensor B, the product A @ B is defined only when the inner dimensions agree (both equal n). When they do, the result has shape m by p: the outer dimensions of the two inputs.

Given an m by n tensor and an n by p tensor, which inner dimensions must match, and which outer dimensions become the shape of the result?

Operand shapes	Defined?	Result shape
2 × 3 and 3 × 2	Yes	2 × 2
2 × 3 and 2 × 3	No, inner dimensions 3 and 2 differ	Not applicable
2 × 2 and 2 × 2	Yes	2 × 2
m × n and n × p	Yes	m × p

Checking whether a product is defined, and what shape it has, from the operand shapes

Mistakes When Predicting or Debugging Products

Common mistake
Assuming the two spellings can give different answers
What it looks like
Expecting tensor2d.matmul(tensor2d.T) and tensor2d @ tensor2d.T to differ because one is a method and one is an operator.
Why it fails
The source shows that the @ operator accomplishes the same multiplication and prints the same tensor.
Do this instead
Treat them as interchangeable for this case, and confirm with a comparison such as torch.equal if you need proof.
Common mistake
Multiplying without checking operand shapes
What it looks like
Trying to multiply a 2 by 3 tensor by another 2 by 3 tensor directly.
Why it fails
The inner dimensions (3 and 2) do not match, so the product is not defined.
Do this instead
Check the shape of each operand first, and transpose one operand if the inner dimensions need to line up.
Common mistake
Pairing a row with a row when the entry needs a column
What it looks like
Computing the top-right entry of X @ Xᵀ using row 0 of X and row 0 of X.
Why it fails
The top-right entry uses the second column of the second factor, which corresponds to row 1 of X in the transposed relationship.
Do this instead
Locate the output position first, then take the matching row of the first matrix and the matching column of the second.
Common mistake
Reading the output position backwards
Why it fails
Row and column indices of the result identify which row of the first matrix and which column of the second were used, so swapping them points to the wrong pair.

Tracing a Result That Looks Wrong

Because the predicted shape and at least one predicted entry act as a reference point, a wrong output can be traced step by step. Work down the checks in order: confirm the shapes, then confirm the row and column pairing for the suspicious entry, then confirm the multiply-and-sum arithmetic.

At which step, shape check, row-column pairing, or summation, does a mismatch between the predicted and actual product first appear?

MEDIUM

Without running code, predict the full result of X @ Xᵀ for X whose first row is 1, 2 and second row is 3, 4. Then compute the top-left entry by pairing row 0 of X with column 0 of Xᵀ. Finally, run your prediction in Python and check which entry, if any, differed from your expectation.

Hints

The result is 2 by 2, because both operands are 2 by 2.
Column 0 of Xᵀ is the same numbers as row 0 of X.
Top-left entry: 1×1 + 2×2.

Your attempt (not saved)

Key Takeaways

PyTorch multiplies a 2D tensor by its transpose with either the .matmul method or the @ operator, and the source shows both produce the same result.
Each entry of a matrix product pairs one row of the first matrix with one column of the second, multiplies the pairs position by position, and adds the products.
The shape rule decides whether a product is defined: for an m by n tensor and an n by p tensor the product exists and has shape m by p.
Predicting the shape and an entry before running code gives a reference point, so a wrong output can be traced to a specific row and column.

Related concepts

More concepts starting with T