Tensor Creation

Learning to follow the state of the tensor line by line is the skill this article is building.

Programming

A Short Program With Long Consequences

This article follows a short PyTorch program in which one small tensor is created, inspected, reshaped, transposed, and multiplied by its own transpose. Each step is simple on its own. The reason to practice them together is that a single wrong assumption early in the program, such as a shape you did not expect, carries forward into every later line and can look like a failure somewhere else entirely. Learning to follow the state of the tensor line by line is the skill this article is building.

What do you think happens?

A tensor has shape 2 by 3. What shape do you expect after you apply .T to it?

2 by 3
3 by 2
6 by 1
Its shape cannot change

Reveal answer

Answer:

3 by 2

The transpose swaps the two dimensions, so the 2 becomes the second dimension and the 3 becomes the first. Rows turn into columns and columns turn into rows.

Starting Tensor and Its Shape

The starting tensor is built from a list of two inner lists, each holding three numbers. PyTorch reads this nested structure and creates a two-dimensional tensor. Its shape attribute reports how many entries lie along each dimension. In this case the first dimension has two entries (the two inner lists) and the second dimension has three entries (the numbers in each inner list). The shape is the first thing to check whenever a later operation behaves strangely.

python

import torch
x = torch.tensor([[1, 2, 3], [4, 5, 6]])
print(x)
print(x.shape)

import torch
x = torch.tensor([[1, 2, 3], [4, 5, 6]])
print(x)
print(x.shape)

Output (expected)

tensor([[1, 2, 3],
        [4, 5, 6]])
torch.Size([2, 3])

Notes on the code

The outer list has two inner lists, so the first dimension has size 2.
Each inner list has three numbers, so the second dimension has size 3.
The shape attribute is the first value to inspect when a later line misbehaves.

Reshape Regroups Values Without Changing Order

The reshape method takes a new shape and returns a tensor with that shape. The starting tensor holds six values, so any new shape must also account for exactly six entries. Asking for a shape with a different total count cannot succeed. The total stays fixed because reshape only regroups the same values: $2 \times 3 = 6 = 3 \times 2$.

The important point is the order of the values. Reshaping does not shuffle or recalculate anything. The six values keep their sequence, and the new shape determines where each row begins and ends. Reading the same six values in order into three rows of two produces a different grouping of the same numbers, not a different set of numbers.

When the six values 1 through 6 are reshaped into 3 rows of 2, each value keeps its place in the sequence and simply moves into the next available cell, filling row 0 first, then row 1, then row 2.

python

y = x.reshape(3, 2)
print(y)
print(y.shape)

y = x.reshape(3, 2)
print(y)
print(y.shape)

Output (expected)

tensor([[1, 2],
        [3, 4],
        [5, 6]])
torch.Size([3, 2])

Notes on the code

The new shape is passed as two arguments: 3 rows and 2 columns.
The pairs are 1 and 2, then 3 and 4, then 5 and 6, which is the original sequence regrouped.

Worked example

Predicting a reshape before running it

Starting from a tensor with six values in order 1 through 6, predict the result of reshape to 3 rows and 2 columns, and state where the value 4 ends up.

1
Count the total entries
The tensor holds six values, so the new shape must hold six entries. 3 times 2 equals 6, so the request is valid.
2
Fill rows in order
Reading the values in their original sequence, the first row takes 1 and 2, the second row takes 3 and 4, and the third row takes 5 and 6.
3
Locate the value 4
The value 4 is the fourth entry in the sequence, which falls in the second row at the second column.

Result

The result has shape 3 by 2 with rows (1, 2), (3, 4), (5, 6), and the value 4 sits at row 1, column 1 when counting from zero.

Tracing Values Through Transpose

The transpose attribute, written .T with no parentheses, flips a two-dimensional tensor so that its rows become its columns and its columns become its rows. The first row of the original becomes the first column of the result, and the same pattern continues for every row. Because the two dimensions are swapped, a tensor of shape 2 by 3 becomes shape 3 by 2.

python

z = x.T
print(z)
print(z.shape)

z = x.T
print(z)
print(z.shape)

Output (expected)

tensor([[1, 4],
        [2, 5],
        [3, 6]])
torch.Size([3, 2])

Notes on the code

The first printed row, 1 and 4, is the first column of the original.
The second printed row, 2 and 5, is the second column of the original.
The shape changes from 2 by 3 to 3 by 2, matching the prediction made earlier.

EASY

Without running the code, write down the first row of x.T for the tensor x = [[1, 2, 3], [4, 5, 6]]. Then say which original row supplied the first column.

Hints

Look at the first column of the original tensor, not the first row.
The first column of the original is 1 on top and 4 below.

Your attempt (not saved)

Matrix Multiplication and Inner Dimensions

The matmul method multiplies two tensors as matrices. In this program the tensor is multiplied by its own transpose. The original is 2 by 3 and the transpose is 3 by 2. The inner dimensions, the 3 at the end of the first shape and the 3 at the start of the second, agree, so the product is defined and has shape $2 \times 2$. The outer dimensions, 2 and 2, give the shape of the result.

In the expression A.matmul(B), the last dimension of A and the first dimension of B must match. The matching inner dimension is consumed, and the outer dimensions of A and B form the shape of the result.

python

P = x.matmul(x.T)
print(P)
print(P.shape)

P = x.matmul(x.T)
print(P)
print(P.shape)

Output (expected)

tensor([[14, 32],
        [32, 77]])
torch.Size([2, 2])

Notes on the code

x has shape 2 by 3 and x.T has shape 3 by 2, so the inner 3 values agree.
The result has shape 2 by 2, built from the outer dimensions.

Worked example

Checking the top-left entry by hand

Verify that the entry in row 0, column 0 of the product is 14, using the first row of x and the first column of x.T.

1
Pick the row and column
The first row of x is 1, 2, 3. The first column of x.T is the first row of x, which is also 1, 2, 3.
2
Multiply pairs and add
Multiply the first row by the first column entry by entry: 1 times 1, plus 2 times 2, plus 3 times 3.
3
Sum the products
1 plus 4 plus 9 equals 14, which matches the top-left value printed by the program.

Result

The top-left entry is 14, so the printed product is consistent with the rule that each output entry combines one row of the first tensor with one column of the second.

Mistakes Beginners Make With Shapes

Common mistake
Expecting reshape to reorder or multiply the values
What it looks like
Expecting x.reshape(3, 2) to produce 1, 4, 2, 5, 3, 6 in the first row
Why it fails
Reshape regroups the same six values in their original sequence and does not recalculate anything.
Do this instead
Read the values in order and fill each row from left to right before checking the result.
Common mistake
Asking for a shape with the wrong total count
What it looks like
Calling x.reshape(2, 4) on a tensor that holds six values
Why it fails
2 times 4 is 8, but the tensor holds only 6 entries, so the total must match exactly.
Do this instead
Multiply the requested dimensions and confirm the product equals the number of elements before running the call.
Common mistake
Treating .T as a method call
What it looks like
```
Writing x.T() instead of x.T
```
Why it fails
The transpose is an attribute, written without parentheses, as the source describes.
Do this instead
Use x.T with no parentheses.
Common mistake
Multiplying tensors without checking the inner dimensions
What it looks like
```
Calling x.matmul(x) on a 2 by 3 tensor
```
Why it fails
The last dimension of the first tensor is 3 and the first dimension of the second is 2, so they do not agree.
Do this instead
Use x.matmul(x.T) so the 3 and 3 agree, or check the shapes first with print statements.
Common mistake
Assuming a surprising result is normal reshape behavior instead of checking the input shape

Finding Where a Result First Goes Wrong

When an output differs from what you expected, the most efficient approach is to move forward through the program one operation at a time and check the shape and values after each. The first line where the observed result departs from your prediction is the failure location. Everything after that point may be a consequence rather than a separate cause. This keeps you from debugging the matmul line when the real mistake was the shape that reshape produced several lines earlier.

1
Print the shape of the starting tensor and compare it with what you expected.
2
After each reshape or transpose line, print the new shape and the values.
3
Compare each printed result with your prediction for that line.
4
Stop at the first line where the printed result departs from the prediction. That line is the failure location.
5
Check the inner dimensions of any matmul before concluding that the multiplication itself is wrong.

Code walkthrough

x = torch.tensor([[1, 2, 3], [4, 5, 6]])
print(x.shape)
y = x.reshape(3, 2)
print(y.shape)
z = x.T
print(z.shape)
P = x.matmul(z)
print(P)

1
Line 2 confirms the starting shape. If this prints anything other than 2 by 3, stop here; every later line depends on it.
2
Line 4 confirms that the reshape produced 3 by 2 with six entries.
3
Line 6 confirms that the transpose swapped the dimensions to 3 by 2.
4
Line 7 multiplies the 2 by 3 tensor by its 3 by 2 transpose, which is valid because the inner 3 values agree.
5
If the final product were wrong while all earlier shape prints were correct, the failure would be in the values, and the top-left entry could be checked by hand as 14.

Output

torch.Size([2, 3])
torch.Size([3, 2])
torch.Size([3, 2])
tensor([[14, 32],
        [32, 77]])

Trace the Program on Your Own

MEDIUM

A tensor w is created with shape 2 by 3 and holds the values 1 through 6 in order. Predict the shape of w.reshape(3, 2) and the shape of w.matmul(w.T). Then state whether w.reshape(2, 4) can succeed, and explain why.

Hints

Multiply the requested dimensions for reshape and compare the product with the number of elements.
For matmul, the inner dimensions are the last of the first shape and the first of the second shape.

Your attempt (not saved)

Key Takeaways

Reshaping regroups a fixed sequence of values into a new shape. The total number of elements must stay the same, and the values keep their original order.
The .T attribute swaps the two dimensions of a two-dimensional tensor, so the first row of the original becomes the first column of the transpose.
The .matmul() method requires the inner dimensions to agree. The result takes its shape from the outer dimensions, and each entry combines one row of the first tensor with one column of the second.
.view() requires contiguous data and fails otherwise, while .reshape() copies data when necessary to produce the requested shape.
When output is wrong, trace the program line by line and compare shapes and values with predictions. The first line where they diverge is the failure location.

Related concepts

More concepts starting with T

A Short Program With Long Consequences

Starting Tensor and Its Shape

Reshape Regroups Values Without Changing Order

Tracing Values Through Transpose

Matrix Multiplication and Inner Dimensions

Mistakes Beginners Make With Shapes

Finding Where a Result First Goes Wrong

Trace the Program on Your Own

Key Takeaways

Related Topics

Related concepts