Tensor Shapes and Dimensions in PyTorch

When you start working with PyTorch, the first data structure you meet is the tensor. Before worrying about any particular operation, it helps to have a clear picture of what kind of object a tensor is and how the different names used for it relate to one another. This article approaches that picture by asking one question again and again: how many dimensions does this data have?

Tensors are the basic container you will pass around in nearly everything that follows in this module. The key idea is that a tensor is not just a pile of numbers. It is a container whose values sit in a definite arrangement, and that arrangement is what the dimension count describes.

The names scalar, vector, and matrix are not separate kinds of containers with unrelated rules. They are labels for tensors that have a particular number of dimensions. A tensor with zero dimensions is called a scalar, a tensor with one dimension is called a vector, and a tensor with two dimensions is called a matrix. Each name is simply a way of reading off the dimension count.

Seen this way, the vocabulary becomes a single ladder rather than a list of unrelated terms. Start at the bottom with zero dimensions and climb one dimension at a time. Each step up produces the next named case. The diagram below shows the same ladder read from the top down, where each level holds the level beneath it.

Read the diagram as a rule you can apply to any tensor. Adding one dimension means the new structure is a collection of the previous structure. A vector is a collection of scalar values. A matrix is a collection of vectors. A 3D tensor is a collection of matrices.

A scalar is a zero-dimensional tensor. It is the value itself with no arrangement around it. A vector is a one-dimensional tensor. Its values lie along a single dimension, so you need only one position to point at any one entry. A matrix is a two-dimensional tensor. Its values are arranged along two dimensions, so picking out one entry takes two positions.

The number of positions you need is the same as the number of dimensions, which is a useful way to check your reasoning. If you are unsure whether a tensor is a vector or a matrix, ask how many positions you would need to point at a single entry. One position means a vector. Two positions means a matrix.

Once you pass two dimensions, there is no specific term in the source material. Rather than inventing a new word for each case, the convention is to name a tensor by its dimension count. A tensor with three dimensions is called a 3D tensor, and the same pattern extends to any higher count. The naming scheme stays consistent as the data grows more complex.

The source presents tensors being created with the torch.tensor function. The naming ideas apply directly to the objects you create that way. Whatever values you pass to torch.tensor, the resulting object is classified by how many levels of arrangement it has. The code below uses nested brackets to build tensors of increasing dimension count.

Each level of brackets corresponds to one dimension. The outermost level groups the whole 3D tensor into its matrices, the middle level separates the rows within one matrix, and the innermost level holds the individual values of one row. Counting the bracket levels gives you the same dimension count you would get by counting positions.