Introduction to Tensors in PyTorch

PyTorch is a large library with many functions, classes, and modules, so a newcomer can easily feel lost on the first day. Trying to memorize every part is a poor starting strategy. A better question is: what broad jobs does the library perform? The source material answers this by grouping PyTorch into three broad components: a tensor library, an automatic differentiation engine, and a deep learning library. Each component solves a different problem, and the three are designed to work together.

This article follows one realistic task through all three components. Each component is then tied to a concrete stage of work rather than being left as an abstract label.

Imagine you are training a small neural network to recognize handwritten digits. Before the network can learn anything, your images must be held in memory as numbers. The network multiplies those numbers by a set of adjustable weights, compares its output with the correct answer, and then nudges the weights so the next guess is a little better. Repeat this thousands of times and the network gradually improves.

Notice that the question has a clear answer only once you know what each component is for. The rest of this article builds that knowledge one component at a time, then returns to the same loop to assign each stage to its owner.

PyTorch's tensor library is built as an extension of NumPy, which is an array-oriented programming library. Someone who already knows how to work with NumPy arrays will find many familiar ideas here. The important difference is that PyTorch adds a feature NumPy alone does not provide: it can accelerate computation on GPUs.

Because of this GPU support, PyTorch offers a seamless switch between CPUs and GPUs. In practical terms, the same tensor-based computation can run on either kind of processor, and the library handles the difference rather than forcing the programmer to rewrite the computation for each device. This is the base layer that the other two components are built on.

Deep learning training depends on gradients, because they tell the model which direction to move its weights. Without gradients, the network in the digit-recognition task would have no principled way to nudge its weights toward better guesses.

Computing these gradients by hand becomes impractical once a model contains many operations. Autograd removes that burden. Because the engine handles gradient computation for tensor operations, the process of backpropagation is simplified. Backpropagation passes error information backward through the model to update its weights. The same mechanism also supports model optimization, since optimization steps rely on the gradients that autograd supplies.

The third component is a deep learning library. It offers modular, flexible, and efficient building blocks for designing and training deep learning models. The word modular matters here. Instead of writing every part of a network from scratch, a developer assembles components that are already defined and can be combined in different ways.

The source notes that these pieces serve both researchers and developers who design and train a wide range of deep learning models. In the digit-recognition task, the deep learning library is the component that organizes the network layers, the error measure, and the update rule into reusable parts.

The three components are not three unrelated toolboxes. The source describes a layered relationship. The deep learning utilities make use of the tensor library and the automatic differentiation engine. The deep learning library therefore sits on top, while the tensor library and autograd form the foundation it relies on.

The diagram makes one dependency easy to miss. Autograd does not compute gradients in isolation. It operates on the tensors and tensor operations provided by the tensor library. This is why the source lists the tensor library first and describes autograd as depending on it. Learning the three components in that order keeps each new idea anchored to the one before it.

Return to the digit-recognition loop with the three roles in mind. Data enters as tensors, the model performs tensor operations on it, autograd produces gradients from those operations, and the gradients then drive backpropagation and the optimizer's weight updates. The diagram below traces that path in order.

Read the flow from left to right and notice what stays out of your hands. You supply the data and the model structure. The autograd engine supplies the gradients, and the optimization step consumes them. This is the handoff the source describes when it says autograd simplifies backpropagation and model optimization, with no manual gradient calculation in the middle.