Timing Code with %timeit in Notebooks

A GPU is often described as the fastest way to run the heavy arithmetic inside neural networks, and matrix multiplication is one of the most common of those operations. Knowing that a GPU is faster in general is not the same as knowing how much faster it is for one specific operation, on one specific machine, with specific matrix sizes. This article shows how to answer that question directly. You will move two tensors onto a GPU, time the same multiplication on both devices, and reason about what the two timings tell you.

The experiment has three parts, and their order matters. First, two random matrices are created on the CPU. Second, both are moved to the GPU. Third, the same matrix multiplication is timed on each device. The timing result only makes sense once you know which device held the data at the moment the multiplication ran, so keep that question in mind as you read: when a GPU result differs from what you expected, where in this sequence of steps would you look first?

Moving a tensor to a CUDA device is a single method call: a.to("cuda"). The call takes a tensor that currently lives on the CPU and places its data on the CUDA device, which is the GPU that PyTorch uses for this work. The call does not change the values in the tensor. It changes where those values are stored and therefore which processor can work on them directly.

The %timeit magic is a notebook command. You place it before a statement, and it runs that statement several times and reports the typical time for one run. The result is written as a mean plus or minus a spread, followed by per loop. The mean is the central estimate of how long one run takes. The value after the plus-minus sign shows how much the individual runs varied around that estimate. A small spread means the timing was stable from run to run, and a large spread means you should be cautious about trusting the mean.

Two timing results are only comparable when they measure the same statement on the same inputs. In this experiment the statement is a @ b in both cases, and the inputs are the same two matrices. The only intended difference between the two measurements is the device on which the tensors reside when the statement runs. If you changed the matrix sizes, or used different data between runs, the two numbers would no longer be a fair comparison.

Now put the pieces together. You create the matrices on the CPU and time the multiplication there first. Then you move both matrices to the GPU and time the same statement again. The transfer happens before the second %timeit line, so the timed statement is only the multiplication itself. The flowchart below shows the order of the steps on the GPU path.

On the CPU path the move step is skipped, because both matrices already live on the CPU. The timed statement is the same in both cases, so the only thing that changed between the two measurements is which processor runs the multiplication and where its operands sit. The comparison below sets the two paths side by side.

A speedup is usually reported as a ratio of the two times. In this article the ratio is the CPU time divided by the GPU time, so a value above one means the GPU run finished in less time. The relationship is written as follows, where t_{CPU} is the mean time on the CPU and t_{GPU} is the mean time on the GPU.

The source summarizes its own experiment by saying the computation was approximately four times faster on a V100. That is a rounded statement about one experiment. The way to check it is to calculate the ratio from the timing values yourself, using the means reported on your own machine.

The ratio depends on the hardware because it is a measurement of one processor pair on one operation. The same code can give a different ratio on a different GPU or a different CPU, because each machine has its own processors and its own timing behaviour for this statement. The ratio also does not say that every operation will speed up by the same factor, and it does not predict what another GPU will do. A speedup number is a property of a specific comparison, not a fixed property of GPUs.