Matrix Multiplication on the GPU

A common surprise in GPU work is that a script which ran quickly for a colleague runs more slowly on your machine, even though the lines of code match exactly. This article treats that gap as a debugging problem. Instead of asking whether GPU acceleration works in general, we ask a narrower question: when a result differs from what you expected, where in the code or the setup should you look first?

The core experiment is short. Two tensors are created, then multiplied with the @ operator, and the time for that single operation is measured with IPython's timing magic. The same measurement is repeated after moving both tensors onto the GPU. The first step is the move itself, which is a separate line from the multiplication.

Read the two lines of output carefully. The CPU figure and the GPU figure each carry a plus-or-minus value, which describes how much individual runs varied around the average. On the hardware used in the source, the GPU run was roughly four times faster. That ratio is the number worth remembering, because it is specific to that machine and that matrix size.

The speed-up depends on where the operation runs, and where it runs depends on where the tensors are. Calling the matrix multiplication does not by itself choose a device. The tensors carry that information. Moving them with .to("cuda") changes the device the multiplication is performed on. If either tensor is left behind on the CPU, the timing you measure is no longer a measurement of the GPU path.

Trace the flow in the diagram. The multiplication takes its device from its inputs. Two lines of your script can move the data, while a third line can look identical whether it is running on CPU or GPU. The practical check is to confirm, before reading any timing, that both tensors were actually moved to the device being measured.

A speed-up is a ratio of two measured times. The source reports one ratio, roughly four, but a single ratio hides how much the underlying runs varied. Written as a formula, the ratio compares the CPU time against the GPU time:

A ratio is only as trustworthy as the two times that produce it. Each time carries a plus-or-minus value describing run-to-run spread. When the spread around either time is large, a ratio that looks precise may fall anywhere within a wide band. Before repeating a speed-up claim, attach three things to it: the hardware on each side, the matrix size, and the measurement spread.

The source's second half rebuilds a model from an earlier chapter, so the rest of the material starts from a known state. Here the consistency question moves from speed to reproducibility. Two runs that build the same model with different random starting values will not produce identical training behavior, so the setup code has to pin down what it can.

Consistent model setup depends on two choices made in order. First, the device is chosen. Second, the random seed is fixed relative to model construction. The seed fixes the random starting values, so its placement matters: a seed set after the model is built does not control the values that were already drawn.

Data is the third source of variation. The source loads a short story, splits its text at ninety percent, and builds two loaders, one for training and one for validation. The split point is computed from the length of the text, so the same file always produces the same boundary. The loader settings then decide how batches are formed.

The seed is set again immediately before the training loader is created. That placement is deliberate: the shuffled batch order is random, and seeding just before construction keeps that order repeatable. The validation loader is not shuffled, so it does not need the same protection, although the split itself still has to be computed the same way every time.

Use this pipeline to locate a divergence in the data path. If two runs see batches in a different order, look first at whether the seed was set before the training loader was built. If the validation numbers shift, look at whether the split was computed from the same text with the same ratio.