PyTorch Tensors and Data Types
All tensors in one computation must be on the same device.
Why Device Location Changes the Arithmetic
Adding two tensors looks the same on the page no matter where the numbers are stored. In everyday mathematics, a sum does not care where its operands live. In PyTorch, it does. Every tensor lives on a particular device, and a computation has to run with its inputs in agreement. This article focuses on that one requirement: keeping the tensors involved in a computation on the same device, and knowing what PyTorch does when they are not.
Many libraries hide details like this, so it is tempting to assume PyTorch will notice that one tensor is on the CPU and another is on a GPU and quietly sort it out before doing the arithmetic. That is a reasonable guess, but the source material warns directly against it. Test your expectation before reading on.
What do you think happens?
What happens when you add two tensors where one lives on the CPU and the other lives on the GPU?
Reveal answer
The operation fails with a RuntimeError that names the devices involved.
Computation with tensors on different devices does not succeed by default. PyTorch does not silently repair the mismatch, so the programmer has to bring the operands onto one shared device.
Where Tensor Values Live
Think of each tensor as carrying a label that says which device holds its values. A tensor on the CPU and a tensor on the GPU are, in this sense, stored in different places. The diagram below shows two tensors whose values sit in separate memory locations. Look at which tensor is in which location before deciding whether they can be combined in one computation.
The lesson of the diagram is that each tensor's device is a fact about where its values are held. A computation combines several tensors, so the library needs all of them to point to the same place before it can do the work. Your job as the programmer is to establish that agreement. Once it holds, PyTorch carries out the rest of the computation.
The Same-Device Rule
Every tensor participating in one computation must be on the same device. If any operand sits on a different device from the others, PyTorch raises a RuntimeError, and the error message names the devices involved.
The rule is about agreement, not about a preferred device. The devices can be whatever they are, as long as all the operands in the computation match.
The rule applies to the whole computation, not to pairs of tensors chosen at random. If an expression combines three tensors, all three must share a device. Checking only two of them and assuming the third is fine is a common source of errors.
Moving a Tensor with to(device)
PyTorch gives you one method for relocating a tensor: calling to with the target device as its argument. The general pattern is tensor.to(device). The device argument names where the tensor should live. The result of the call is assigned back to a name, so the code that follows uses the relocated tensor.
Read the flow from left to right. The tensor is passed to to with the target device, and the call returns a tensor on that device. Assigning the returned value to a name is what lets the rest of your code use the relocated tensor. The source material demonstrates this with a CPU target.
Tracing a Mismatch That Moving One Tensor Does Not Fix
The source example is instructive because it does not succeed on its own. It starts with two tensors on the GPU, moves one of them to the CPU, and then adds it to the tensor that remains on the GPU. Relocating a single operand only changes which device is the odd one out. The tensor that was moved now matches nothing, and the mismatch persists.
a_gpu = ... # on the GPU
b_gpu = ... # on the GPU
# both operands agreetensor a: GPU, tensor b: GPU
a_cpu = a_gpu.to('cpu') # now on the CPU
b_gpu = ... # still on the GPU
# operands disagree againtensor a: CPU, tensor b: GPU
In the before state, the two operands agree. After the move, only tensor a has changed, and it now disagrees with tensor b. Moving one operand changed the agreement in the wrong direction. The fix is not to pick whichever tensor is convenient to move. The fix is to bring every operand to one common device.
Aligning Both Operands on One Device
Making two mismatched operands agree
a_gpu lives on the GPU and b_cpu lives on the CPU. You want to add them. Make the computation valid.
- 1Identify the devices
One operand is on the GPU and the other is on the CPU, so the operands disagree.
- 2Choose one shared device
Pick a single target device that both operands will use. Moving only
a_gputo the CPU would also work here, but the principle is that every operand ends on the chosen device. - 3Move each operand to that device
Call to(device) on each tensor and assign the returned value to a name, so the code uses the relocated tensors.
- 4Run the computation
With both operands on the same device, the addition can proceed.
Both operands are on one device, and the sum is computed without a device mismatch.
Reading the RuntimeError
When operands sit on different devices, PyTorch raises a RuntimeError. The source material states that the error message names the devices involved. Your first task on seeing this error is to read those device names, because they tell you which operands disagree. Do not guess the devices from memory when the error already reports them.
The devices named in the error are the evidence you need. Compare them against where you believe each tensor was created or moved, then decide on one shared device for all operands.
Checking Device Alignment Before Computing
- 1List every tensor that will take part in the computation.
- 2Determine which device each tensor currently lives on.
- 3Choose one shared target device for the whole computation.
- 4Call
tensor.to(device)on every operand that is not already on that device, and assign each returned tensor to a name. - 5Confirm that all operands now share the device, then run the computation.
Common Mistakes with Tensor Devices
Common mistake Assuming PyTorch will move tensors automatically before doing arithmetic
Why it failsThe source material warns that computation with tensors on different devices does not succeed by default.
Do this insteadTreat device agreement as your responsibility and check it before running the operation.
Common mistake Moving only one tensor and expecting the error to disappear
What it looks likeMoving tensor a to the CPU while tensor b stays on the GPU leaves the two operands disagreeing.
Why it failsRelocating one operand only changes which device is the odd one out, so the mismatch can persist.
Do this insteadMove every operand to one common device.
Common mistake Forgetting to assign the result of to(device)
Why it failsto(device) returns a tensor on the target device, so calling it without using the returned value leaves the original tensor where it was.
Do this insteadAssign the result to a name and use that name in the computation.
Common mistake Ignoring the device names in the RuntimeError and guessing instead
Practice Aligning Devices
You have three tensors that will be combined in one expression. Tensor x is on the GPU, tensor y is on the CPU, and tensor z is on the GPU. Describe the steps that make the expression valid, and explain why moving only y is not enough if you then want to use a device that x and z do not share.
Hints
- Start by listing the device of every operand.
- Every operand in the expression must end on the same device, including z.
- Moving one operand only changes which device it matches.
Key Takeaways
- All tensors in one computation must be on the same device, and PyTorch does not silently repair a mismatch.
- Mixed devices produce a
RuntimeError, and the error message names the devices involved. tensor.to(device)moves a tensor to a target device, and the returned tensor should be assigned to a name for later use.- Moving only one operand may not fix the problem. Align every operand on one shared device before running the computation.