Monitoring Training Progress Across Multiple Processes

Training a PyTorch model on several GPUs raises a practical question before any distributed logic enters the picture: which GPUs will the program actually touch? A machine might have one GPU, two, or four, and the same script may need to run on all of them. This article follows a single script, named ch02-DDP-script.py, through these situations. At each point it asks where the observed behavior comes from and where the output can go wrong.

The script is designed to run on both single-GPU and multi-GPU machines, so its code does not change between those cases. The differences you will observe come from two places: the environment the script runs in, and the way the distributed training machinery spreads work across the available GPUs. Keeping those two sources separate is the central habit of this article. When you look at a run, the goal is to say which line of reasoning explains each thing you see.

Before looking at multiple GPUs, it helps to know what a single-GPU run should look like. When the script runs on one GPU, the first lines report the PyTorch version, whether CUDA is available, and how many GPUs were found. Then each epoch appears with its batch size and its training or validation loss. The tag at the start of each line, GPU0, identifies which process printed that line.

The source treats this output as a sanity check. A sanity check is a quick comparison that tells you the basic setup behaves as expected before you trust the detailed results. For a single GPU, the header should report one GPU, and the per-epoch structure should match what you would anticipate for one process. Once that baseline is clear, any difference in a multi-GPU run stands out.

The simplest way to restrict how many GPUs a training run uses is the CUDA_VISIBLE_DEVICES environment variable. You set it on the same line you use to start the script, so the restriction applies only to that one run. Instead of launching the script plainly, you prefix the command with the variable assignment. The variable takes GPU indices, and the script sees only the GPUs you list.

The variable also accepts a comma-separated list of indices. The source describes a machine with four GPUs where you want only the first and the third. You list those indices together, separated by commas, on the same command line. The same script then runs with exactly those GPUs visible.

The diagram above shows the remapping that matters when you read output. The script does not refer to the physical numbering of the machine. It refers to the GPUs it was given, starting again from zero. Listing indices 2 and 3 therefore produces devices named cuda:0 and cuda:1 inside the script, even though the hardware keeps its original labels.

The reason this approach is convenient is that GPU allocation happens without modifying your PyTorch scripts. The same file can be launched with different environment settings to use different hardware. That keeps the code independent of the machine it happens to run on, which is useful when one script must work on a laptop with one GPU and a server with four.

Now run the same command and the same script on a machine with two GPUs. The header now reports two GPUs available. The epoch lines carry tags for both GPU0 and GPU1, and they appear in an interleaved order rather than in a single tidy sequence. This interleaving is what you would expect when two processes run at the same time, and it is the first thing to notice when you read a multi-GPU log.

Trace the last four lines of this log with a specific question in mind: how many times does the test accuracy appear, and from which processes? You will find one line from GPU1 and one from GPU0. Those two lines are not a bug in the arithmetic. They are two independent processes each reporting the same kind of result, and that is the place where the duplicated output comes from.

The GPU tag on each line is your main tool for reading a multi-process log. It tells you which process wrote the line. When two lines have identical message text, the tag is what separates them. A repeated message with two different tags is a record of two processes, not a mistake in the result.

The source explains the duplication with a direct chain of reasoning. Distributed Data Parallel, usually called DDP, replicates the model onto each GPU, and each process runs independently. In this setup a process and a GPU correspond to one another: each process drives its own GPU with its own copy of the model. Because each copy works on part of the data, the batches of training examples are divided between the processes.

The chain continues into the output. If your code contains a print statement inside the testing loop, then every process reaches that statement and executes it. Each execution produces a line, so the same message appears once per process. Nothing in this chain requires the processes to coordinate their printing. Each one simply does what the code tells it to do.

If the duplication bothers you, the source suggests using the rank of each process to control print statements. The rank identifies which process is executing the code. A condition that checks for rank zero lets only the first process perform the print, and the other processes skip it. The guard is an ordinary if statement wrapped around the print call.

Notice what this change does and does not do. It changes which process prints the line, so the message appears once. It does not change the computation: every process still runs the testing loop and computes accuracy. The guard controls only the output step. This separation is worth keeping in mind whenever you add a guard, because a guard placed around the computation itself would change what the training run does.

DDP is not the only route to multi-GPU training in PyTorch. The source mentions add-on APIs for readers who prefer a more straightforward way to use several GPUs. One example it names is the open-source Fabric library, which it describes as an add-on API for PyTorch. The source points to a separate write-up on Fabric for further detail, so this article does not teach Fabric's own interface.

What you can take from this section is the idea of an add-on API: a library that sits alongside a PyTorch training loop and helps manage multi-GPU use. The output issues in this article, including the repeated lines and the rank-based guard, are the kinds of concerns you will still need to reason about when reading the logs of any multi-GPU setup.