Threads and Networking Essentials

Imagine a server that talks to several clients at once. Most of the time its threads are not calculating anything. They are waiting for network input and output to finish. Before you learn how the server should be designed, make a prediction. Suppose the server runs on a machine with a small number of processors, and each of its threads is blocked around 90 percent of the time while it waits for network traffic. How many threads would you expect the server to want, compared with the number of processors?

The source is clear that waiting is not an unusual event for network code. It is part of how such code normally behaves. A network program therefore has to be written with that waiting in mind, rather than assuming that every read returns immediately with data.

Blocking is not always a problem. A command line program that reads text typed by the user can simply stop and wait, and user input is another form of blocking input and output. What separates a network program from that case is what else the program could be doing during the wait. A program with nothing else to do can afford to wait. A program with other work available benefits from letting some parts keep running while other parts are blocked.

Follow a single thread through one network read. The thread asks the network for data. It then has to wait for an unpredictable amount of time. When the data arrives, the thread can continue. The diagram below shows these steps as messages between the thread and the network. The source does not describe the internal mechanism that wakes the thread when data arrives, so the diagram shows only the sequence of events, not how the wake-up happens.

While the thread waits, it does not compete for the processor. Only the active, unblocked threads compete for processing time. This is the key fact behind everything else in this article. A waiting thread takes up a place in the program's design, but it does not take up processor time.

Threads can be added to a program for two different goals, and the right number of threads depends on which goal you have. The first goal is computational speedup: using threads to make calculations finish sooner. The second goal is handling blocking input and output: letting the program keep working while some of its threads wait.

For computational speedup, one thread per available processor is the natural choice. If only one processor exists, adding more threads gives no speedup. It actually slows the program down because of the overhead of creating and managing the threads. Speedup is limited by how many threads can make real progress at the same time.

For blocking input and output, the threads are not all trying to compute at once. Many of them are waiting, and a waiting thread does not compete for the processor. This means a program can have many more threads than processors without overloading the processors, because most of those threads are not running at any given moment.

A common rule of thumb says that a program should never run more threads than the computer has processors. For many programs this is a sensible starting point, but it comes from one specific situation: threads that do computation and want to run at full speed. Applied to every program, it ignores what the threads are actually doing. The network case is the one where threads spend much of their time waiting.

This is why the useful number of threads for a blocking program is tied to how much of the time the threads spend waiting, not only to how many processors are present. The more a thread waits, the more threads the program can keep in its design without overloading the processors.

The rough target for blocking I/O is about one active thread per processor. Because mostly blocked threads are inactive for much of the time, that target can mean many more threads in total. The calculation below turns the blocked fraction into a rough thread count. It is an estimate, not a fixed answer.

A thread in a network program moves between two situations. While it waits for data, it is blocked and does not compete for processor time. When the data it needs has arrived, it is active and competes for processing time again. The diagram shows this back-and-forth. The source does not give a detailed list of thread states, so this diagram keeps to the two situations the source describes.

Because a blocked thread does not compete for processor time, a program with many threads can keep most of them in the blocked situation at any given moment. The processors are then shared among the few threads that are active. This is the reason many threads can be useful for network code even when there are few processors.

The source contrasts two network programs. A client that sends requests to a server may be fine with a single thread, if it has nothing else to do while it waits for responses. A server is different. It can usually be connected to several clients at the same time, and while it waits for data from one client, it can communicate with the others.

When the server gives each client connection its own thread, a block on one connection does not stop the server from talking to the rest. This is the situation the source describes as especially valuable for servers that serve several clients at once.