Network Client and Server Programs

Network input and output is blocking I/O, because those operations can block for indefinite periods of time, so network programs must be prepared for waiting.

Programming

Predicting a Server's Thread Count

A common rule of thumb says a program should never run more threads than the computer has processors. For many programs that rule is a sensible starting point. It comes from one specific situation, though: threads that are doing computation and want to run at full speed. Network programs often look different. Their threads spend much of their time waiting for data to arrive or for a send to finish. This article follows how a waiting thread behaves, and then uses that behavior to work out how many threads a network program can actually use.

What do you think happens?

Suppose a server runs on a machine with a small number of processors, and each of its threads is blocked around 90 percent of the time while waiting for network input and output. Compared with the number of processors, how many threads would you expect such a server to want?

About one thread per processor
Several times more threads than processors
Fewer threads than processors

Reveal answer

Answer:

Several times more threads than processors. Each thread is active only about 10 percent of the time, so the useful thread count grows well beyond the processor count.

The source's rough target is about one active thread per processor. If most threads are blocked, many more threads can exist while only about one active thread per processor is running at any moment. The exact number depends on how much blocking is actually measured.

What Blocking I/O Means

Network input and output is called blocking I/O because those operations can block for indefinite periods of time. A program that asks the network for data may have to wait an unpredictable amount of time before the data arrives. While it waits, the thread that made the request cannot continue with the next step. The source is direct about what this means for network code: programs that use the network must be prepared to deal with this blocking. Waiting is not an unusual event for network code. It is part of how the code normally behaves.

Follow one thread through a single network read. It starts out running, asks the network for data, and then becomes blocked. A blocked thread does not compete for the processor. When the data arrives, the thread can move back to running and continue its work. The diagram traces those three moments.

What state is the thread in while a network read is pending, and what makes it move back to running?

When Waiting Is Acceptable

Blocking is not always a problem. In some programs it is acceptable to stop all other processing and wait for input. A command line program that reads input typed by the user is one example, and user input is another form of blocking input and output. The difference with a network program is what else the program could be doing during the wait. A program with nothing else to do can afford to wait. A program with other work available benefits from letting some parts keep running while other parts are blocked.

Client Versus Server Waiting

The source contrasts two network programs. A client that sends requests to a server might be fine with a single thread, provided it has nothing else to do while it waits for the server's responses. A server is different. It can usually be connected to several clients at the same time, and while it waits for data from one client, it can communicate with the others. When the server gives each client connection its own thread, a block on one connection does not stop the server from talking to the rest.

python

import threading
import socket

def handle_client(conn):
    # Each connection gets its own thread.
    # A blocking recv here only blocks this thread.
    data = conn.recv(1024)
    conn.sendall(data)
    conn.close()

def serve(listener):
    while True:
        conn, _ = listener.accept()
        worker = threading.Thread(target=handle_client, args=(conn,))
        worker.start()

import threading
import socket

def handle_client(conn):
    # Each connection gets its own thread.
    # A blocking recv here only blocks this thread.
    data = conn.recv(1024)
    conn.sendall(data)
    conn.close()

def serve(listener):
    while True:
        conn, _ = listener.accept()
        worker = threading.Thread(target=handle_client, args=(conn,))
        worker.start()

Notes on the code

Each accepted connection is handed to its own thread.
recv blocks only the thread that called it, so other clients keep being served.
The accept loop stays free to take new connections.

Two Reasons to Use Threads

The reason the processor rule breaks down becomes clear when you compare two goals for threads. The first goal is computational speedup, where threads help calculations finish sooner. The second goal is handling blocking input and output, where threads let a program keep working while parts of it wait. The two goals ask the same question in different ways: how many threads can actually be doing work at the same time? The answers differ because a blocked thread is not doing work.

How do the target thread counts and the effect of extra threads differ between computational speedup and blocking I/O?

Overlapping Waits Keep the Processor Busy

When several clients each block on network input and output, their waiting periods can overlap in time. While one thread is blocked, another thread can be running, so the processor is not left idle. The timeline below shows three threads at one moment in a server. Thread A and Thread B are waiting on clients, while Thread C has data and is running.

When several clients each block on network I/O, how do their waiting periods overlap so the processor stays busy?

The point of the timeline is that blocking on one connection does not stop the server as a whole. The processor is shared among whichever threads are unblocked at a given moment. Blocked threads simply drop out of that competition until their data arrives.

Estimating a Useful Thread Count

The source's rough target for blocking I/O is about one active thread per processor. If most threads are blocked, that target can mean many more threads in total. The source's example uses a blocked fraction of 90 percent and arrives at a much larger thread count than the processor count, around 40 for a small number of processors. The pack does not reproduce the full arithmetic, so the worked example below is an illustration of the same reasoning rather than the source's own calculation.

Worked example

Thread Count for a Server Blocked 90 Percent of the Time

A server runs on a machine with 4 processors. Each thread is blocked about 90 percent of the time. Roughly how many threads can the server make use of?

1
Find the active fraction
If a thread is blocked 90 percent of the time, it is active 100 percent minus 90 percent, which is 10 percent of the time.
2
Find threads per processor
Keeping one processor fully busy needs about one active thread. If each thread is active only 10 percent of the time, about 10 threads are needed to keep roughly one processor's worth of work going.
3
Scale to the processors
With 4 processors, the same reasoning gives about 10 threads for each processor, so roughly 40 threads in total.
4
Check the limit
The count depends on the measured blocked fraction. If threads were blocked less often, the number would be much smaller.

Result

About 40 threads. This is many more than the 4 processors, and it follows from the blocking behavior rather than from a universal rule.

Mistakes With the Processor Rule

Common mistake
Applying the one-thread-per-processor rule to every program
What it looks like
A network server with mostly blocked threads is limited to as many threads as it has processors.
Why it fails
The rule comes from threads that are computing and want to run at full speed. It ignores what blocked threads are actually doing.
Do this instead
Ask how much of the time the threads spend waiting, then estimate how many threads can be active at once.
Common mistake
Assuming extra threads always speed up a program
What it looks like
Adding threads to a CPU-bound calculation on a single processor and expecting it to finish sooner.
Why it fails
Extra threads add overhead for creating and managing them, and with one processor they cannot make more progress at once.
Do this instead
For computation, match the thread count to the available processors.
Common mistake
Treating blocking as always a bug
What it looks like
Trying to remove every wait from a command line program that reads user input.
Why it fails
Some programs have nothing else to do during a wait, so waiting is acceptable for them.
Do this instead
Decide based on whether other useful work is available during the wait.
Common mistake
A server that gives one thread to all clients can be stalled by a single slow connection.

Practice: Reasoning About Thread Counts

MEDIUM

A server runs on a machine with 2 processors. Its threads are blocked about 75 percent of the time while waiting for network input and output. Estimate roughly how many threads the server can make use of, and explain whether this is more or fewer than the processor count.

Hints

Start by finding the fraction of time a thread is active.
Ask how many threads together make about one processor's worth of active work.
Compare your result with the processor count and say why blocking changes the answer.

Your attempt (not saved)

Summary of Waiting and Thread Counts

1
Network input and output is blocking I/O, because those operations can block for indefinite periods of time, so network programs must be prepared for waiting.
2
Threads used for computational speedup should roughly match the number of processors, while threads used to hide blocking I/O can outnumber processors because blocked threads do not compete for processing time.
3
A server benefits from one thread per client connection, since blocking on one connection does not stop it from serving the others.
4
The useful thread count for a blocking program depends on how much of the time its threads spend waiting, so it must be estimated from measured blocking rather than taken from the processor count alone.

At a glance

Network input and output is blocking I/O because it can wait indefinitely, so network programs must be ready for waiting.
Threads for computation should roughly match processors, while threads for blocking I/O can greatly outnumber processors because blocked threads do not compete for the processor.
A server can give each client connection its own thread so one blocked connection does not stop the others.
The useful thread count depends on the measured fraction of time threads spend blocked, not on the processor count alone.