Using a Thread Pool

Imagine a server that handles many clients over its lifetime. Each client connects, asks for something, and leaves. Many of these visits are brief, lasting only a moment. The server has to accept every one of them and do the work each connection needs, and it has to do this again and again. How the server organizes its threads for that job has a large effect on how efficiently it runs.

Before reading on, try to answer this for yourself: if every incoming connection gets its own brand-new thread, what work does the server repeat over and over, even when the connection itself needs very little time?

A simple design gives every incoming connection its own new thread. That design works, but the source points out a cost: it is not very efficient to create a new thread for every connection, especially when the connections are typically very short-lived. If each connection lasts only a moment, the server spends a meaningful share of its effort starting and stopping threads rather than doing the work the connections need.

The left column shows the cost the source describes: each connection pays for creating a thread and then for that thread finishing. The right column shows the alternative. The threads are created once and then stay available, so the cost of making threads is paid for the pool instead of being repeated for every connection.

The key word in that definition is reuse. The pool does not grow with the number of connections. Its threads are created once and stay available. Each thread in the pool runs in an infinite loop, and each time through that loop it handles one connection. When one connection finishes, the same thread goes back around the loop and is ready for the next one.

This is where the efficiency comes from. The number of threads stays the same no matter how many connections arrive over time, and each thread is used for many connections in turn.

A pool of waiting threads is only useful if the main program can hand them work. The chosen tool is a blocking queue, named connectionQueue in the course's example. The main program places each new connection into this queue, and the connection-handling threads take connections out of it. The main program does not need to know which pool thread will handle a given connection; it only needs to put the connection into the queue.

Read the diagram from top to bottom. The main program is the only producer, placing connections into connectionQueue. Each connection leaves the queue once and is taken by exactly one pool thread. The pool threads are the consumers, and each of them takes the next connection when it is free to do so.

The word blocking matters. A blocking queue is defined as a queue that blocks a thread when it is empty and wakes it up when an item becomes available. So a connection-handling thread that finds connectionQueue empty does not spin or keep checking in a tight loop. It waits, and it resumes when a connection is placed in the queue.

The diagram traces one full cycle. A thread runs while it handles a connection, loops back when that connection finishes, and blocks if the queue has nothing for it. The only event that moves it out of the blocked state is a connection arriving in the queue. The thread does not need to be recreated for that new connection; it is the same thread, now woken up.

The ideas in a thread pool are simple to state but easy to misread. The following mistakes come up often when learners first explain how the pool works.