Throughput and Responsiveness in Concurrent Programs

Throughput asks how much work gets finished. Responsiveness asks whether the program still reacts while some of its work is blocked. A concurrent design should be judged against the goal it was built for.

Programming

Why Doing One Thing at a Time Falls Short

Imagine a program that must fetch information from somewhere far away. While it waits for the reply, the user still wants the rest of the screen to respond to clicks and typing. A program that can only do one thing at a time has two bad options: it can stop everything and wait, or it can ignore the user until the reply arrives. Neither option feels good to the person using the program.

Concurrency is one of the main ways programmers avoid that choice. Before reading on, pause and ask yourself: what would a program need to be able to do in order to keep working while part of it is stuck waiting? Keep that question in mind, because the rest of this article answers it step by step.

Tasks That Overlap in Time

Definition

Concurrency is the execution of several tasks simultaneously. The source treats it as a general capability rather than a rare special feature, and it notes that all modern programming languages provide support for it.

The key difference between concurrent and non-concurrent execution is the arrangement of work over time. Without concurrency, one task finishes before the next one starts. With concurrency, the tasks are in progress at the same time, so their time spans overlap. The diagram below shows the two arrangements for three tasks, one arrangement under each heading.

How do tasks that overlap in time differ from tasks that run one after another?

Throughput Versus Responsiveness

The source gives two reasons a program uses concurrency. The first is to improve throughput, which means using several processors for a single computation. The second is to improve responsiveness, which means letting one part of a program progress while another part waits for a response. These are different goals. A program can be designed around one of them without necessarily being designed around the other.

How do the goals of finishing more work per unit of time and staying quick to react to input differ?

Key point

Tracing a Program That Waits

What do you think happens?

A program must wait for a far-away reply but must still respond to clicks. What must it be able to do to keep working?

Reveal answer

Answer:

It must be able to run more than one part at once, so the waiting part can be set aside while another part keeps going.

A program that can only do one thing at a time has no other part available to act while one step waits. Concurrency supplies that ability.

Worked example

A Fetch That Does Not Freeze the Screen

A program must fetch a reply from a distant source. The user keeps clicking and typing while the reply is pending. Trace what happens without concurrency and with concurrency.

1
Without concurrency
The single sequence of steps reaches the fetch and must wait. The whole program has nothing else it is able to do, so clicks and typing are ignored until the reply arrives.
2
With concurrency
The waiting part is set aside while another part keeps going. The screen continues to react to clicks and typing during the wait.
3
When the reply arrives
The waiting part resumes and uses the reply. The user experienced a program that kept responding throughout.

Result

Concurrency lets the program stay responsive, because the waiting part does not block the rest of the program.

Handing Slow Work to a Background Task

One way to get the responsiveness described above is to hand slow work to a background task. The main part of the program then stays free to handle new events. The sequence below follows that hand-off one step at a time.

1
The main part of the program reaches a slow piece of work.
2
That slow work is given to a background task, which runs alongside the main part.
3
The main part continues, so it can still react to clicks, typing, and other new events.
4
When the background task finishes, its result is available to the program.
5
The program uses the result, having kept responding while the slow work ran.

What the C++ Standard Library Provides

C++ is one of the modern languages that supports concurrency. The source describes the standard library's support as a portable and type-safe variant of what has been used in C++ for more than 20 years. Portable means the facilities are meant to be usable across different environments rather than tied to one platform. Type-safe means the facilities are designed so that the compiler checks how values are used, which helps prevent a class of mistakes.

The source also notes that this kind of support is almost universally supported by modern hardware, so the standard library has real hardware to build on. The standard library's support is primarily aimed at systems-level concurrency. It does not directly supply sophisticated higher-level concurrency models. Those models can be provided by libraries built using the standard library's facilities. The standard library is therefore a foundation, and richer models are layered on top of it.

Which concurrency building blocks sit inside the C++ standard library, and which higher-level abstractions are left to libraries built on top?

Mistakes Beginners Make with Concurrency

Common mistake
Treating throughput and responsiveness as the same goal
What it looks like
Assuming that a program which finishes more total work must also stay quick to react to clicks.
Why it fails
The source presents these as two different reasons for using concurrency, and a program can be designed around one without the other.
Do this instead
Decide which goal the design serves, and judge the program against that goal.
Common mistake
Expecting the standard library to supply every high-level model
What it looks like
Looking in the C++ standard library for a ready-made, high-level concurrency model and being surprised when it is not there.
Why it fails
The source says the standard library is primarily aimed at systems-level concurrency and does not directly supply sophisticated higher-level models.
Do this instead
Expect higher-level models to come from libraries built using the standard library's facilities.
Common mistake
Believing concurrency is a rare or optional feature
What it looks like
Assuming that only a few specialized languages include concurrency support.
Why it fails
The source states that all modern programming languages provide support for it.
Do this instead
Treat concurrency as a general capability that programmers across languages are expected to be able to use.

Checking Your Understanding

EASY

A program downloads a large file in the background while the user continues to type in a text editor. Which of the two reasons for concurrency does this scenario illustrate, and why?

Hints

Ask whether the main goal is finishing more work or staying quick to react to input.
Look for the part that waits and the part that keeps going.

Your attempt (not saved)

MEDIUM

Explain in your own words why the standard library's support is described as a foundation rather than a complete set of high-level concurrency models.