Algorithmic Decision-Making Limitations

Once you have built a machine learning system, you still have to decide how it should be used in practice. One of the first design questions is whether a person should take part in its decisions at all, and if so, at which points. This article is about that question. It does not ask whether algorithms are good or bad at deciding. It asks how a system should divide its work between an algorithm and a human, given what the application needs.

Keeping a human in the loop means a person is placed at a point where their judgment can change the outcome. The algorithm does the first pass on every case. The person then sits at the point where the algorithm is least sure of itself. Tracing one uncertain output step by step shows how this works in practice. The output does not go straight to the outcome. Its path depends on whether the algorithm has flagged it as uncertain.

Read the diagram from left to right. The output from the algorithm either passes through automatically or enters a pending state. The pending state is the point where a person's judgment is inserted. The person's decision is then the one that gets recorded. The path that goes straight to acceptance never involves a human at all, which is why the choice of what to flag matters so much to the design.

The phrase in the loop describes where the person sits in the flow of work. Notice that the definition is limited to uncertain decisions. The person is not checking every output. They are checking the subset the algorithm handles least confidently. This keeps the human effort focused on the cases where judgment is most likely to change the result.

The first question to ask about any application is how much time is available between the input and the decision. The answer determines whether a human can sit in the loop at all. Some processes must produce a decision immediately. Other processes do not need an immediate response, and for those it can be possible to have a person confirm the decisions the algorithm is uncertain about.

The contrast is the core of the design choice. A process that must decide now leaves no room for a human step, so the algorithm is the only decision-maker in that moment. A process that can pause gives you a choice. You can insert a person at the uncertain points, because the timeline allows it. Two processes can use the same algorithm and still need different designs, because one has time to wait and the other does not.

The second reason to involve a human concerns precision, not speed. The source notes that medical applications might need very high levels of precision, and that this level possibly cannot be achieved by a machine learning algorithm alone. The point is not that algorithms are weak in general. It is that a particular application can demand a standard of accuracy the algorithm, on its own, may not meet.

When precision is the binding requirement, a human can be used to check the cases where the algorithm is less certain. This is the same logic as the time-based case, applied for a different reason. The human is not there because the process is slow. The human is there because the required accuracy is higher than the algorithm can deliver by itself.

The source makes a practical point that is easy to overlook. An algorithm does not need to handle every case in order to be useful. Many applications are dominated by simple cases. The algorithm makes those decisions automatically, and the relatively few complicated cases are rerouted to a person. Every incoming case goes to the algorithm first. Simple cases leave the system as automatic decisions, and complicated cases are passed to a person.

The reason this design reduces response time and cost is that the simple cases no longer queue up for a person. The automatic path finishes quickly, and the human attention is concentrated on the cases that need judgment. Partial automation is enough to produce a benefit, which is why the source treats routing as a practical design choice rather than an all-or-nothing decision.