AI Pipeline Orchestration

Many developers meeting AI applications for the first time assume that a system which answers a question is one big piece of logic.

Programming

Why One Question Needs Several Parts

Many developers meeting AI applications for the first time assume that a system which answers a question is one big piece of logic. A useful AI application is usually built differently. One part cleans up the question, another fetches supporting material, another asks a model for an answer, and another decides whether the answer can be shown to the user. The way these parts are wired together is the subject of this article.

What do you think happens?

Is an AI question-answering application best understood as one big piece of logic?

Yes, it is one large function that takes a question and returns an answer
No, it is several smaller parts that are wired together

Reveal answer

Answer:

No, it is several smaller parts that are wired together.

Each part does one job, takes an input, and produces an output. The design problem is how those parts connect, and that connection is what this article calls chaining.

The belief that one big function is enough looks sensible, but it leads to design mistakes. Once you accept that the application is a set of parts, the next questions are how data moves between those parts and who makes sure it arrives in a usable form. Those questions lead directly to chaining and to the orchestrator.

Tracing a Query Through the Chain

A pipeline is described as a list of steps that begins with a user query and ends with a finished task. The steps are: process the raw query; retrieve relevant data based on the processed query; combine the original query and the retrieved data into a prompt in the format the model expects; let the model generate a response from that prompt; evaluate the response; and then either return a good response to the user or route the query to a human operator. Each step consumes what the previous step produced. That dependency is what turns a set of tasks into a chain.

Where does the chain begin and end, and what does each step hand to the next one? The evaluation step at the end is a branch: one result returns to the user, the other goes to a human operator.

Worked example

Following one question through every step

A user asks: Can I get a refund? Trace what each step receives and produces.

1
Step 1: process the raw query
Input is the raw question. The output is a processed version of it that the retrieval step can use.
2
Step 2: retrieve relevant data
Input is the processed query. The output is supporting material, such as a passage about the refund policy.
3
Step 3: build the prompt
Input is the original question together with the retrieved material. The output is one prompt in the format the model expects.
4
Step 4: generate a response
Input is the prompt. The output is the model's answer.
5
Step 5: evaluate the response
Input is the answer. The output is a judgment about whether it is good enough to show.
6
Step 6: branch on the result
A good answer goes back to the user. A failed answer is routed to a human operator. The chain ends in one of these two places.

Result

Each step consumed exactly what the step before it produced. The chain began at the user's question and ended when the response was returned or routed to a human operator.

Chaining as Function Composition

Chaining builds a larger operation out of smaller operations, where the result of one operation becomes the input of the next. In mathematical and programming terms this is called function composition. If one component is a function f and the next component is a function g, the chained operation takes an input x, passes it through f, and then passes that result through g, giving $g(f(x))$.

Formula

A component in an AI system plays the role of one of these functions. It takes an input, performs one job, and produces an output. Chaining connects these components so that the whole sequence, from the moment a user asks something to the moment the task is finished, is treated as one pipeline.

python

def process(query):
    return query.strip()

def retrieve(query):
    return ["refund policy: 30 days"]

def build_prompt(query, docs):
    return f"Context: {docs[0]}\nQuestion: {query}"

question = "  Can I get a refund?  "
prompt = build_prompt(question, retrieve(process(question)))
print(prompt)

def process(query):
    return query.strip()

def retrieve(query):
    return ["refund policy: 30 days"]

def build_prompt(query, docs):
    return f"Context: {docs[0]}\nQuestion: {query}"

question = "  Can I get a refund?  "
prompt = build_prompt(question, retrieve(process(question)))
print(prompt)

Output (expected)

Context: refund policy: 30 days
Question: Can I get a refund?

Notes on the code

process strips spaces from the raw question, so its output is the input to retrieve
retrieve returns a list, and build_prompt reads the first item from it
The nested call build_prompt(question, retrieve(process(question))) is a hand-written composition of three functions

The nested call at the end is a hand-built chain. It works for a small script, but every connection is written by hand. The orchestrator's job, described in the next section, is to carry those connections for you as the number of steps grows.

The Orchestrator's Handoff Check

Once the steps are chained, something must carry data from one step to the next and keep track of whether the handoffs are working. That responsibility belongs to the orchestrator. You describe the sequence of steps from receiving a query until the task is complete, and the orchestrator moves each output into the position of the next input. The components can stay focused on their own jobs because the orchestrator handles the connections between them.

The orchestrator has two further duties. First, it should provide tooling that helps confirm that the output of the current step has the form the next step expects. Second, it should ideally notify you when the flow of data breaks, either because a component failed or because the data did not match what the next component needed.

Does the orchestrator pass the output on, or does it stop and react when the format does not match what the next step expects?

The flowchart shows the decision that sits at every connection. The orchestrator does not simply pass whatever it received. It compares the output against the shape the next step needs, and it reacts when the two do not match.

Tracing a Handoff Failure

What do you think happens?

If retrieval returns a list of documents but the prompt builder expects a single string, what should the orchestrator do?

Pass the list on silently and let the prompt builder deal with it
Stop at the handoff, check or convert the data, and report the mismatch if it cannot be fixed

Reveal answer

Answer:

Stop at the handoff, check or convert the data, and report the mismatch if it cannot be fixed.

The guidance is that the orchestrator should help confirm the output fits the next step's expected form, and should notify you when the flow of data breaks because the data did not match.

Worked example

A mismatch between two steps

The retrieval step returns a list of two passages. The prompt builder was written to accept one passage as a single string. Trace what the orchestrator must check.

1
Retrieval produces output
The output is a list: a Python list containing two text passages.
2
Orchestrator checks the handoff
The orchestrator compares the output's form with the form the prompt builder expects, which is one string.
3
Mismatch detected
The shapes differ, so passing the list forward would give the prompt builder data it cannot use as written.
4
Check or conversion applied
If a conversion is defined, such as joining the passages into one string, the output is reshaped before the next step runs.
5
Report if no fix exists
If the data cannot be made to fit, the orchestrator reports that the handoff failed instead of letting the chain continue silently.

Result

The chain either continues with correctly shaped data or stops with a clear report of which handoff broke. It does not continue with data the next step cannot use.

When the Chain Branches

Not every chain runs in a straight line. The evaluation step at the end of the pipeline produces a result that decides what happens next. A good response goes back to the user, while a response that does not pass evaluation is routed to a human operator. The same chain can finish in two different ways depending on a check.

AI Pipeline Orchestrator Versus General Workflow Orchestrator

A general workflow orchestrator, such as Airflow or Metaflow, is a different category from an AI pipeline orchestrator. A general workflow tool can run steps in sequence, and that is the feature a learner often notices first. An AI pipeline has needs beyond running steps in order. Data passed between components is produced by models and retrieval systems, so the checks on each handoff matter. Branching on an evaluation result is common. Because of these needs, the category should be checked before a tool is chosen.

Both can coordinate steps, but an AI pipeline orchestrator is built around the handoff checks and branching that AI chains require.

When orchestrators are evaluated, three aspects are weighed: integration, support for complex pipelines together with ease of use, and performance and scalability. These are the criteria that connect the choice of tool to the needs of the chain you are building.

Running Independent Steps in Parallel

Latency shapes chain design. For an application with strict latency requirements, the guidance is to run as much as possible in parallel. Two steps that do not depend on each other can run at the same time instead of one after the other. An example is a routing component that decides where a query should go alongside a component that removes personal information. Neither needs the other's output, so neither needs to wait.

Mistakes When Choosing and Building Chains

Common mistake
Choosing a tool because it can run steps in sequence
What it looks like
A team selects a general scheduler because it runs tasks one after another, then discovers it has no way to check model output formats or branch on evaluation results.
Why it fails
A general workflow orchestrator and an AI pipeline orchestrator are different categories. Running steps is only part of what an AI chain needs.
Do this instead
Check the category first, then judge the candidate tools on integration, support for complex pipelines, ease of use, performance, and scalability.
Common mistake
Assuming that handoffs never break
What it looks like
A retrieval step returns a list, but the prompt builder expects a string, and the chain keeps running with malformed input.
Why it fails
Each output must fit the format the next step expects. Without a check, mismatched data travels silently down the chain.
Do this instead
Confirm each output's form at every handoff, and make sure the orchestrator reports when the flow of data breaks.
Common mistake
Reaching for an orchestrator on the first day of a project
What it looks like
A developer adds an orchestration tool before the application has any real steps to coordinate.
Why it fails
An external tool adds complexity, and an orchestrator can hide how the system works, which makes it harder to understand and debug.
Do this instead
Build the application first without an orchestrator, and consider adding one in later stages once the application has grown enough to benefit.
Common mistake
Treating a branching chain as a straight line when a check can send the query elsewhere

Practice: Trace the Chain

MEDIUM

A pipeline has four steps: clean the question, retrieve passages, build a prompt, and generate an answer. The retrieval step returns a list of passages, but the prompt builder expects a single string. Describe what the orchestrator should check at that handoff and what it should do if the data cannot be reshaped.

Hints

Ask what each step receives and what it produces.
Look for the point where one step's output must fit the next step's expected format.
Consider whether the orchestrator should pass the data on silently or stop and report.

Your attempt (not saved)

EASY

Name two steps in an AI application that could run in parallel because neither depends on the other's output, and explain why a step that uses another step's output cannot be moved to run at the same time.