Semantic Parsing with Language Models

Suppose a language model answers a question in a chat window. A person reading that answer can cope with a lot of imperfection. The wording may be a little awkward, an extra sentence may appear at the start or end, and a line may be missing, yet the reader still understands the point and fills in the gaps. Software is far less forgiving. A program that receives a model's reply has to find specific pieces of information in predictable places, and if those pieces are not where the program expects them, the program cannot continue.

This gap is the reason structured outputs exist. In production systems, a model is often asked to produce its output in a particular format so that the next step of the system can use it. Semantic parsing, the topic of this article, is one of the clearest examples: a model turns ordinary human language into a form that a machine can act on.

Notice what the definition does not require. It does not say the content must be correct, only that the shape must be defined. That distinction runs through the rest of this article. A structured output can be perfectly shaped and still contain wrong values, and a shape can be broken even when the values are right.

Follow the path of a model's reply through a simple application. The reply arrives as raw text. The program then needs to locate the specific pieces it cares about. Each stage can reject or alter what it receives, which is why the format must be decided before the reply is ever used.

The diagram shows the central point of this section. Model text is not directly usable in an application. It first has to be parsed, and the parsed result then has to be checked against the format the application expects. A reply can fail at the check stage even though the model produced something that looked reasonable to a human reader. The application only proceeds when the pieces it needs are present in the form it expects.

The source separates the reasons for structured output into two situations. Telling them apart helps you decide how strictly the format must be enforced. In the first situation, the task itself produces an answer that has a required form. Semantic parsing and classification are examples: the answer is structured because the task is structured. In the second situation, the task's text may be ordinary, but another piece of software has to read it. The format matters because a downstream program depends on it.

The same model and the same task can produce either kind of output. What changes is whether the result is shaped for a program to read directly. A chat answer for a person can stay loose. The same information, when a program consumes it, needs a fixed shape.

In semantic parsing, a user writes ordinary English. The model converts that request into a formal representation in a target language. That representation must be valid for the target language, because a database cannot execute a sentence. The output is therefore a precise, formal statement rather than a description of one. Think of the model as performing a translation from free-form language into named, structured parts.

The same idea appears in a different target. The source describes a prompt given to GPT-4o asking it to create a regular expression that represents all the ways an item can be written. The system prompt asked the model to return only the regex. The regexes produced for a US phone number, an email address, and dates are formal patterns, not descriptions of one particular phone number or date.

Here is an original illustration of the mapping. Imagine a request such as: book a flight to Paris on Friday for one adult. A semantic parser would assign each part of that sentence to a named field, so that the program can read the fields directly rather than interpreting the sentence.

Each named field now holds one piece of meaning from the request. The program can use the destination without parsing the English sentence again. The field names themselves are part of the contract between the model and the software that reads them.

Asking a model politely for the right format is not a guarantee. The source shows a different approach: restrict what the model is allowed to generate. The framework guidance can limit a reply to a choice from a list, or limit an answer to digits. The generation step is confined to allowed forms, so the output is something a program can use without guessing at its shape.

The source names several frameworks that support structured generation: guidance, outlines, instructor, and llama.cpp. It also notes that each model provider may use its own techniques to improve how well a model produces structured outputs. The guidance examples are one concrete way a framework can restrict generation to options or a pattern.

JSON mode is a common way to ask for structured output. It typically guarantees valid JSON, meaning the reply can be parsed as JSON. It does not guarantee that the content of that JSON is correct. The field names may be wrong, a value may be inaccurate, or the content may simply be incomplete. Syntactic validity and semantic correctness are separate properties, and each one needs its own check.

There is a second limit, and it is more surprising. Valid-looking JSON can still be unusable if generation stops too soon. The source describes the case where generation reaches the maximum output token length. The reply is cut off, and the resulting JSON can be truncated and therefore not parsable. The model did not produce a wrong answer; it simply ran out of room before the structure could close.

Trace the example in the diagram. The model begins writing a JSON object and spends its token budget. If the budget runs out before the object closes, the text stops mid-object. The failure does not appear inside the model. It surfaces later, when the downstream parser tries to read the reply and finds that the structure never finished.