Context Length and Efficiency

Every prompt you send to a foundation model has a ceiling on its size. Whatever you want the model to read, including instructions, background documents, examples, and the question itself, has to fit inside that ceiling. Practitioners call this ceiling the context length. Before you can write a good prompt for a long task, you need to know how large the ceiling is for the model you are using, because the ceiling decides how much material you can put in front of the model at once.

Keep one question in mind as you read this article: if you were handed a stack of documents to analyze, how would you decide whether they could all go into a single prompt? By the end, you should be able to answer that question with a token estimate and a comparison against a specific limit.

The source defines context length as the maximum amount of information a single prompt can include. Think of it as a budget that every prompt spends from. The budget is the same size regardless of how short or long your question is, so every piece you add to the prompt takes away from what remains.

A long background document can use up most of the budget before you have written a single instruction. This is why context length is not only a property of the model. It is also a constraint that shapes how you choose and arrange your material. The diagram below shows one illustrative way a small budget could be spent.

In this illustration the background document takes most of the 1K budget, which leaves little room for anything else. The numbers are invented for teaching, but the logic is the one the source describes: material and budget are compared directly, and whatever you add is subtracted from what remains.

Context length is measured in tokens. This matters because word counts and page counts are not the same as token counts, and comparing a word count directly against a context length limit gives the wrong answer. The source gives you a conversion to work with: a book of about 120,000 words is equated to about 160,000 tokens. That pair of figures lets you turn a word count into a token estimate.

The first three generations of GPT models had context lengths of 1K, 2K, and 4K. The source describes 1K as a small budget, barely enough for a college essay and far too small for most legal documents or research papers. Expanding the context length quickly became a competitive goal for model providers and for the people who build applications on top of them.

Within about five years, the limit grew from GPT-2's 1K to Gemini-1.5 Pro's 2M. The source's figure traces this growth between February 2019 and May 2024. The timeline below lays out the endpoints and the intermediate generations so you can see the order of the steps.

The increase is not gradual in the way a slow, steady project might be. The source describes it as rapid, and the factor of roughly 2,000 is what makes that description concrete. The practical meaning is that a prompt budget that once forced you to cut a document into pieces may now hold far more material in a single prompt.

Numbers like 1K or 2M are hard to feel until they are attached to things you already know. The source ties several context lengths to familiar material. The table collects those reference points so you can compare them side by side.

Read this table as a ladder. Each row is a budget, and each reference point tells you what that budget can hold. A 1K budget is a single short essay. A 100K budget is a moderate book. A 2M budget is a whole library-scale collection of text or a large software project.

The same procedure works for any body of material. Estimate its token count, compare that estimate against the limit, and then decide what to do with the result. The flowchart below lays out those decisions.

The source states that everything you want the model to read has to fit inside the ceiling. When it does not fit, the flowchart shows the two possible outcomes of the check. The options for the no branch, shortening, splitting, or excluding material, are reasonable responses to an overflow, though the source does not prescribe a specific procedure for them.