Measuring Knowledge Base Size in Tokens

Suppose you have a body of reference material, such as product manuals, policy documents, or research notes, and you want a language model to answer questions about it. You face a practical choice. You could place the whole body of material inside the prompt you send to the model, an approach often called long context. Or you could use retrieval-augmented generation, usually abbreviated RAG, where the system first searches the material for passages relevant to each question and then sends only those passages to the model.

Neither option is automatically better. The choice depends on how large the material is, and the guidance examined here ties that choice to a specific size threshold for Claude models. As you read, keep one question in mind: at what size does the decision change, and why would size be the deciding factor?

Think of the guidance as a single yes or no question that you ask before you design anything. Start with the size of the knowledge base, measured in tokens. If it falls below 200,000 tokens, the recommended path is to include the whole thing in the prompt, with no need for RAG or similar methods. If it does not, the guidance does not give a direct instruction, so the comparison has to be made with more information.

Notice the direction of the comparison. The threshold is an upper limit, so the condition is that the knowledge base is smaller than 200,000 tokens. The open branch in the diagram is open because the passage does not settle it, not because the passage recommends RAG there.

The two approaches differ in what enters the model's context. With prompt inclusion, you place the whole body of material inside the prompt, so the model receives the full knowledge base along with your question. With RAG, the system first searches the material for passages relevant to each question, and only those retrieved passages are sent to the model. The model never receives the rest of the material on that request.

The token count is the figure the guidance uses to draw the line. The source also gives a rough human scale: the 200,000-token figure is described as about 500 pages of material. Treat that page count only as an approximation, since it is an estimate that depends on how dense the pages are.

Boundary cases are where careless reading causes errors. A collection of exactly 200,000 tokens is not smaller than the threshold, so it falls outside the recommendation. The passage does not say what to do at that size, and it should not be read as a signal either way. A reader who rounds 200,000 down to a smaller figure, or who treats the threshold as inclusive, is misreading the condition.

The source notes a gap in the wider field: it would be helpful if other model developers provided similar guidance on RAG versus long context for their own models. When you meet a model without such guidance, you cannot borrow the Claude threshold and treat the decision as settled. Instead, look for guidance from that model's developer, or evaluate the two approaches yourself against the material and the questions you need answered.