Guides

Context window optimisation: a working guide

Choosing what a model reads and checking whether it helps.

An agent can find the right document and still miss the answer. Extraction may drop a table, selection may leave out a caveat, or a long conversation may obscure the evidence it needs. The same pipeline can also spend most of its tokens on repeated instructions, tool results and page markup.

Context optimisation means following that material from source to answer: what enters the window, what survives each transformation, what the model can use and what it costs. This guide works through those decisions, from counting tokens and acquiring sources to memory and evaluation. It assumes familiarity with prompts and the outline of retrieval-augmented generation, but no prior reading of the research.

Prices, limits and tooling carry the dates they were checked. Specialist terms are explained as they arise or in the glossary; the companion evidence review gives a more detailed assessment of the supporting evidence.

Contents

1. What is actually in the window
2. How more context changes an answer
3. What the benchmarks do and do not tell you
4. Acquisition: getting the right material
5. Extraction and chunking
6. Ranking and reranking
7. Selection and ordering under a budget
8. Compression
9. Caching, and what it costs to get the order wrong
10. Agent memory, compaction and isolation
11. Proving a change helped
12. Where local measurement still matters
13. Glossary
14. Appendix: one production system, end to end
15. Appendix: tooling
16. About this guide
17. Sources

1. What is actually in the window

Before deciding which documents to retrieve, work out how much room they will have. A context window is the bounded working material a model can hold at once, usually measured in tokens. Text, encoded images and other supported inputs can all count towards it.

The window includes the system prompt, conversation messages, tool results, attached documents and the definitions of tools the model can call. The model’s output counts too, including reasoning before its answer.¹ Input and output share an overall limit, although the output may have a lower limit of its own. A million-token window therefore does not leave a million tokens for documents: everything else needs space as well.

Why there is a limit

Attention lets a model relate one part of its input to another. In naive dense attention, the comparison work grows quadratically: doubling the input means roughly four times as much work.² That helps explain why long inputs are costly to process.

Serving systems use optimisations beyond that naive calculation, and vendors do not publish all the details of how they serve million-token windows. The quadratic relationship explains the underlying problem; it is not a formula for predicting a particular API’s price or latency.

Count tokens with the model you will use

A token is a short word, part of a longer word, punctuation or another piece of text. The familiar estimate of four English characters per token is useful for a rough sketch, but too loose for a budget.

A tokeniser determines those pieces. Different models can count the same sentence differently, even across generations from one provider. Anthropic’s documentation in August 2026 described a newer tokeniser for Claude 4.7 and later producing about 30% more tokens for identical text. With unchanged per-token prices, that also raises the cost of sending the document.³

OpenAI’s tiktoken does not give an exact count for Claude or Gemini. Anthropic’s guidance puts its undercount at 15–20% on ordinary Claude text, and higher on code.⁴ Neither Anthropic nor Google published a downloadable tokeniser in August 2026. Use the relevant provider’s free counting endpoint when the count needs to be accurate.³

How much of the window does real content take?

Two files of the same character length can occupy quite different amounts of context. This matters when an agent reads a mixture of prose, source code, logs and generated files. The table below measures nine files from this project with one named tokeniser, so it provides a starting point rather than a universal conversion factor.

These measurements used tiktoken o200k_base on 4 August 2026, through docs/guide/measurements/token_density.py. The two prose documents, the JSON record and the event log measured then are not published; the script now reads published files in their place, with JSON test vectors standing in for the event log and no JSON API record, so a rerun gives comparable rather than identical figures. They do not give exact Claude or Gemini counts; measure those routes separately.

Code in this sample used about 20–40% more tokens per character than prose. The lockfile was denser still. That is useful when deciding whether to send a whole generated file or select the relevant entries. Logs, repeated JSON records and tool schemas deserve the same attention: their visible length can understate their share of the token budget. Nine files and one tokeniser are too small a sample to establish a rule for every corpus.

A large avoidable cost: raw HTML

A fetched web page contains both readable content and the machinery used to display it. Sending the raw response to a model can spend most of the budget on that machinery before the model reaches the article.

Four pages were measured as retrieved and after a crude regular expression removed tags and scripts:

Removing HTML machinery saved 87–99% of tokens on these pages, with a median of 92%. The remaining text still included navigation, footers and related articles. A content extractor tries to remove those too; chapter 5 explains how to check that it also preserves the material you need.

Page type matters. The BBC entry is an index rather than an article, and it accounts for about half the raw tokens in this small sample. A token-weighted saving of 95% mostly reflects that page; excluding it gives 92%. Neither figure establishes how much of a typical page is boilerplate, because the script did not identify boilerplate and only four pages were measured. The original HTML inputs were not retained, so these exact figures cannot be reproduced. They illustrate why extraction is worth measuring on your own sources.

What is in there before you add anything

Tools occupy context even before they return a result. Their definitions tell the model what each tool does and how to call it. A large catalogue can therefore consume tens of thousands of tokens before a conversation begins: Anthropic reported about 55,000 tokens for one five-connector setup.⁵

Much of that text is JSON Schema, the description of the arguments a tool accepts. In an independent measurement of one server, 97% of its tokens were in the input schema. Redesigning that schema reduced its footprint from 17,161 tokens to 773.⁶ The size of a catalogue depends as much on the shape of its definitions as on the number of tools.

In Anthropic’s request layout, tool definitions precede the system prompt and messages. As a session continues, messages and tool results can outgrow that fixed overhead. One published session breakdown attributed about 28,000 tokens to fixed overhead and 185,000 to conversation and tool results.⁷ The initial inventory and the growing history need separate attention.

Available, loaded, invoked

A useful inventory distinguishes three states:

• Available: the tool exists and can be discovered. A short discovery entry may be present, while its full definition remains outside the window.
• Loaded: its definition is in the window and occupies space on each request carrying it, whether or not the tool is called.
• Invoked: the agent calls it, adding arguments and a result to the conversation as well.

Deferred loading makes this distinction practical. Anthropic’s implementation sends definitions to the service but keeps unused ones out of the model’s window. In its reported example, the loaded tool set fell from 77,000 tokens to 8,700.⁵ Skills follow the same pattern: a name and short description cost about 100 tokens while available, and full instructions load when the skill is triggered.⁸ An independent count of the official skills found a similar discovery footprint.⁹

This is why counting calls alone misses a substantial cost. Two hundred rarely used tools can still fill the window if all their definitions load on every request.

What you cannot see

Hosted APIs expose usage totals, but those totals do not explain the whole request. You may receive input, output and cache counts, and on Anthropic a reasoning-token count, without a breakdown separating system instructions, tools, history and retrieved evidence.

Keep a local inventory of those categories, while allowing for limits in the provider’s reporting:

• Counting and billing can differ. Anthropic says its count may include automatically added system tokens for which the customer is not billed.³
• A reasoning-token count does not necessarily expose the reasoning content.
• A cache miss usually appears as no cache reuse, without an error explaining which part of the prefix changed.
• Usage counters do not identify which evidence influenced the answer.

The last limitation matters when a request fits comfortably within the window but the model still misses something you supplied. Understanding that failure requires looking at what models can do with long inputs.

2. How more context changes an answer

Adding context gives a model more evidence, but also more material to sort through. A document can contain the answer and still fail to help: the model may overlook it, confuse it with a similar passage, or carry forward an assumption from an earlier turn. These failures depend on the model, task and kind of input. The advertised window size alone cannot predict them.

Where the answer sits

Imagine giving a model twenty documents, only one of which contains the answer. Moving that document from the beginning to the middle changes no facts, so differences in the answer reveal sensitivity to position.

That was the design of Lost in the Middle. In its GPT-3.5-Turbo experiment, accuracy was about 75.5% with the relevant document first, about 53% around the middle, and about 63% near the end: a U-shaped curve. In some settings, placing the answer in the middle produced a lower score than giving the model no documents at all.¹⁰ A larger-window variant also did not necessarily use the supplied material better than its smaller-window counterpart.

Those were early models on inputs of roughly 2,000–6,000 tokens. The findings explain why ordering became a concern, but the size of the effect cannot be carried over to current models. A later controlled replication found nearly flat performance across placements for ordinary text retrieval. Its results were especially sensitive to which questions were sampled.¹¹ Another audit found end-to-middle drops on reasoning tasks, with smaller drops in newer releases.¹²

As of August 2026, those results had not been reconciled. Position deserves attention for reasoning, tool output and multimodal input; for a modest set of retrieved text passages, its effect may be small. Chapter 7 turns that distinction into an ordering strategy you can test.

Context rot: when length adds difficulty

A model can also become less reliable as an input grows even when the useful information stays in place. This is often called context rot.

One way to see the problem is to remove almost all reasoning from the task. Chroma asked models to reproduce lists of repeated words. Accuracy fell from near-perfect on short lists to roughly 40–60% on lists of 10,000 words.¹³ The operation remained simple; keeping track of a long input did not.

The same evaluation found that retrieval deteriorated faster when the question and answer shared fewer words. Plausible distractors made it worse, and their effects differed: adding one misleading passage could matter more than adding several obvious irrelevancies. A focused prompt containing the needed memory also outperformed that memory embedded in a much longer input.

Surrounding text had an effect of its own. Shuffling the filler to destroy its coherence improved retrieval across the models tested. That suggests a useful caution when assembling context: readable surrounding prose can still compete with the evidence the question needs.

A separate study held a two-fact reasoning problem constant and padded it with irrelevant text. Accuracy declined well below the models’ window limits.¹⁴ These experiments isolate aspects of length; they do not show that every longer document will produce a worse answer.

Effective length versus advertised length

Effective context length means the length at which the model can still do a particular job reliably. The advertised length is how much input it accepts. A useful test needs to establish the first for your task.

RULER extends simple retrieval tests with harder tasks and found that many models did not sustain their short-input performance across their advertised windows.¹⁵ NoLiMa removes the shared wording between question and answer, so the model has to make an association rather than match a phrase. Most of the long-window models it tested had fallen below half their short-context baseline by 32K tokens.¹⁶

One proposed explanation concerns training: even a long training window may contain relatively few examples that require connecting distant positions. Experiments remapping position offsets recovered some of the lost performance.¹⁷ That is a possible mechanism, not a way to calculate a universal usable fraction of a window. No traceable source was found for the often-repeated estimate of 60–70% of advertised length.

Improvements depend on the model and task

Effective length has also improved substantially. Epoch AI tracked the input length at which leading models maintained 80% accuracy, and found rapid gains over its observation period.¹⁸ Older failures remain useful diagnostic examples without defining a permanent limit.

A larger comparison of long-context retrieval found some models declining after 32K or 64K tokens while others improved roughly monotonically to 100K.¹⁹ The failures differed too: some models increasingly refused for copyright reasons, while others failed to follow instructions. These need different remedies. Before shortening a prompt, inspect whether the model missed evidence, refused to use it or answered the wrong question.

Too many tools

Choosing a tool is another retrieval problem. As a catalogue grows, the model must distinguish more names, descriptions and argument schemas. Similar tools can be hard to tell apart even when the correct one is present.

LongFuncEval varied tool-definition length, tool-result length and conversation length separately. All three affected performance, with large differences between models.²⁰ A production-catalogue study found that shortlisting tools recovered part of the loss as the catalogue grew.²¹ But even a shortlist guaranteed to include the correct tool left a gap of about ten points. Finding the right candidate did not ensure the model would choose it. Clear names and distinct descriptions therefore deserve attention alongside the index.

The point at which this became troublesome was specific to that catalogue. It does not establish a maximum tool count for every agent. Likewise, a preprint testing many simultaneous instructions found a limit to perfect compliance, with refusal becoming the failure near capacity.²² Adding instructions can make a request harder to satisfy even when there is ample room for their tokens.

Conversations decay

A conversation can spread one complete task across many turns. The model starts acting before it has all the information, and later messages must correct or extend decisions already made.

A study that split fully specified instructions across turns found a large average performance drop across the models tested.²³ Its decomposition showed both reduced capability and a larger increase in unreliability. Putting the same pieces back into a single turn largely restored performance. The information had not changed; its conversational arrangement had.

For a long-running agent, this suggests keeping the current task and decisions explicit. A growing transcript is not necessarily a clear account of what the agent should do next. Chapter 10 looks at ways to maintain that account.

Pruning can improve accuracy

Old tool results often accumulate after their immediate purpose has passed. Removing some of them can leave the model with a clearer view of the current work, provided the necessary facts survive.

Microsoft tested this in a 50-task expense-agent benchmark.²⁴ Keeping only recent tool call/response pairs improved accuracy; adding windowed summarisation improved it further:

The combined approach used about 63% fewer tokens and improved accuracy by 20.6 percentage points. These are the authors’ results on their own agent, with further checks across task categories and another model, rather than an independent replication. Input accounted for almost all token use in that workload, which explains why pruning history mattered more than shortening answers.

Tool history also has its own position effects. LongFuncEval found newer tool results easier to use than older ones.²⁰ That differs from the original text-retrieval U-curve, another reason to evaluate the actual input type.

A vocabulary for diagnosing failures

Drew Breunig’s terms poisoning, distraction, confusion and clash offer a way to describe what went wrong.²⁵ An earlier error can be carried forward; irrelevant material can draw attention away; similar tools can be confused; and conflicting instructions can pull the model in different directions. These descriptions help organise a failure log before choosing a remedy.

The terms come from practitioner writing, and their empirical backing differs. The conversation, tool-selection and long-context studies above examine several of these mechanisms. For propagation of an injected false fact, described here as poisoning, the sources checked in August 2026 offered an anecdote rather than a controlled measurement. Keep that distinction when using the taxonomy.

What adding noise teaches us

Extra documents sometimes appear to help for reasons unrelated to their content. An early study reported improved retrieval-augmented answers after adding random documents.²⁶ A replication reproduced the result under the original configuration, then found it disappeared with normal prompting. Short generation limits and formatting choices had disproportionately harmed the small-context baseline.²⁷

For practice, the useful finding concerns plausible mistakes in the retrieved set. Passages that resemble the question but supply the wrong answer can be particularly harmful, and they occur in dense-retrieval results.²⁸ This connects context management to evaluation: a change should be tested through the final answer, with the rest of the setup held constant.

3. Benchmarks

A long-context benchmark might ask a model to find a sentence, reconstruct a file or reason across several documents. Those tasks exercise different abilities. Before using a score to choose a model or pipeline, identify which ability your application needs.

Match the benchmark to the work

HELMET compared long-context evaluations across many models and found that synthetic-task scores did not reliably predict downstream performance. Even its own task categories correlated poorly with one another.²⁹

A model that leads on retrieval may therefore be a different choice from one that leads on synthesis or state tracking. Use benchmark results to narrow a shortlist, then test the tasks your system will actually perform.

Needle-in-a-haystack is a floor test

A needle-in-a-haystack test hides one sentence in filler and asks the model to find it.³⁰ It checks a necessary ability: access to a fact buried in a long input. By August 2026, frontier models’ near-perfect scores made the simple version a weak way to distinguish them.

The task changes when a question no longer shares wording with the hidden answer. NoLiMa tests that association and finds much earlier declines.¹⁶ Reasoning across retrieved facts is harder again: GPT-4.1’s launch material reported perfect needle retrieval to a million tokens, alongside 19% accuracy on multi-hop graph traversal above 128K.³¹

For an application that must use a document to reach a conclusion, test that use directly. Finding one sentence is only the first step.

What each benchmark is actually for

The following is a snapshot of the benchmark landscape in August 2026. The question in the middle column is more useful than an overall rank.

LoCoDiff is a useful example of a task that needs more than fact retrieval. It gives a model a git history and asks for the exact final file.³² There is no filler: changes throughout the history may affect what must be reproduced. In the reported evaluation, all models fell below 50% above 25,000 tokens, despite much higher scores on long needle tests.

τ-bench adds a different requirement: consistency. Its pass^k metric counts tasks completed successfully on all k independent attempts.³³ A high single-attempt score can conceal repeated failures that matter to users who rely on the same workflow every day.

Test the reader as well as retrieval

Suppose the retriever finds every required passage. A long prompt can still make the answer worse. One study isolated this by replacing irrelevant text with whitespace, restricting attention and placing the evidence immediately before the question. Performance still declined across the models tested.³⁴

These controls help separate length from distractor content and position. They also explain why retrieval recall alone cannot validate a context pipeline: you need to test the model that reads the assembled prompt.

Allow for benchmark contamination

A public benchmark may overlap with a model’s training material. The model can then benefit from remembering benchmark-specific content, which is a different ability from solving a new task. SWE-bench experiments found higher rates of file identification and function reproduction for benchmark material than for comparison material outside it.³⁵

Detecting such overlap is itself unreliable. An evaluation of contamination audits found that the tested methods often gave incorrect results, particularly under distribution shift.³⁶ Passing an audit cannot establish that a benchmark was absent from training.

Held-out material with known provenance and refreshed tasks can reduce this problem. EvoBrowseComp, for example, generates live-web questions from newer material.³⁷ For your own evaluation, retain a private set alongside public benchmarks and record when its source material became available.

Record dataset versions and omitted tasks

Vendor launch evaluations are useful sources, but inspect what they test and which version they use. OpenAI’s MRCR and Graphwalks datasets were corrected after their initial publication, making scores from different versions hard to compare.³¹ Public datasets also leave outsiders without the vendor’s private held-out material.

An advertised capability may have no corresponding launch score. Google’s Gemini 3 launch material highlighted a million-token window while reporting other kinds of benchmarks rather than a long-context benchmark.³⁸ The window size establishes capacity; a suitable evaluation still needs to establish performance.

Compare the cost of a score

Two systems with similar accuracy can have very different operating costs. Terminal-Bench was unusual among these leaderboards in publishing both. Its 4 August 2026 snapshot illustrates why the cost belongs beside the score:³⁹

The reported accuracy intervals overlap, while the costs differ by about 3.7 times. For deployment, that can matter more than the small difference in mean accuracy. Compare cost per correct answer and latency as well as quality.

Fill the gaps with your own evaluation

These benchmarks leave several concerns only partly measured or unmeasured. If these matter to the application, include them in your own suite:

• Source selection: whether the agent finds an authoritative, primary source, rather than merely a page containing the expected answer.
• Attribution: whether a claim is supported by the cited passage. HELMET includes citation tasks, but their scores transfer poorly to other task categories.
• Conflicting evidence: whether the model notices and handles sources that disagree. RECON begins to test this, with low reported performance outside its oracle condition.⁴⁰
• Freshness: whether the system recognises that evidence has become stale.
• Abstention: whether it declines appropriately when evidence is missing, rather than receiving credit only for attempting an answer.
• Untrusted content: whether retrieved text can divert the agent from its task.
• Operating cost and delay: what each correct answer costs and how long the user waits, including retries and failures.

This gives acquisition a clearer objective. Finding a page is useful only if it supplies evidence the system can safely and accurately use.

4. Acquisition

Acquisition decides what material the rest of the pipeline can work with. If the necessary source never arrives, ranking and compression cannot recover it. Start with the shape of the task: a question about one document, a search across a known collection and an investigation on the live web need different ways of gathering evidence.

Choosing between whole documents and retrieval

For a question that compares sections or needs several reasoning steps, sending the whole document preserves connections that chunk retrieval might miss. For a narrow factual question, a few well-chosen passages may supply all the evidence at much lower cost.

A comparison across nine datasets found that long context generally produced better answers when given enough budget. Yet about 63% of queries produced identical answers with long context and retrieval.⁴¹ That agreement makes routing useful: reserve the expensive path for questions that need it.

The same paper’s Self-Route method asks the model whether the retrieved chunks suffice, then escalates to full context when they do not. It retained near-long-context quality while reducing cost in the tested setups.⁴¹ This is a concrete alternative to choosing one strategy for every query.

The point where retrieval becomes preferable depends on model and task. LaRA found advantages for long context on reasoning and comparison, while retrieval helped with hallucination detection and helped smaller models at long input lengths.⁴² Its crossover near 128K tokens is a result of those experiments, not a general routing threshold. Test whole-document and retrieval baselines on the same questions, especially where the task needs evidence from distant sections.

Diagnose the query before changing the index

A retriever needs a query it can match to useful material. Manual error analysis in the long-context comparison identified four ways that can fail:⁴¹

• A multi-step question needs a chain of lookups that one retrieval call does not perform.
• A vague question gives too little detail to match reliably.
• A long, complex question mixes requirements the retriever fails to separate.
• An implicit question asks for something whose wording or meaning is not directly present in a passage.

These suggest different experiments. Query rewriting can clarify a vague request; decomposition can turn a multi-step task into smaller searches. Improving the representation may help when the connection to the answer is implicit. The taxonomy comes from one paper, but it provides a useful way to choose a change from observed failures instead of adding every retrieval technique at once.

How many sources is enough

A retriever can keep finding relevant documents after the reader has stopped benefiting from them. In the original Lost in the Middle experiments, increasing the retrieved set from 20 to 50 documents improved reader accuracy only slightly while recall continued to rise.¹⁰

The extra passages also create opportunities for confusion. In one evaluation, over 60% of queries had at least one convincing but wrong passage among a dense retriever’s top ten results.²⁸ A close match can require more careful discrimination than an obviously irrelevant document.

Anthropic found twenty passages worked best in its contextual-retrieval setup; practitioner starting points after reranking often use five to ten.⁴³ Those are settings to test, not a universal answer. Tune the admitted count, usually called k, against answer quality and cost in your complete pipeline.

Choose techniques by the failure they address

The following techniques change different parts of the search. Their value depends on what happens after they return candidates.

What hybrid search combines

A literal error message and a loosely worded description of a fault call for different kinds of matching. Hybrid search combines them so the first stage can find both.

Full-text search, including BM25,⁴⁶ retains exact terms that embeddings can blur: error strings, flag names and host names. A literal match can be especially useful when someone pastes the error they received. Embeddings help with paraphrase: someone asking “restore hangs after manifest load” may need an answer describing a “checkpoint stalled on the NFS mount”, despite having little wording in common.

Term rarity, or inverse document frequency,⁴⁷ helps distinguish specific words from conversational filler. “Sounds good, thanks!” provides little search signal even if its embedding lies near many queries. Recency can be another signal when older answers describe infrastructure that has changed. A date should inform relevance alongside the content, rather than act as an unexamined cut-off.

The result lists need a shared ranking. Reciprocal rank fusion assigns each document weight / (k + rank) in each list where it appears, then adds those contributions. The original method used k = 60.⁴⁸ A document with moderate ranks in several lists can then beat one found near the top of just one list. Working with ranks also avoids having to make BM25 scores and embedding similarities share a numerical scale.

Recency weighting is a practitioner design choice for handling stale answers; the cited material does not quantify its benefit. The same care applies to the more elaborate approaches in the table.

Multi-query fusion, for example, found more relevant candidates in an enterprise deployment but reduced Hit@10, the rate of finding a relevant result in the final ten.⁴⁹ The reranker and token budget constrained what could reach the reader. Broader retrieval added work without improving that final set. Include the downstream reader when evaluating any technique that widens recall.

Iterative retrieval lets a model refine a search after seeing the first results. In a direct comparison, a well-configured one-shot search slightly outperformed iteration, while iteration made substantially more retrieval calls. The iterative approach’s advantage was token efficiency in that setup.⁵⁰ Search-R1’s larger reported gains came from training a search policy with reinforcement learning.⁵¹ Prompting an existing model to use a search tool repeatedly does not reproduce that training intervention.

GraphRAG’s reported benefits need another distinction. Its evaluations favoured answers for comprehensiveness, diversity and usefulness on open-ended tasks; faithfulness was similar to baseline retrieval.⁵² Those preferences may matter to a product, but they do not establish improved factual accuracy. Indexing cost also matters: Microsoft’s LazyGraphRAG work reports a large reduction against full GraphRAG.⁵³ As of August 2026, no independent ground-truth comparison was found establishing that full GraphRAG beats a well-tuned hybrid pipeline.

How much do two search providers agree?

A second search provider may find sources the first misses. To explore that, the authors sent 24 queries across six types—factual, fresh, technical, commercial, vague and analytical—to three providers on 4 August 2026. Each request asked for ten search results. Four of the 72 calls failed and were excluded, rather than treated as empty result sets.

The comparison uses hosts rather than exact URLs. That is a coarse view of source diversity: two hosts can repeat one source, while two pages on one host can be independent.

A concentration of 1 would mean every result came from one host. The lower values here indicate that each provider spread results across hosts. To judge what a second provider adds, look at overlap between the pairs:

Any one provider returned about eight or nine distinct hosts per query; the combined set averaged 15.5. Exa contributed a larger proportion of unique hosts in this sample, while Tavily and Firecrawl overlapped more. This suggests measuring provider combinations, rather than assuming any second provider will add the same coverage.

Host variety also says little about source quality. Among the most frequent hosts were Reddit, YouTube, arXiv, Medium and the vLLM documentation site. A retrieval pipeline still needs to decide which kinds of evidence are appropriate for the task.

These results describe one English-language run over a small query set, not a stable ranking of providers. The script is docs/guide/measurements/source_overlap.py; rerun it on your own queries and inspect what the additional sources contribute to answers.

Live web acquisition

Search results help choose what to read. Fetching every candidate can expand the input dramatically: the provider examples used here put an average page around 2,500 tokens, a large documentation page around 25,000 and a research PDF around 125,000.⁵⁴ Ten full PDFs can exceed a million-token window. Search first, then fetch the pages whose contents are needed.

Billing reinforces that distinction. At the 4 August 2026 check, Anthropic charged per search query and billed fetched content as input tokens; other providers used combinations of query charges, page charges, credits and output-token minimums.⁵⁵ The cheapest search call is not necessarily the cheapest route to an answer if it leads to much more reading.

Fetch tools can also impose constraints on acquisition. The platform tools described here respect robots.txt and require a URL already present in context, which makes search-then-fetch part of the workflow. Publisher access can carry a separate price: Cloudflare’s pay-per-crawl uses HTTP 402 and cryptographic crawler signatures to let publishers allow, charge or block access.⁵⁶ Acquisition therefore needs to account for access as well as search relevance.

5. Extraction and chunking

Once a document arrives, two steps determine what can be retrieved from it. Extraction separates useful content from the page around it. Chunking divides that content into pieces small enough to match and select. A failure in either step can remove evidence before the answer model ever sees it.

Extract the body and check what survives

Chapter 1’s four-page measurement showed how much raw HTML can cost. An extractor goes further than stripping tags: it tries to identify the body and leave out navigation, adverts and footers. That saves space, but an extractor can also omit a table, paragraph or whole page that the task needs.

Zyte’s article-extraction benchmark compares extracted content with an article-body reference using overlapping four-word sequences.⁵⁷ Precision indicates how much returned material matches the body; recall indicates how much of the body survives. Both matter when the output will become model context.

The converters near the bottom keep most body text but also return much of the surrounding page. That can be desirable for conversion, while being a poor fit for compact article context. The scores compare word sequences, so they cannot directly establish a token-cost multiplier.

This benchmark covered 181 pages and was archived in June 2026. Treat it as a comparison on those pages, then inspect extraction on your own sources. In particular, distinguish Readability implementations and evaluation runs: the JavaScript entry above has high recall; lower figures elsewhere cannot be applied to it indiscriminately. MarkItDown is a converter whose HTML path wraps markdownify, rather than a body extractor. None of these tools renders JavaScript, leaving another source of missing content.

Check page survival as well as scores on pages that survive. A separate study found that extractors with similar benchmark scores retained different sets of pages. Combining their outputs increased token yield without a benchmark regression.⁵⁸ A clean-looking result is not enough if the extraction stage silently discards documents.

Start with simple chunks

Small chunks make it easier to match a specific passage. Larger chunks retain more of its surrounding explanation. Overlap preserves text around boundaries but also returns repetition. These are the trade-offs a chunking strategy needs to manage.

In Chroma’s evaluation, a simple recursive splitter was competitive with more elaborate semantic approaches.⁵⁹ The LLM-guided splitter found slightly more relevant text, but returned substantially more irrelevant text with it:

For a starting configuration, a recursive splitter at 200–400 tokens with no overlap is worth testing. Change it when failure inspection shows that boundaries or missing surroundings are losing answers. This is one company’s evaluation on five corpora, not proof that one size suits every document.

Parent-document retrieval separates matching from reading: search small chunks, then give the model the larger section each winning chunk came from. That preserves precise matches while restoring some context. No published measurement establishing its benefit was found; it is a design pattern to evaluate through the reader.

Restore the context a chunk lost

“The company” and “this quarter” mean little once their paragraph is detached from the document. Contextual retrieval addresses this by having a model write a short situating header for each chunk at index time. The header, typically 50–100 tokens, accompanies the text used for search.

Anthropic measured retrieval failures as it added contextual embeddings, contextual BM25 and reranking.⁴³ This is the vendor’s evaluation of its own technique, with no independent reproduction found as of August 2026:

The progression matters because it shows where each step helped. Contextual headers reduced missing evidence; reranking removed further failures after both search routes were present. Anthropic reported an indexing cost of about $1.02 per million document tokens, made possible by caching the shared document prefix during header generation.

The same header can make sibling chunks harder to distinguish. A chunking study found improvements for search across a corpus but degradation when searching within one document.⁶⁰ Document-level context helps tell documents apart; repeated across one document’s chunks, it can blur their differences. Choose the technique with the search scope in mind.

Other ways to prepare an index

Some systems take contextual preparation further by rewriting each source into a common structure. A support thread might become a record containing the question someone would search for, a summary, the resolution and the systems involved. The index embeds that record. The aim is to make a code comment, incident record and wiki passage more comparable when they address the same problem.

Another approach filters material before embedding it, using signals such as rare terms, minimum length, reactions or links. This can keep conversational filler out of the candidate pool and runs once per source item rather than on every query. It also makes an early selection decision that needs checking for lost evidence. The production example in the appendix uses both patterns; neither comes with published quality measurements in the sources cited here.⁶¹

Late chunking preserves surrounding context in a different way. It embeds the whole document at token level, then pools those representations into chunks. Each chunk’s representation has therefore been influenced by its neighbours, without a separate header-generation call for every chunk.⁶²

Jina’s evaluation of its own technique found gains that varied by dataset and grew with document length. A head-to-head comparison put contextual retrieval slightly ahead but at much higher cost.⁶³ Both approaches are worth understanding as ways to retain context during indexing; their benefit still depends on the documents and questions being matched.

6. Ranking and reranking

Retrieval and ranking are different jobs. Retrieval asks “which hundred of these million documents might be relevant?” Ranking asks “which five of these hundred actually are?” The first must search cheaply over a large collection. The second can spend more work on each candidate because there are far fewer of them.

Why a second pass helps

First-stage retrieval compares pre-computed representations. A passage’s embedding is created before the question is known, so it cannot be shaped by that question.

A cross-encoder reads the question and one candidate passage together, then scores the pair. That allows more detailed matching. It is too expensive to apply to every document in a large corpus, but affordable for a shortlist of fifty candidates.

This division of work explains the common pattern: retrieve many candidates, rerank them, then admit a smaller set to the answer model.

What the second pass can recover

The contextual-retrieval experiment in chapter 5 shows a reranker adding value after two search routes were already in place. In Anthropic’s own evaluation, reranking reduced top-20 retrieval failure from 2.9% to 1.9%, removing about a third of the failures left by the earlier stages.⁴³

That is a useful measure for a reranker: how often does its final shortlist contain evidence the reader needs? A ranking score alone does not tell you whether a missed answer has been recovered.

Choose model size against the gain

A larger reranker can cost substantially more without improving the shortlist much. The published MS MARCO cross-encoder family illustrates diminishing returns.⁶⁴ Here, nDCG@10 rewards placing relevant documents near the top, while throughput indicates the processing cost:

Moving to six layers improved the score but cut throughput by a factor of five. Moving to twelve layers nearly halved throughput again for little score change. If your queries resemble this evaluation, the six-layer model is a reasonable starting point; measure elsewhere before transferring the result.

Newer rerankers offer capabilities beyond size, including a 32,000-token context for whole documents and instructions defining relevance for a query. A vendor evaluation found instruction following contributed substantially to the gain.⁶⁵ An instruction can express which kind of evidence the task requires, something a generic similarity score may not capture.

Dedicated rerankers and general models

A general-purpose model can be prompted to order passages, but it brings the cost of generation to a task that a cross-encoder performs in a batched scoring pass. Listwise LLM rerankers may also need several sliding-window passes to process all candidates.

One vendor’s comparison found its cross-encoder ahead of frontier models in quality, latency and cost.⁶⁶ The direction had independent support in the cited material, while the vendor-configured magnitudes should be treated cautiously. The same comparison found LLM reranking more useful when the first-stage retrieval was poor. In that situation, test improvements to the candidate pool alongside changing the reranker.

Work out whether reranking pays

Chapter 8 works through an example where scoring 20,000 candidate tokens costs about $0.001 and selecting 2,000 of them saves $0.09 in uncached frontier-model input. That leaves ample room for the reranker’s cost. The calculation changes when the saved tokens would have been read from a warm cache or sent to a cheap model: reranking can then cost more than the input it removes.

Latency needs its own measurement. One third-party comparison reported a substantial gap between hosted and self-hosted rerankers, but did not state enough about hardware, batching or concurrency to isolate the cause.⁶⁷ Measure end-to-end latency under your expected load before deciding whether self-hosting earns its operational cost.

The options, roughly ordered by cost

Late interaction retains term-level detail that a single document vector compresses away. A controlled comparison using the same backbone found only about one nDCG@10 point between the two approaches.⁶⁸ Compression and on-disk token embeddings reduce its storage burden, but for English text it still needs to justify its cost against dense retrieval plus a cross-encoder.

For visual documents, the design offers a different benefit. Late interaction over page images can search PDFs, slides, scans and tables without a separate OCR and layout-parsing pipeline.⁶⁹ That makes it a relevant option when converting the page to text is itself a source of lost evidence.

Rank for the support an answer needs

A passage can resemble a question without supporting its answer. In a legal question-answering benchmark, semantic similarity was a poor predictor of which paragraphs the model cited; similarity ranking surfaced those paragraphs worse than random selection.⁷⁰

That finding is specific to the benchmark, but the distinction applies to citation-bearing products: evaluate whether selected passages support the answer, alongside whether they resemble the query. Otherwise a relevance gain can leave attribution unchanged or worse.

Restore surrounding text after ranking

A chunk may omit the heading that names a software version, the preconditions above it or the caveat below. Once the winning chunks are chosen, retrieving their neighbouring sections can give the answer model a more complete passage.

Doing this after reranking keeps the scorer’s inputs focused. Expanding first would change what the reranker compares, because each candidate would include additional surrounding material. This is parent-document retrieval applied at the end of selection: match and rank small chunks, then restore the larger section. Measure whether expansion helps the reader and account for the extra tokens before final packing.

Measure the final shortlist

Alongside ranking scores, measure retrieval failure at the k you actually admit: how often is the needed answer absent from what the model finally sees? This exposes a failure the rest of the pipeline cannot repair by reading more carefully. Then measure the reader’s answer too, because presence alone does not guarantee successful use.

7. Selection and ordering

The ranked list is a set of candidates. The prompt is a bounded collection of evidence. Turning one into the other means deciding what fits, what repeats material already selected and what must remain together for the answer to be understandable.

Work with the remaining budget

Retrieved evidence shares the window with instructions, tools, memory, conversation history and space reserved for output. In the session example from chapter 1, conversation and tool results had grown far larger than the fixed overhead.⁷ A budget that worked at the start of the session may be too large several turns later.

Count those categories before packing evidence. Deferred tool loading can recover substantial room in a tool-heavy setup; pruning history may matter more later in a long-running agent. Which change helps most depends on the inventory, rather than on a fixed percentage assigned to retrieval.

Scope before you select

A compiler engineer and an infrastructure operator may search the same organisation’s collection while needing different default sources. Scoping the query to the relevant project or source bundle reduces the candidate pool before retrieval starts. Material outside that scope cannot distract the reranker or reader.

The trade-off is cross-domain recall. A question may genuinely need material from another team’s sources, so make scope visible and changeable rather than an unexplained permanent restriction. This is a practical filter to test, especially in collections full of similar terminology from different projects.

Tune how much reaches the reader

The studies in chapters 2 and 4 point to three useful considerations: reader accuracy can level off while retrieval recall continues to climb; convincing near-misses can be harmful; and removing older tool results can sometimes improve answers.¹⁰²⁸²⁴

Together they suggest starting with a broad candidate search and a selective final prompt. Five to ten reranked chunks is a practitioner starting point; Anthropic found twenty best for its own setup.⁴³ Vary the admitted count and token budget against the final task score. An improving recall curve does not by itself justify a larger prompt.

Select a useful set, not just high-scoring items

If the five highest-ranked passages repeat the same fact, selecting all five can leave no room for a lower-ranked passage containing another fact the answer needs. Per-item relevance and set usefulness are different objectives.

Maximal marginal relevance, or MMR, addresses repetition by balancing a candidate’s relevance against its similarity to items already selected.⁷¹ More recent work frames the problem as selection under a token budget: each candidate has a cost, and the value of adding it depends on the current set.⁷²⁷³⁷⁴

This resembles a knapsack problem: choose useful items that fit within a capacity. If the objective has diminishing returns—a submodular objective— particular algorithms can provide an approximation guarantee. The clinical-text study’s budget-aware algorithm guarantees roughly 63% of the optimum under its stated assumptions.⁷² That mathematical guarantee applies to its objective and algorithm, not to answer accuracy or every greedy packer.

A facility-location objective rewards a set for representing the candidate pool well. MMR penalises repetition; facility location also asks how well unselected candidates are represented by the chosen ones. In one comparison, that distinction improved answer accuracy despite slightly lower evidence recall:⁷³

These results do not establish a universal winning packer. They show why the reader’s score belongs in the comparison. Another study checked whether the gold answer survived as a contiguous span in the packed context. That predicted exact-match accuracy better than document recall, with substantial variation even among questions for which every gold document had been retrieved.⁷⁴ Retrieving a document and preserving its answer are separate steps.

The three studies had not been independently replicated as of August 2026. Include simple baselines: under a tight budget, taking the beginning of a document can work well when the corpus puts key facts first, and poorly when it does not.⁷²

Removing redundancy

Exact duplicates are a straightforward place to recover space. Compare the bytes of candidate chunks and keep one copy of each repeated passage. A study found very different savings across its corpora:⁷⁵

The study reported no quality regression across four model vendors. The large conversational saving describes that corpus, rather than conversation in general; the near-zero academic saving is equally useful when estimating whether the step will matter.

After exact matching, fingerprint methods such as MinHash or SimHash can find near-duplicates in scraped or conversational text. Embedding similarity can also find paraphrases, at higher processing cost.⁷⁶ Add that complexity when inspection shows useful remaining redundancy, and check that near-match removal does not discard distinct facts.

Order for the input type and task

The original Lost in the Middle result suggests placing important evidence at the edges. As chapter 2 explained, later work did not consistently reproduce that pattern for text retrieval.¹¹ Small sets of test questions were a major source of unstable conclusions.

In the controlled replication, ordering made little difference with five to ten text passages. With fifty to a hundred passages on multi-hop questions, putting the best evidence last gave a small gain on one dataset. The other dataset showed no ordering sensitivity. These are useful starting hypotheses for your own prompt, rather than a rule for all retrieval.

Other input types behaved differently. Visual question answering showed a substantial advantage for placing the answer-bearing image first.⁷⁷ Tool-result experiments favoured newer results over older ones.²⁰ And Chroma’s filler experiments suggest the content surrounding evidence can matter as well as its position.¹³

For a short list of text chunks, first test whether order makes a material quality difference. If it does not, stable ordering may offer a caching benefit. For long multi-hop or visual inputs, compare placing the best evidence at the beginning and at the end.

Order for the cache

Caching can reuse the beginning of a prompt when it matches a previous request. Reordering the same evidence may therefore change processing cost and latency even if answer quality stays similar.

One study maintained a prefix tree of recently served evidence sequences and placed reusable prefixes first. This reduced median time to first token by 20–33%, with no measured answer-quality loss.⁷⁸ It operated at the prompt layer, without changes to the serving system.

The result had not been independently replicated, but it gives a reason to measure reuse before repeatedly reordering a small evidence set for marginal relevance gains. Chapter 9 explains the cache mechanics and costs.

Reserve room for the evidence you need

Instructions and coordination messages can fill the window until little evidence remains. In one fixed-window experiment, performance fell sharply when only a few hundred evidence tokens survived. When coordination tokens were added outside that fixed budget, preserving the evidence, the tested commercial models stayed correct even at a very high coordination ratio.⁷⁹

In that setup, the loss came from evidence being displaced. It suggests reserving enough room for the task’s required material and then fitting coordination around it. Tool schemas are worth inspecting early: a separate study recovered substantial accuracy by compressing schemas under a tight budget.⁸⁰

How far other context can be cut needs a local measurement. One study retained near-baseline success at 75% of the original context, with a sharper decline between 50% and 35% retention.⁸¹ Those are results from one unreplicated setup, not safe and unsafe percentages for every agent. Inspect what was removed at each step, especially facts and instructions needed later.

What still needs local measurement

The cited work does not supply a validated allocation formula for instructions, tools, memory, evidence, history and output. It also leaves output reservation largely unmeasured: there is no established rule for how much space to hold back for the answer.

Other gaps remain around why text and visual ordering effects differ, and how source overlap changes useful evidence coverage. The provider comparison in chapter 4 is only 24 queries in one run. Treat these as parts of the pipeline to instrument, rather than filling unknown quantities with fixed percentages.

8. Compression

Compression reduces material you have already chosen to send. It can make a large, changing input affordable, but it also creates another opportunity to lose a fact, qualification or citation. Before paying for that transformation, check whether extraction, selection and caching can solve the budget problem.

Match the compression claim to the material

Repeated examples in a few-shot prompt contain scaffolding a compressor can remove many times. A web page with a single crucial detail presents a different problem. The ratio achieved on one does not establish what can be safely removed from the other.

LLMLingua’s widely quoted “up to 20×” result came from a mixture including few-shot reasoning prompts with repetitive structure.⁸² Its successor, LLMLingua-2, reported 2×–5× compression.⁸³ The retrieval-oriented LongLLMLingua reported improved accuracy with around four times fewer tokens on NaturalQuestions.⁸⁴ Each result describes its own data and setup; together they give a range of possibilities to test, rather than one expected saving for retrieved documents.

The implementation’s maintenance also belongs in the decision. At the 4 August 2026 check, LLMLingua remained available under MIT but had little recent commit activity. The tooling appendix retains that dated snapshot.

The families, briefly

Compression methods differ in what they preserve and what access they need:

Text-based approaches can sit in front of a hosted API. Methods that inject compressed internal states require access below that API boundary, so they are not interchangeable deployment options.

RECOMP also allows its compressor to return an empty string when retrieved documents are irrelevant.⁸⁵ This combines compression with an admission decision: some material should contribute no context at all. Measure that choice separately from how tightly useful material can be compressed.

The arithmetic

Suppose a search returns fifty chunks of 400 tokens each: 20,000 candidate tokens. Selecting five leaves 2,000, saving 18,000 tokens in the answer call. Whether the selection step pays depends on the rate those removed tokens would otherwise have cost.

The example below uses prices checked on 4 August 2026. It compares the cost of processing the candidates with the value of the input removed. The LLM compression rows count its input cost only, so its output adds further cost.

A ratio below one means the processing step costs more than the input tokens it saves. Reranking has ample room to pay against uncached frontier input in this example, but less against inexpensive cached reads. LLM compression costs about twenty times as much as reranking before its output is counted. A large reduction in token count can therefore leave the bill higher.

Compression and caching interact

A cache can reuse a stable prompt prefix without processing its tokens again. Query-aware compression creates different text when the question or selected documents change, reducing that opportunity for reuse. A stable compressed result can be reused, but a new summarisation pass on every request needs to earn its cost against the alternative of caching the original material.

Compare the complete routes: compression cost plus the reduced answer input, and the writes, reads and misses of the uncompressed cache. The observed hit rate matters. One study found compression overtaking caching at around a sixfold reduction, with a cache-hit plateau near 0.83 on the API it tested.⁸⁶ That is one provider and one recent study, not a general crossover threshold.

Latency follows the same dependency. A compression call must finish before the answer model can start reading its result. On a cold cache, reduced prefill may recover that delay. On a warm cache, the original prompt’s processing might already have been reusable. Measure the serial compression call as part of end-to-end latency.

Check the details that survive

A compression ratio describes the size change. Task quality depends on what remains: the needed facts, their qualifications and the evidence supporting them. Measure downstream accuracy, claim recall and citation support alongside the ratio.

Preserve applicable source conditions as well as answer evidence. Extraction, selection and compaction must retain attribution, permitted-use and reporting references for material still in use. These can live in structured records enforced outside the prompt, provided later actions can still resolve and honour them. A shorter prompt must not silently remove a condition or turn a source’s reporting request into permission to disclose private data. The companion guide explains how publisher conditions differ from instruction authority. Establish policy eligibility before ranking; a higher usefulness score cannot override it.

An Anthropic cookbook example makes the distinction concrete. A research agent read eight documents, then compacted its context. All three checked high-level facts survived, while none of three checked appendix-table details did.⁸⁷ A summary can preserve the subject of the documents yet lose the numbers a later question needs. This was one small demonstration, so it motivates a fidelity check rather than establishing a general loss rate.

Decide when compression earns a place

A practical sequence is to reduce avoidable input before transforming useful input:

1. Fetch material selectively, using search results to choose what needs a full read.
2. Extract the body from raw pages, checking for lost content.
3. Rerank and select passages against the task and token budget.
4. Cache the stable part where reuse makes it worthwhile.
5. Test compression for the remaining large, redundant material, especially when it changes per request and will go to an expensive model.

Each step needs an answer-quality check. Selection can omit evidence just as summarisation can. The advantage of this sequence is that the compressor only processes material still worth sending, and its savings are compared with a realistic caching alternative.

9. Caching

An agent often sends the same instructions, tools and earlier messages on successive requests. Prompt caching reuses the work of reading that repeated prefix. When much of the prompt stays stable, the read discount can make a large difference to cost without removing any content.

How prompt caching works

While reading a prompt, a model computes intermediate results for its tokens: the KV cache. Prompt caching retains reusable work on the server for later requests with a matching prefix. Providers differ in whether they charge for writes, reads or storage, and in how long the cache remains usable.

Cached tokens still occupy the context window. Caching reduces processing and billing for repeated material; a 50,000-token cached block still takes 50,000 tokens of room alongside the new question and answer.

The numbers, checked 4 August 2026

The table records the providers’ pricing and caching documentation at the original check date.⁸⁸ Read the columns together: a low read price may come with a write premium, a minimum prefix size or storage charges.

TTL is the time to live: the period during which an entry remains eligible for reuse. With Anthropic’s five-minute cache, one write and one read cost 1.25 + 0.1 = 1.35 times the base input price, compared with 2.0 for two uncached requests. The one-hour cache needs two reads to recover its higher write premium.

Google’s explicit caching adds hourly storage. At the recorded rate for Gemini 3.1 Pro Preview, holding 200,000 tokens for an hour cost $0.90 before any read. That makes the expected number and timing of requests part of the decision; retaining a rarely reused cache can outweigh its read savings.⁸⁸

Minimum prefix sizes also vary by model. Anthropic processes requests below the minimum without caching and returns no error.⁸⁹ Check the relevant model’s threshold and inspect the cache counters instead of assuming that a cache request succeeded. The prices and limits are a dated snapshot; check them before using the calculation for a deployment.

Build a stable prefix

A cache matches an exact request prefix. Changing content before a breakpoint prevents reuse at or after that change, even if the new wording means the same thing.⁸⁸ Arrange stable material before volatile material:

tools (frozen)  →  system prompt (no timestamps)  →  cached history
                →  [BREAKPOINT]  →  retrieved evidence, this turn's question

Retrieved evidence usually changes with the query, so it belongs after the last stable breakpoint unless you have observed reuse. Repeated jobs can be different: recurring evidence sequences may form reusable prefixes, as in the ordering study in chapter 7. Measure that reuse before deciding how to arrange the retrieved blocks.

This is a real trade-off with relevance ordering. Reordering the same passages might slightly improve reading on one task while invalidating an otherwise useful prefix. Compare both answer quality and cache behaviour.

What mis-ordering costs

Consider a 50,000-token evidence block on the recorded Opus 5 prices. The same content has three costs depending on whether the server can reuse it:

A changed prefix that forces a write on every request costs more than ordinary uncached input, because each request pays the write premium. In this example, that is about $288 more per thousand requests than warm reads, and 25% more than uncached input. For a deployment estimate, include the initial write, later reads, expiry and the measured hit rate.

Find why a prefix changed

Many cache misses begin with small changes unrelated to the main task. Inspect the prefix for timestamps, request IDs, UUIDs, per-user data, conditional sections and tool sets that vary between requests. Also check whether JSON serialisation or iteration over an unordered collection changes the order of otherwise identical content.

Two provider-specific behaviours deserve separate checks:

• Anthropic’s breakpoints look back at most twenty content blocks. If a turn adds thirty tool calls, the next request can miss the earlier reusable prefix. Intermediate breakpoints, placed about every fifteen blocks in the documented pattern, help within the four-breakpoint limit.
• A cache entry becomes available only after the first response begins. Launching identical requests simultaneously can therefore make each pay for fresh processing. When the workflow permits it, starting one request and waiting for its first token lets later requests reuse the entry.

Anthropic’s diagnostics beta reports the earliest divergence with labels such as system_changed, tools_changed, messages_changed and model_changed.⁹⁰ That helps distinguish a content change from a model or tool configuration change.

What requires control of the server

Research on KV-cache eviction, attention sinks, compression and quantisation works below the hosted API boundary. StreamingLLM, H2O and SnapKV require control of the inference implementation.⁹¹ A caller of a hosted API can arrange prompts and use the exposed cache features, but cannot substitute those internal algorithms.

For self-hosting, the August 2026 tooling check found automatic prefix caching enabled by default in vLLM and SGLang’s RadixAttention providing configurable eviction. Compare performance against a well-configured server. Large speedup multipliers from early KV-cache papers used older models and baselines and do not predict the gain over those newer configurations.

10. Agent memory

A long-running agent needs to remember decisions, recover earlier evidence and avoid rereading everything on each turn. The transcript alone is a costly way to do that: it grows continuously and contains both useful state and the intermediate work that produced it.

Separate fixed overhead from growing history

In the session breakdown used earlier, listed instructions, tools, memory files and skills totalled about 22,600 tokens, while messages occupied about 185,400.⁷ The source’s broader grouping put fixed overhead around 28,000; under either grouping, conversation and tool results dominated.

This gives you two maintenance jobs. Deferred loading reduces definitions that would otherwise be present before work begins. Managing history controls the tool results and messages that accumulate during work. Solving the first does not stop the second from growing.

Four operations for memory

LangChain organises context management around four verbs:⁹²

• Write: store information outside the window, in files, memory records or scratchpads.
• Select: bring back the parts needed for the current task.
• Compress: summarise or trim material that remains in context.
• Isolate: give separate work its own window.

These operations can be combined. An agent might write a progress record, select the relevant source files on resumption and use a separate context for a bounded research task. The design question is what each operation must preserve for the next step to succeed.

Compaction, and what it loses

Compaction replaces a long conversation with a summary. It creates room to continue, but the summary becomes a dependency: information omitted from it may no longer be available to the model.

Anthropic’s server-side implementation, as documented in August 2026, triggered by default at 150,000 input tokens, with a minimum trigger of 50,000. Its documentation states that raw history is permanently discarded after compaction.⁹³ The small fidelity example in chapter 8—high-level facts preserved, appendix details lost—shows the kind of omission to test for.⁸⁷

Claude Code’s documented sequence clears older tool outputs before resorting to summarisation. Clearing is mechanical and requires no inference; compaction adds a model pass. That ordering can remove expendable bulk before asking a summary to preserve the remaining state.

Survival also depends on where instructions came from. In August 2026, Claude Code re-injected the system prompt and project-root instructions from disk. Path-scoped rules and nested instruction files were unavailable until a matching file was read again. Invoked skills were re-injected with per-skill and total caps of 5,000 and 25,000 tokens respectively, truncated from the end. These details make the placement and reload mechanism for critical instructions worth checking.

For work across sessions, keep explicit handoff artefacts such as a progress file, task list and git history. Anthropic’s long-running-agent guidance found compaction insufficient by itself for preserving instructions across sessions.⁹⁴ A durable record lets the next session read the current state directly instead of reconstructing it from a compressed transcript.

Sub-agents

A sub-agent can read widely in its own window and return a short result to the main agent. The main context then carries the conclusion instead of every intermediate tool response.⁹⁵ One documented Claude Code example read three files and returned 420 tokens, with those file reads staying outside the main window.

The handoff is selective by design. Give the sub-agent a bounded task and consider which details the parent will need from its answer. A short report saves context only if it retains enough information for the next decision.

Match isolation to the work

Independent reading tasks are easier to split than changes with shared dependencies. Several agents can investigate different sources and report back; agents editing connected parts of a system also need to agree on the decisions their edits imply.

Anthropic’s internal research evaluation found a substantial gain from an orchestrator with parallel sub-agents, at substantially higher token use.⁹⁶ Its stated fit was work with heavy parallel reading, information beyond one window and many complex tools. It explicitly cautioned about tasks needing identical shared context, many dependencies and much coding work.

Cognition described the coordination problem through agents building incompatible halves of the same game.⁹⁷ Its later guidance favoured additional agents contributing analysis while writes remained single-threaded, including review from a fresh context.⁹⁸ These accounts support a practical starting point: isolate read-heavy work with clear boundaries and keep a single owner for interdependent writes.

Architecture is only one explanation for a measured gain. Anthropic’s own analysis found token use explained much of the performance variation in one benchmark. A separate comparison found single agents matching or beating multi-agent systems under matched thinking-token budgets.⁹⁹ Compare systems at similar budgets before attributing the difference to delegation.

Failure analysis reinforces the importance of the surrounding design. A study of multi-agent traces found substantial failures attributable to specification and system-design issues.¹⁰⁰ More agents create more handoffs whose requirements need to be clear.

Progressive disclosure

Skills, deferred tools and just-in-time retrieval share one pattern: keep a short pointer in the window and load full content when it becomes relevant. An available skill might cost about a hundred tokens; loading all its instructions in advance would consume their full size on every request.⁸

Progressive disclosure shifts the question from “what might ever be useful?” to “what does this step need?” It works best when the pointer gives the model enough information to recognise when to load the detail.

Keep tools narrow and let the caller orchestrate

A narrow tool runs a query, applies light scoring or returns evidence through stable inputs and outputs. The caller decides which tool to use next and how to combine results.¹⁰¹ Keeping those decisions above the tools allows the same primitives to support different workflows.

The production example in the appendix uses one retrieval layer for both an agent that chooses its own sequence and a fixed plan-execute-synthesise web pipeline.⁶¹ A model call hidden inside every tool invocation would add cost and delay whether that reasoning was needed or not.

Narrow tools can create a large catalogue, bringing back chapter 2’s selection problem. Deferred loading helps reconcile the two goals: provide specific capabilities, while presenting a relevant subset of their definitions at each step. Clear names and descriptions still matter once that subset is loaded.

Evaluate memory against simple storage

A memory layer earns its place by recovering useful information at acceptable cost. Compare it with the full conversation and with simple file storage before comparing it only with other memory products.

On LoCoMo, the full-context baseline in one commercial memory paper scored about 73%, above the system’s roughly 68%.¹⁰² Letta separately reported 74% using conversation histories stored in a file.¹⁰³ These results show that the benchmark allowed simple approaches to perform well; they do not settle how memory will behave on your longer or more selective tasks.

Configuration also changed reported rankings. A vendor dispute produced substantially different scores for the same system after changes to the harness and inclusion of a disputed category.¹⁰⁴ A controlled study found that changing only the embedding model could reverse the conclusion about which memory approach performed better.¹⁰⁵

These results make LoCoMo a weak discriminator between memory approaches. LongMemEval offers harder tasks, and MemoryAgentBench includes selective forgetting. Whatever suite you choose, hold the surrounding pipeline fixed, include simple baselines and measure the outcomes your own agent needs: recovering a past decision, retaining a detail or leaving obsolete information out of the next prompt.

11. Proving a change helped

Suppose a reranker reduces the prompt by half and the next answer looks better. That is a promising example, but it leaves several possibilities open. The reranker may have selected better evidence, the model may have had a good run, or the question may simply have been easy. An evaluation needs to separate those explanations well enough to support the decision you are making.

Start by naming that decision: whether to adopt the reranker, how much quality loss a saving can justify, or whether a memory strategy retains needed details. Then choose comparisons and measurements that can answer it.

Include the simplest useful baseline

A complex component can look good against its peers while adding little to a simple pipeline. Chapter 10’s memory results illustrate this: full context and plain file storage performed well against specialised memory systems on LoCoMo.¹⁰²¹⁰³

For retrieval, compare with no retrieval and with a simple single-provider top-k pipeline. For memory, include the full transcript where it fits and simple storage with retrieval. Keep costs beside quality. A component that matches a baseline more cheaply may still be useful; one that adds cost needs a corresponding benefit.

Plan enough test cases for the decision

An evaluation score estimates performance on a wider population of tasks. A small sample can move several points simply because different questions were included. For a pass/fail score, the standard error is √(s̄(1−s̄)/n), where s̄ is the observed pass fraction and n the number of independent items. That uncertainty can be as large as the improvement you hope to measure.¹⁰⁶

Plan around the smallest effect you would act on. The figures below illustrate the scale of the problem; they depend on the task variation, test design and assumptions of their sources.

Repeating a task measures how consistently the system handles that task. Adding tasks measures how performance varies across the work you care about. These address different uncertainty. The cited analyses found diminishing returns from repeats, with task variation especially important for agents.¹⁰⁶¹⁰⁷ When a suite is too small to detect the intended effect, additional distinct tasks may help more than another round over the same few questions.

Lowering temperature is not a substitute for a better experiment. It changes the output distribution and can introduce bias or move variation into parts that repeats cannot reduce.¹⁰⁶ Evaluate the settings the system will use.

Questions drawn from the same source are also related. If ten questions all come from one document, treating them as ten independent observations can understate uncertainty. Group, or cluster, the error calculation by the shared document, repository or other source. One worked comparison found the naive error estimate more than three times too small.¹⁰⁶

Hold the surrounding system constant

Changing an embedder alongside a memory method makes it difficult to tell which caused the gain. One controlled study found that the embedder alone could move accuracy enough to reverse the ranking of memory approaches.¹⁰⁵ Vary one component in each comparison and retain the configuration of the rest.

“The same model” needs more detail than a display name:

• Temperature zero does not guarantee identical outputs. An inference study found different completions of the same prompt because numerical behaviour varied with server batching.¹⁰⁹
• Hardware, quantisation and inference backends can shift benchmark scores.¹¹⁰
• Prompt formatting can change performance substantially, including model rankings.¹¹¹

Fix the model identifier, prompt template, decoding settings, reasoning effort and evaluator version. Where you control the server, fix its backend and hardware class too. Where the provider hides those details, record the limits of your control and interleave configurations in time so changing load or service behaviour is less likely to favour one arm.

Measure consistency across runs

Two configurations can have similar average accuracy but different patterns of failure. One may usually succeed on each task; another may alternate between success and a serious mistake. Users experience those repeated attempts, not just the mean across a suite.

τ-bench’s pass^k counts success on all k attempts and makes that distinction visible.³³ The multi-turn study in chapter 2 also found that growing unreliability contributed substantially to the loss from splitting a task across turns.²³

Repeat enough items to estimate run-to-run variation, report the spread and inspect differences smaller than that variation cautiously. Combine this with a sufficiently varied task set; repeated attempts cannot supply missing task coverage.

Learn from a context change that did not help

Standing context files are meant to supply repository knowledge an agent would otherwise need to discover. Whether they help depends partly on whether missing knowledge is causing the failures.

A study comparing AGENTS.md-style instructions across repositories and two agents found no correctness benefit from the tested context strategies.¹⁰⁷ It used paired, task-aware statistical comparisons and inspected near-miss tasks. Failure analysis suggested that agents were struggling with implementation rather than lacking the repository knowledge the files supplied. Adding that knowledge did not address the observed failure.

The study also makes a useful distinction between a null finding and evidence of equivalence. With its small task set, the minimum effect it could reliably detect remained large. A modest benefit or harm could therefore go unnoticed. Its power analysis estimated 120–200 tasks to detect a ten-point effect.¹⁰⁷

For your own change, inspect whether it reached the intended stage and whether that stage was the source of the problem. If an extractor never saw the pages that failed, an extraction change cannot explain their outcome. If a context file supplied facts the agent already knew, a null result says little about cases where those facts are missing. Report what the experiment could detect, including the uncertainty around an apparent lack of difference.

Separate missing evidence from failed use

When an answer is wrong, first ask whether the supplied context contained enough information to answer. If it did not, acquisition or selection needs attention. If it did, the reader or prompt may need attention. A final accuracy score combines both failures.

Google’s Sufficient Context work labels question-context pairs according to whether the material is enough to answer definitively.¹¹² Its automated rater was accurate enough in its own evaluation to explore that distinction, though it remains a vendor-evaluated instrument rather than a perfect label.

Two results show why the distinction matters. Models sometimes answered correctly despite insufficient supplied context, so a correct answer did not prove successful retrieval. Conversely, one model hallucinated much more with insufficient context than with no context.¹¹² Partial evidence can encourage an answer it cannot support.

Include a sufficiency check and a path for recognising gaps. “Some related material was found” should not be treated as equivalent to “the evidence needed for this answer is present”.

Use benchmarks as proxies, then check the outcome

A benchmark is useful when success on it predicts success at your task. HELMET’s weak correlations across long-context tasks show why that relationship needs checking.²⁹ A retrieval test, a synthesis test and a state-tracking test need not rank systems alike.

Choose the closest available tasks, then compare their scores with the real outcome on a held-out sample. If an improved retrieval metric leaves answer quality unchanged, inspect the link between the two before continuing to optimise the metric.

Evaluate the judge as part of the pipeline

Many context evaluations use a model to score another model’s answer. That makes the judge a component with its own errors and configuration, much like the retriever or reader. The memory-benchmark dispute in chapter 10 illustrates how harness choices can change reported results.¹⁰⁴

A judge may also answer a narrower question than you intend. An answer can faithfully follow a retrieved document while that document is wrong, stale or inappropriate. Grounding in the supplied text does not validate the source.

Automatic fact checkers have measurable error rates. In the aggregate leaderboard discussed here, performance was much weaker on long-form expert answers than across the overall set.¹¹³ Use those tools with calibration against the kinds of claims and sources your system produces.

Distinguish system comparisons from individual verdicts

A metric may rank whole systems well while often misjudging individual answers. Averaging across many examples can cancel errors that remain serious when one score determines whether to release one answer.

AutoAIS, a standard automatic attribution metric, reported strong correlation with human judgement at system level, alongside much lower and more variable agreement on individual examples.¹¹⁴ That supports using it to compare configurations over a suite, with suitable calibration. It does not establish that a threshold on one answer is a reliable production gate.

Choose the use of a metric at the level for which it has evidence. If you need to act on individual outputs, validate that decision separately, including which kinds of errors the metric misses.

Use inexpensive scoring where it is valid

A fine-tuned entailment checker can judge whether a claim follows from a passage at far lower cost than a frontier model. MiniCheck’s comparison scored about 13,000 claim-document pairs for $0.24, against $107 for GPT-4 at comparable balanced accuracy.¹¹³

That cost difference can make it practical to score every response in both arms of an experiment. It does not remove the need to validate the checker or make every other metric equally cheap. Treat scoring cost and scoring quality as separate parts of the evaluation design.

Measure faithfulness alongside coverage

A system that answers less can make fewer unsupported claims. If the only metric is faithfulness, a change that causes evasive or incomplete answers can look like an improvement.

Vectara’s hallucination leaderboard illustrates the need to inspect answer rates alongside low hallucination rates.¹¹⁵ The vendor supplies the detector, whose own errors also limit the precision of absolute scores.

Pair faithfulness—whether claims follow from the evidence—with coverage— whether the answer contains the facts needed for the task. An appropriate abstention when evidence is missing is different from omitting facts the system could have supported. Inspect both so a shorter answer does not earn credit merely for making fewer checkable claims.

Check what a citation supports

A working link, a relevant page and a supporting passage are three different checks. A page may discuss the right subject while supplying no evidence for the sentence that cites it.

An evaluation of research reports across fourteen models measured these separately:¹¹⁶

Link validity alone would therefore give a much more favourable picture than claim support. The same work found support accuracy falling as tool calls increased. More retrieval can expand coverage while making attribution harder.

A deep-research benchmark also found a trade-off between precision per citation and the number of effective citations.¹¹⁷ Report both support and coverage, and trace claims to passages where possible. A larger bibliography is not itself a measure of a better-supported answer.

Check for bias against synthesis

An answer that combines distant passages or paraphrases them heavily can be harder for an automatic checker to recognise as supported. An evaluation of factuality metrics found both kinds of bias.¹¹⁸

This matters directly to context optimisation. A change that enables broader synthesis may score worse even when its conclusions are supported. Calibrate the metric on a human-annotated sample containing the kinds of paraphrase and cross-document reasoning you want to enable, rather than assuming performance on local quotation transfers.

Match detectors to the source format

Code, tool results and structured records differ from the prose used to train many grounding detectors. In one study, a prose-trained detector performed poorly on code-agent sources, while a detector fine-tuned on that material performed substantially better.¹¹⁹

That single study does not establish how every detector transfers. It does show why source format belongs in validation: test the checker on code and tool output before using its scores to decide whether a context change helped an agent working with those inputs.

Choose metrics with their omissions in view

No single metric covers the journey from selecting a source to producing a useful, supported answer. Use the table to locate the question each one asks:

Choose the unit deliberately. Document recall counts a source as present even if the useful words were cut from it. An applied study found word-level recall correlated much better with human answer scores than document-level recall.¹²⁰ The same metric name can conceal a substantial difference in what is counted.

Validate a proposed metric across at least two systems as well. Correlation within one system may mainly reflect which questions are easy, rather than whether the metric distinguishes better and worse systems.¹²⁰

Put cost and latency beside quality

The Terminal-Bench example in chapter 3 showed similar accuracy at a 3.7-fold cost difference.³⁹ Without cost, the practical difference between those configurations is hidden.

Report accuracy alongside cost per correct answer and latency, including failed attempts and retries. A context change may be worth adopting because it preserves quality while reducing spend or delay. A quality gain may also be too expensive for the intended workload. Both decisions need the complete route’s costs.

Protect the value of held-out tasks

Public tasks may overlap with training material, and contamination detectors cannot reliably establish that they do not.³⁵³⁶ Supplement public benchmarks with private tasks, material known to post-date a training cut-off where that date is available, or refreshed evaluations.

Keep the held-out set for the final decision rather than repeatedly tuning against it. Record the provenance of its inputs and replace it when repeated use has turned it into part of the development process.

A practical evaluation checklist

The checks below follow the experiment from design to deployment. Their scale should match the decision: an exploratory result can guide the next test, but should not carry the certainty of a well-powered comparison.

1. Write down the decision rule first: the minimum useful effect, primary metric and statistical power needed.
2. Size the task set from that effect. If it is too small, report the uncertainty and detectable-effect limit rather than claiming equivalence.
3. Keep always-pass and always-fail tasks in headline performance. Any separate comparative analysis should be declared before seeing results.
4. Inspect a few intended interventions and near-misses. Check that the change reaches the relevant stage and addresses the failure you mean to test.
5. Fix and record model, serving configuration where available, decoding, reasoning effort, evaluator version and prompt template.
6. Include a do-nothing baseline and a simple alternative.
7. Vary one component per comparison.
8. Interleave configurations so time, service updates and load do not align with one arm.
9. Use repeats to estimate variability. Four to six is a starting point from the cited analysis; adding tasks may be more valuable than further repeats. Keep the intended temperature setting.
10. Analyse paired per-item differences and cluster errors by the shared source.
11. Report pass^k when repeated success matters to the workflow.
12. Score faithfulness and coverage together, including appropriate abstention.
13. Score every response where the chosen validated metric is affordable; otherwise record the sampling design and its uncertainty.
14. Check citation support separately from link resolution and page relevance.
15. Calibrate automatic metrics against human-labelled examples from your data. Size that sample for class prevalence and tolerated uncertainty; report sensitivity, specificity and confidence intervals.
16. Record cost and latency for each configuration.
17. Record missing evidence as well as the evidence that scored.
18. Retain retries, refusals and failures in the evaluation record.
19. Keep raw provider responses where retention is permitted, so the run can be inspected and replayed. Record any retention limits and the gaps they leave.
20. Check the evaluator against the real outcome on held-out data; validate new metrics across at least two systems.
21. Supplement public benchmarks with private tasks and reserve the holdout for the final decision.
22. Preserve the comparison systems’ configurations as well as your own.
23. Follow deployment with an online test designed to remain valid under repeated checking. Repeatedly applying a fixed-horizon significance test to an accumulating dashboard can create false positives.¹²¹

12. Where local measurement still matters

The techniques in this guide give you ways to inspect and change a context pipeline. They do not supply a universal token budget, tool limit or retention percentage. Some useful questions remain open, and several common rules of thumb go beyond the measurements behind them.

Treat rules of thumb as hypotheses

A few examples recur across the chapters:

• Effective window size: no traceable basis was found for a universal 60–70% of advertised length. Effective length depends on the task and model.
• Token density: code was 20–40% denser than prose in the nine-file sample, which cannot establish a ratio for other repositories or tokenisers.
• HTML overhead: stripping machinery removed 87–99% of tokens from four pages. That is a different measurement from the unsupported claim that a typical page is 80% boilerplate text.
• Helpful random documents: a replication traced the reported benefit to prompting and generation constraints, rather than a transferable advantage from noise.²⁷
• False-fact propagation: the poisoning terminology is useful, but the August 2026 source check did not establish a controlled propagation result behind the anecdote discussed in chapter 2.²⁵
• Tool-count limits: the reported point where one catalogue’s selection performance deteriorated was specific to that catalogue.²¹ Tool names, schemas, ambiguity and the model all affect the problem.

Use such numbers to choose an initial experiment, then measure the relevant quantity in your system. Their value is in helping frame the question, not in removing the need to ask it locally.

Measure the parts of source use that scores omit

A final answer score leaves much of acquisition unexplained. Your evaluation may need to establish whether the chosen source is authoritative, whether a citation supports the exact claim and whether the evidence is still current. These are distinct from finding related text.

The same applies to gaps and conflicts. A system needs a way to recognise that it lacks enough reliable evidence, and to handle two sources that disagree. The benchmarks discussed here cover these unevenly: RECON begins to examine source conflict,⁴⁰ while chapter 4’s provider comparison measures host overlap without settling which source should prevail.

Retrieved web content also arrives as untrusted input. Ordinary relevance scores do not establish that following instructions embedded in that content is appropriate. Include the source-selection, attribution, freshness, abstention and input-handling decisions that matter to your application, rather than assuming answer accuracy covers them.

Results worth testing in other settings

Several findings suggest useful experiments without yet supporting broad policies.

One corpus-scale study found plain keyword search overtaking an agentic searcher above roughly ten million corpus tokens, at much lower query-token cost.¹²² That is one team’s corpus construction; replication would help establish whether the result concerns search policy, corpus scale or the particular task design.

The tool-catalogue study left about ten points of selection loss even with a perfect shortlister.²¹ If that pattern transfers, part of the remedy lies in clearer descriptions and disambiguation after retrieval. Test whether the correct tool is missing from the shortlist or present but not chosen.

Long-context degradation also varied substantially across models and failure types.¹⁹ A policy fitted to one model needs checking when the reader changes. And the small compaction example preserved broad facts while losing table details,⁸⁷ leaving open how to measure fidelity across different tasks and longer runs.

Carry the method into the next experiment

When a new technique appears, connect its reported result to the decision in front of you:

1. Identify the material and task it was tested on. Repeated examples and retrieved web pages have different opportunities for compression.
2. Keep the model and date with the result. An older finding can explain a failure mode without predicting its size on a newer model.
3. Inspect the baseline. Full context, simple file storage or one search call may be the comparison that matters.
4. Check who configured the experiment and whether its details are available, especially in vendor comparisons.
5. Check whether the benchmark still separates systems and whether it predicts the outcome you need.
6. Read follow-up work for changes in setup, narrower results and replication.

Then make one bounded change, preserve the evidence needed to explain it and compare the complete answer path. A useful context policy is one you can connect to better answers, lower cost or more reliable behaviour on the work it is meant to support.

Glossary

Attention — the mechanism by which a model decides, for each word it produces, how much to draw on each earlier word.

Balanced accuracy — the average of the success rate on positive examples and the success rate on negative examples, so a large class cannot dominate.

Bootstrap — estimating uncertainty by repeatedly resampling the observed items and recalculating the result.

Compaction — replacing a long conversation with a written summary of it and carrying on from there.

Context window — the bounded working material a model can hold at once, counting what you send and what it writes back in that turn. Encoded images and other supported inputs can count too.

Context rot — the tendency for a model to get less reliable as you give it more input, even when the extra input adds no reasoning requirements.

Cross-encoder — a model that reads the question and one candidate passage together and scores how well they match. Used over a shortlist because scoring every query-passage pair across a whole collection would be expensive.

Dense retrieval — finding documents by meaning: both question and passage become lists of numbers, and closeness between the lists stands in for closeness in meaning.

Distractor — a passage that looks relevant but is not, and which makes the answer worse.

Effective context length — the length up to which a model still does the job properly, as opposed to the length it will accept without erroring.

Equivalence test — a test of whether a difference is small enough to count as practically equivalent, rather than merely failing to find a difference.

Embedding — a list of numbers representing a piece of text, arranged so that similar texts have nearby representations.

Extraction — pulling the real content out of a web page and throwing away the navigation, footers and adverts.

Hard negative — a distractor that is especially convincing because it closely resembles what was asked for.

Hybrid search — running word-matching and meaning-matching lookups at the same time and combining the two result lists.

F1 — the harmonic mean of precision and recall. It is high only when both are high.

Facility location — a selection objective that rewards a chosen set for representing the whole candidate pool.

JSON Schema — the machine-readable description of what arguments a tool accepts. Usually the bulk of a tool definition’s size.

KV cache — the intermediate results a model works out while reading your input, kept in memory so it does not recompute them for every word it writes.

Late chunking — feeding a whole document through the meaning-encoder first, then cutting the result into pieces, so each piece carries traces of its surroundings.

Lost in the middle — the pattern where a model uses material at the start and end of a long input well and largely misses what sits between them.

MCP (Model Context Protocol) — a standard for connecting tools and other resources to an agent. Tool definitions occupy context when loaded.

MMR (maximal marginal relevance) — a selection rule that balances relevance to the query against similarity to items already selected.

nDCG at k — a ranking score that gives more credit when relevant results appear near the top of the first k positions.

Oracle — an idealised condition supplied with information a real system would have to discover, used to separate that discovery step from later work.

Needle in a haystack — a test where one specific fact is hidden inside a large body of unrelated text and the model is asked to find it.

Parent-document retrieval — matching on small pieces because they match precisely, then handing the model the larger section each small piece came from.

pass^k — the fraction of tasks a system gets right on every one of k separate attempts, rather than on its best attempt.

Prefill — the phase in which a model processes input before generating output. Reusable cached work can reduce the processing needed.

Precision — the share of selected material that is relevant. High precision does not show that all relevant material was found.

Permutation test — a comparison that repeatedly shuffles which result is labelled as belonging to each system to estimate how surprising the observed difference is.

Prefix — the content at the very start of a request. Prompt caching reuses it only when the provider recognises an exact matching prefix.

Progressive disclosure — keeping a short pointer in the window and fetching the full text only when it turns out to be relevant.

Prompt caching — storing the unchanging front portion of a request on the server so repeat requests are cheaper. It does not make that portion smaller.

Quadratic cost — in naive dense attention, the property that doubling the input roughly quadruples the comparison work.

Submodular objective — a score with diminishing returns: adding an item helps less when similar useful items have already been selected.

Sensitivity and specificity — respectively, the share of true positive cases and true negative cases that a test identifies correctly.

TTL (time to live) — how long a cached entry remains eligible for reuse.

RAG (retrieval-augmented generation) — fetching relevant material and putting it in the prompt before asking the model to answer.

Recall at k — of everything in your collection that would have helped, the share that appeared in the k things you handed the model.

Reranking — taking a rough shortlist and re-ordering it with a slower, better model before deciding what to keep.

Reasoning tokens — text the model writes to work a problem out before answering. You are billed for it as output even when you never see it.

Sub-agent — a second instance of the model, given its own fresh window, that does a job and reports back a short answer.

Token — a chunk of text the model treats as one unit: a short word, part of a long one, or a piece of punctuation. Roughly four characters of English.

Tokeniser — the software that chops text into tokens before the model sees it. Different vendors use different ones, so the same sentence has different token counts on different services.

U-shaped curve — a graph of accuracy against the position of the important information: high at both ends, low in between.

Appendix: one production system, end to end

Cerebras’ account of its internal knowledge base brings the earlier techniques together in one pipeline.⁶¹ It serves questions across data-centre operations, chip design, hardware, training, inference and cloud. People, automations and agents use the same underlying sources.

The sequence shows where each decision happens:

sources          Slack, wiki, code, incidents, team databases
   |             one connector per source, one row shape
distillation     LLM rewrites each item into a canonical record       -> ch 5
   |             filtered by term rarity, length, reactions           -> ch 5
index            a single embeddings table, any source queryable
   |
scope            default project narrows the corpus                  -> ch 7
   |
retrieve         six lists in parallel: lexical, semantic, per-source -> ch 4
   |
fuse             reciprocal rank fusion, k=60, combines the lists     -> ch 4
   |
rerank           small model scores merged candidates, keep 10        -> ch 6
   |
expand           pull neighbouring sections into the selected text   -> ch 6
   |
synthesise       answer with citations

Filtering runs before indexing, and project scoping narrows the pool before retrieval. Both reduce the work passed to later stages. Expansion happens after ranking, so the ranker compares focused chunks while the answer model receives more complete passages.

The same primitives support two ways of coordinating the work. Agents call tools and choose their own sequence; the web interface follows a fixed plan-execute-synthesise pipeline. This is chapter 10’s separation between retrieval operations and the caller’s decisions.

What it cannot tell you

The account reports use at about 15,000 questions a day three months after launch, but gives no retrieval-quality scores, simpler-baseline comparison or evaluation method. That usage describes adoption. It cannot identify how much the rewriting, filters, parallel searches or reranker each contribute.

Use the account to understand how the stages can fit together. To decide whether the same complexity belongs in another system, compare individual stages against a simpler pipeline as chapter 11 describes.

Appendix: tooling, checked 4 August 2026

This snapshot records GitHub and package-registry checks made on 4 August 2026. Maintenance status can change, so check the current project before adopting it. The distinctions below—converter versus extractor, reference implementation versus maintained package, legacy server versus replacement— remain useful questions to ask.

How to check whether a project is alive

A recent pushed_at timestamp can reflect a README edit or dependency bot. Commit participation gives a longer view: GitHub’s repos/OWNER/REPO/stats/participation endpoint returns weekly commit counts. Read those alongside the README and release history to understand where active development is happening.

Letta illustrates why the time window matters. Its high annual count concealed much lower recent activity, and its README identified the repository as the legacy server and directed users elsewhere. The organisation’s active repositories concerned a TypeScript coding agent. A package install and a project name alone would not reveal that change of focus.

Picks by stage

These are candidates and cautions from the dated snapshot, to be considered alongside the relevant chapter’s evaluation:

Dependency counts help estimate the installation footprint of a memory system, though they do not measure its quality. The same snapshot recorded:

LangMem’s recent commits were dependency-bot changes. Among evaluation tools, ARES had no commit since March 2025; OpenAI Evals’ last release was May 2024 and its README pointed to a hosted product; Deepchecks’ last LLM release was December 2024. RAGAS had moved organisations and had not released since early 2026. These are maintenance observations in August 2026, not claims about their status today.

For extraction, the earlier notes also recorded Readability recall of 0.729 in one benchmark run, distinct from the JavaScript result in chapter 5. The figures need their implementation and run attached before comparison. html2text was GPL-3.0 with little activity; its extraction score appears in chapter 5.

Read licence files as well as repository badges. The snapshot found Morphik under BSL 1.1, context-mode under Elastic License 2.0, Dify and holaOS under modified Apache-2.0 terms with multi-tenancy restrictions, and OpenViking and chunkr under AGPL-3.0. Other repositories marked NOASSERTION by GitHub contained Apache-2.0 or MIT licences. The badge alone did not explain the terms.

Stars indicate attention rather than maintenance or fitness for the task. Compare activity over time, the maintainers’ own status notes and the component’s behaviour in your pipeline before relying on popularity.

About this guide

The authors build software that decides which content an AI agent may take in and keeps a record of what it took in, including a plugin for Anthropic’s Claude Code. Anthropic accounts for about one citation in six; where an Anthropic post evaluates an Anthropic technique, the text marks it as a vendor evaluating itself. Anthropic’s API documentation is treated as authoritative for the behaviour of its own products.

Three measurements are the authors’ own: token density across nine repository files, HTML stripping on four pages and search-provider overlap across 24 queries. Each is labelled with its sample size and limitations where it appears.

Sources

Source checks date from 4 August 2026. Recorded URL checks returned HTTP 200 except where a reference notes an exception, including the Cerebras article in reference 115. A resolving URL establishes availability; the quoted figures were also checked against source content rather than automated summaries. Several vendor results come from internal evaluations with unpublished task composition, identified where that affects interpretation.

The three original measurements have different scopes:

• Token density used tiktoken o200k_base across nine files from this project. Four of them are not published; the measurement script reruns the method on published files of the same types.
• HTML stripping used the same tokeniser on four fetched pages, before and after removing tags and scripts. The four original HTML inputs were not retained, so their published figures cannot be reproduced exactly. Neither token measurement transfers exactly to Claude or Gemini tokenisers.
• Provider overlap used 24 English-language queries across six categories, sent once to three search providers. Four of 72 calls failed and were excluded. The script is docs/guide/measurements/source_overlap.py; a new run samples changing search results rather than reproducing the snapshot.

In the provider sample, Exa contributed 61% unique hosts and the other two providers about 26% each. Across 295 distinct hosts, the most frequent were reddit.com (30 appearances), youtube.com (24), arxiv.org (19), medium.com (19) and docs.vllm.ai (19). These describe host coverage, not independent evidence or source quality.

References