RAG vs Context caching

I have an agent that answers questions about a set of governed definitions. What a concept means, which rules apply to it, which comparisons and tests are valid, what the defaults are when nothing more specific applies. The definitions live as markdown files in git, in an OKF bundle, and the agent already knows how to research them with tools.

The open question was what to give the model on top of that. There’s a second layer of reference material, longer-form product documentation, that fills in context the definitions don’t carry. Roughly 70k tokens of it. Two obvious ways to get it in front of the model:

1. Put all of it in context, and use prompt caching^[1] so you’re not paying full price for it on every turn.
2. Index it, and let a retrieval service pull back the relevant pieces per question.

The received wisdom is that you use RAG for this. I wasn’t sure it applied to a corpus this small, so I built both and measured.

The two arms #

Both arms start the same way: a research agent with tool access reads the governed OKF files and writes a short brief. Every fact in that brief is tagged with where it came from: read from a governed file, inferred from the user’s wording, or missing. That part is identical. What differs is what happens next.

Arm A: corpus in context, cached. The research phase runs with tools on and the product docs off. Then a separate synthesis turn runs with tools off and the docs on. The whole rendered corpus goes in as the first block of the prompt, with a cache breakpoint right after it. The question and the research brief come after the breakpoint.

question
   │
   ▼
research agent  ── reads OKF files ──►  research brief
(tools on, no docs)
   │
   ▼
synthesis  [ CACHED: full product docs ]   ← stable prefix
(tools off) [ DYNAMIC: question + brief ]

Arm B: retrieval. A coordinator runs the OKF research and a managed enterprise retrieval service in parallel, then a synthesis model merges the two briefs. The synthesizer never sees the full corpus, only what retrieval returned.

question
   ├──► OKF research agent ──┐
   └──► retrieval service  ──┴──► synthesis (merge both briefs)

The split in Arm A matters more than it looks. If the research phase used tools and had the corpus in context, tool output and retries would end up interleaved with the reference text, and the cached prefix would stop being byte-identical between requests. Keeping research and synthesis apart is what keeps the cache key stable.^[2]

How I scored it #

This was a micro-suite, not a bake-off. Two questions, both of the kind users actually ask: one broad (“what analyses are valid for this concept?”) and one narrow (“which test applies to this kind of figure?”). Three runs each, per arm.

Scoring had three layers:

• An LLM judge, a separate model with its own credentials, scoring the full answer 0 to 5 on structure, parsing of the question, disambiguation, edge cases, efficiency and latency. A mean of 4 or more is a pass. This was the headline signal.
• Deterministic fact coverage. A checklist of the governed facts a correct answer has to contain, checked against the answer. This is the one that tells you about grounding, as opposed to tone.
• A keyword gate, which I only used for diagnosis. String matching is too cheap to be a headline number.

I also counted every token: uncached input, cache writes, cache reads, output, and on the retrieval arm, the tokens billed by the retrieval service’s own model. Then I dropped every run the judge couldn’t score. One caching run came back with an empty answer and one wasn’t judged, so the caching arm has four scored runs and the retrieval arm has six. Small numbers. Keep that in mind for everything below.

What happened #

Caching won on every quality axis I measured.

The rubric gaps were widest on disambiguation, parsing and edge cases. Those are the criteria that depend on the model knowing the constraints around an entity: which rule applies here and not there, what the exception is, which of two similar concepts the user meant. That’s exactly what you lose when the passage stating the constraint doesn’t make the retrieval cut.

Two things surprised me.

First, fact coverage was mediocre on both arms. Half the checklist is not good. The broad question has a large rule set behind it and both arms under-specified it. Caching made the answers better structured and more careful. It didn’t make them complete. That’s a prompting and evaluation problem I haven’t solved yet, and the caching result doesn’t paper over it.

Second, caching was cheaper. I expected the opposite. The caching arm’s token counts are dominated by cache reads, which is the same 70k prefix read again on each agent turn, not a growing context. The retrieval arm never puts the full corpus in front of the synthesizer, but it runs more models: an OKF research loop, the retrieval service’s own model, and a synthesis loop. That orchestration adds up to roughly 1.8× the tokens per run.

Why the corpus-in-context arm wins here #

Four reasons, roughly in order of how much I think they matter.

Guaranteed presence beats probable presence. If the corpus is closed, bounded and fits in the window, injecting all of it means the relevant fact is there, every time. Retrieval turns that guarantee into a probability, and a missed chunk is an unrecoverable grounding gap. The synthesizer can’t safely invent a governed rule it was never shown, so it hedges or goes vague, and the judge marks it down.

Position still matters, even when retrieval works. Models make less use of material in the middle of a long context. That’s the “lost in the middle” result.^[3] A pile of retrieved passages with no careful ordering or reranking makes that worse. A stable, well-structured corpus that always sits in the same place is easier to use than a different pile of chunks on every question.

A stable prefix is what caching is for. The reference material changes far less often than the questions asked against it. That’s the ideal shape for prompt caching:^[1] an identical prefix, everything per-request after the breakpoint. In the usage numbers you can see it working: lots of cache-read tokens, few cache-write tokens.

Warm loops pay for the prefix once. An agent takes several turns. Each one reads the cached prefix instead of prefilling it again, and each read refreshes the TTL. The more turns, the better the prefix pays off.

None of this says the retrieval service is bad. It’s built for a different problem. For a small, closed set of governed documents, the thing it’s good at, selectivity, is the thing I didn’t need.

When this flips #

The micro-suite tests quality and cost on a fixed 70k-token corpus. It says nothing about growth or churn, and on both of those the literature points the other way.

Scale. The cached corpus has to fit in the context window, and quality degrades as the window fills.^[4] Per-query cost and latency grow with the size of the corpus, because every query reads all of it. Retrieval reads a handful of chunks whatever the size of the index, so its cost stays roughly flat.^[5]

Change. With caching, any edit to the corpus changes the prefix bytes, invalidates the cache entry, and the next request pays the full prefill again. With a short TTL you’re paying cold writes regularly anyway. Retrieval re-indexes only the documents that changed. It costs more to run up front, with chunking, embeddings, a vector store and a reranker to keep in sync with the source, but the cost of each change stays flat as the corpus grows.^[4]

So the honest summary: caching is simpler and cheaper while the corpus stays small and changes rarely. As size and update rate go up, the two approaches swap places. For my corpus, today, that’s a comfortable margin. It’s also a ceiling I can see from here.

What I’m taking from it #

• If your reference material is bounded, stable and reused a lot, try putting it in context with a cache breakpoint before you build a retrieval pipeline. It’s less infrastructure, and here it was better on quality and cost.
• Separate tool-using research from reference-grounded synthesis. It keeps the cache key stable, and it keeps the provenance of each fact clear.
• Measure grounding directly. A judge score tells you the answer reads well. A fact checklist tells you whether it’s right. I needed both to see that neither arm was complete.
• Count every token, including the ones billed by the other model in your pipeline. The “RAG is cheaper because the prompt is smaller” intuition didn’t survive a real count.

This is early. Six runs per arm is a sketch, not a result. Next I want a bigger question suite, cold-cache versus warm-cache splits, an audit of what the retrieval index actually returned for the questions it failed, ablations with each source on its own, and a proper test of the thing that will eventually decide it: grow the corpus, change a document, and time re-caching against re-indexing.