Context compression: a paired A/B cost trial of headroom
A client running AI-agent workflows wanted to know whether a context-compression layer cuts cost and tokens without making the output worse. We ran the same real agent task twice, once without headroom and once with it, and recorded every model call, token and cent. Cost fell by 65.7% and the target site received the same audit score. Output quality was not independently assessed. Input tokens fell by only 4.4%, and most of the difference came from the run with headroom taking 5 model calls instead of 12 and writing far fewer output tokens. With one pair, we cannot say how much of the saving headroom caused.
The exact run date was not recorded with the data; both runs took place on the same day, on or before , when the result was first published.
The tool tested
headroom
headroomlabs-ai/headroom on GitHub, Apache 2.0 licence
An open source library and proxy that compresses tool outputs, logs, files and conversation history before they reach the model. We tested it rather than rely on its published figures.
Disclosure: we also contribute to headroom from outside the project.
How it was run
- Workload: a real run of an AI-visibility audit, an agent task that checks how ready a website is for AI crawlers and assistants to find, understand and cite it. It was chosen because it is a realistic, tool-heavy, multi-step job, not a synthetic benchmark.
- Target: a real, independent website, not the client’s own, used as a stand-in with a normal mix of pages, documentation and structured content. Neither the client nor the site is named.
- Model: one fast, low-cost model through a model-routing service, reasoning mode off, the same settings on both runs.
- Runs: 1 paired run (2 runs) of the same audit, headroom off and then on: same prompt, same model, same target, same day.
Results
| Metric | Without headroom | With headroom | Change |
|---|---|---|---|
| Model calls | 12 | 5 | −58.3% |
| Input tokens | 152,704 | 145,985 | −4.4% |
| Output tokens | 66,422 | 10,818 | −83.7% |
| Tokens read from the prompt cache | 548,736 | 206,080 | −62.4% |
| Billed tokens (input plus output) | 219,126 | 156,803 | −28.4% |
| All tokens processed (including cache reads) | 767,862 | 362,883 | −52.7% |
| Cost | $0.1875 | $0.0643 | −65.7% |
| Target-site audit score (out of 100) | 93.5 | 93.5 | no change |
| Runs that finished | 1 of 1 | 1 of 1 |
Input tokens here are the tokens sent to the model that were not read from the prompt cache; cache reads are counted on their own row. The score rates the target site, not the audit output. It was the same on both runs, but matching scores do not establish that the outputs were equally accurate or complete.
Where the difference came from
Most of this difference did not come from compressing the input. headroom shrinks what goes into the model, and input tokens fell by only 4.4% (152,704 to 145,985). The run with headroom also took 5 model calls instead of 12 and wrote 10,818 output tokens instead of 66,422. Fewer output tokens make up 89.2% of the drop in billed tokens and, as output is priced higher than input, probably most of the drop in cost. Cache reads fell by 62.4%, but every call re-reads the cached context, so fewer calls explain most of that too: per call, cache reads fell by 9.9% (45,728 to 41,216). An agent can take a different path through the same task from one run to the next, and with a single pair we cannot tell how much of the difference headroom caused.
A reliability problem before the final runs
The paired run above was not the first attempt. An earlier pass, on a heavier reasoning model, showed compression working, but every repeat run with headroom on failed to finish, while the same task with it off finished every time. The cause was not headroom. It was a bug in the agent framework running the task: a timeout setting was capped by a hidden idle limit in a different part of the code, a regression of an upstream bug that had already been closed, and headroom’s extra processing time made runs more likely to cross that limit. We reported it upstream and worked around it before the final runs by also setting a timeout at the model-provider level. After that, the paired run above, on a second model, finished first time both ways. The failed attempts are not in the results.
Method and caveats
- 1 paired run (2 runs): 1 workload, 1 model, 1 day. There is no variance estimate, and the percentages will differ on other work.
- Input tokens barely moved. Most of the measured saving came from the run with headroom taking fewer steps and writing fewer output tokens, which one pair cannot attribute to headroom.
- The reliability question is not fully closed: after the workaround, the clean paired run was done once.
- The target-site audit score was compared between the two runs. Output quality was not independently assessed: the audit output was not checked line by line for accuracy or completeness.
- The target was a stand-in site, not the client’s own production workflow. How this carries over to their workload is a judgement from the shape of this run, not a measurement on their traffic.
- This is a one-off observational run, not a pre-registered experiment: there was no fixed hypothesis, sample size or verdict rule.
What a stronger test looks like
Repeat the paired run several times, on more than one workload and more than one model, and report a range instead of a single pair of numbers. headroom is built to shrink large tool outputs, such as fetched pages, documents and API responses, so workloads with many of those are where we would expect it to matter most. That is an expectation, not something this run shows.
Related
This run is also the headroom baseline on the token-savings page, which tracks what an open source LLM gateway saves on the traffic sent through it.
Want this for your team?
We test compression, gateways and caching on real agent workloads, with paired runs and the limitations written down. If token spend is a line item you want to shrink, get in touch.
Talk to Buildeon about this experiment