A 100K Context Is Enough: Compress the Memory Instead of Buying a Bigger Window

The window is not the problem

Every few months a lab ships a model with a bigger context window and the conversation resets: two hundred thousand tokens, then a million, then two million. The implicit promise is that memory is a hardware problem, and someone will keep making it bigger.

billion-context, a TypeScript project by ranxianglei that has been climbing since late September, takes the opposite position in its tagline: a 100K context is enough. Not "enough for now." Enough, if you stop treating the conversation as an append-only log.

It is a proxy that sits between your coding agent and the model API. Requests go through it, get compressed, and get forwarded. Responses are rewritten on the way back. Nothing about the model changes. The argument is that the expensive part was never the size of the window, it was what we kept putting in it.

How the compression works

The proxy parses each request in either the Anthropic or OpenAI shape, runs its compression kernel over the conversation, and injects four tools that the model can call itself: compress a range of messages into a summary, decompress a range when exact details are needed again, search the compressed summaries plus visible messages by keyword, and check context usage.

The important design choice is that the model decides when to compress, not a hard threshold. A host's built-in summarizer typically fires when the window fills and replaces everything before a cut point with one summary that cannot be undone. billion-context compresses in small ranges, keeps them reversible, and keeps the prefix cache intact.

That last part is the engineering detail worth caring about. Prefix caching is what makes long agent sessions affordable; if every compression rewrites the beginning of the conversation, the cache misses and you pay full price for the whole history again. The project claims a healthy session holds a 95-97% prefix-cache hit rate and that compression itself costs no more than 2% overhead. Those are its own figures, and it ships commands to attribute a drop in hit rate to a cause, usually upstream cache TTL expiry or a model switch.

The number that makes the claim credible

Vendors claim token savings constantly. What is unusual here is a longitudinal study bundled with the code: four and a half months, three hosts, 174,327 model calls, 18.76 billion cumulative input tokens across one host and about 24.7 billion across all of them. The report says the longest sessions ran 8,584 to 12,049 calls, with zero context-window violations on 204,800-token models.

Zero window violations is the sentence that matters. It means no session died because the conversation outgrew the model. Everything else in the study is downstream of that.

It also reframes what "long session" means. Most people experience context limits as a slow degradation: the agent gets vaguer, forgets a constraint from two hours ago, starts re-reading files it already read. By the time the window is actually full, the session has usually been bad for a while. Compression that runs continuously addresses the degradation, not just the hard stop.

Whether it works as advertised on your workload is a separate question. The claims are self-reported, the benchmark is the author's own usage, and "5× fewer tokens" depends entirely on how repetitive your sessions are. A session that reads forty files and edits three will compress beautifully. A session that reads one enormous file and reasons about it line by line has less redundancy to fold.

What it changes about the daily loop

The practical difference shows up in the boring middle of a long task. An agent working through a refactor has three kinds of memory: what it decided, what it read, and what it tried and discarded. A raw transcript keeps all three at the same resolution, which is why a forty-file refactor fills a window with stack traces that no longer matter.

Compression lets the agent keep the decisions at full fidelity and fold the evidence into summaries, then reach back for the exact lines only when it needs them. That is roughly how a person works: you remember that you moved the validation into the middleware, you do not remember the six attempts that failed first.

The failure mode is equally human. If a summary drops the one constraint that mattered, "this endpoint must stay backwards compatible", the agent will cheerfully break it forty minutes later, and the compression layer will have saved you money while costing you a bug. Reversibility is what makes that survivable: you can decompress and check what the summary actually said.

Why this is a category now

There are at least three distinct approaches competing for the same problem. One is the lab's answer: build a bigger window. Another is the harness's answer: retrieve instead of remembering, and keep only a working set in context. The third, which billion-context represents, is to keep the conversation and compress it in place, reversibly.

The trade-offs are real. A bigger window is simplest and the least controllable; you pay for it on every call, whether or not the history is relevant. Retrieval is cheapest but loses continuity, because the agent does not know what it forgot. In-place compression tries to keep continuity and cost down at the same time, at the price of sitting inside your request path.

That last cost is the one to think about. A proxy in the middle sees every prompt and every completion. For a solo developer that is fine. For a company with data handling rules, it is a compliance conversation, and "we read the source, it is fine" is not the same as an audit.

There is also the small matter of the license. The technical paper is MIT and lives in the repository. The repository's own license is reported as "Other" rather than a standard OSI license. If you are deploying this inside a company, that distinction will come up.

The uncomfortable implication

If compression works well enough that 100K is enough, then a lot of the context-window marketing is aimed at a problem that is cheaper to solve in software. That does not make bigger windows useless, a single 400-page document still needs to fit somewhere, but it does mean that "we have a million tokens" is a weaker selling point than it sounds. It is like advertising a bigger fuel tank instead of better mileage.

It also changes how you should write an agent. If memory is compressible, the agent should be encouraged to write down what it learned in structured form, because structured notes compress far better than raw logs. A session that says "the auth middleware reads the tenant from the header, not the cookie" is worth a thousand lines of stack traces. The compression layer rewards good bookkeeping, which is a nice property for a tool to have.

The project has somewhere north of a thousand stars and roughly 400,000 monthly npm downloads, with badges for more than twenty clients. That is not a hobby-scale install base. It suggests the pain is real and widely shared, and that the answer people are reaching for is not another zero on the window size.

Which is the useful takeaway, whichever tool you end up using. Context is a budget, not a container. The teams that treat it that way will keep getting more done per dollar, and the ones waiting for a bigger window will keep paying for the privilege of not deciding what matters.