Saluki 27B Fits a 27B Model Into Two Bits and Still Beats It at Tool Calling

A 27B model squeezed into two bits per weight

On October 9, Underdog AI released Saluki 27B, a compressed build of the Qwen 3.8 27B model. The compression is aggressive: two bits per weight, which packs the whole thing into a file around 7.89 gigabytes. Underdog says the result keeps about 96 percent of the original's benchmark performance, and on tool-calling tasks it actually beats the full-size model. It ships under an Apache 2.0 license for local agent workflows.

The numbers sound like a typo. Cutting a model to a quarter of its bit depth usually trades away a lot of quality. Saluki's claim is that it did not, and that on the one task that matters most for agents, it got better.

Why quantization at two bits is hard

Model weights are stored as numbers, and the number of bits per weight sets how much precision each one keeps. At 16 bits, a weight can take one of tens of thousands of values. At two bits, it can take one of four. You are asking a network built on fine distinctions to express itself with a vocabulary of almost nothing.

The reason it usually fails is that errors compound. Round each weight a little and the layer is slightly off. Stack dozens of slightly off layers and the output drifts far from the original. Most quantization work treats four bits as a practical floor and accepts some quality loss below it.

The methods that push lower do not simply round harder. They decide which weights can afford to be crude and which must stay sharp, and they often fine-tune the compressed model afterward to repair the damage. Getting two bits to hold together is a careful, targeted process, not a blunt one.

The tool-calling result is the interesting part

That Saluki matches the original on general benchmarks is notable. That it outperforms the original on tool calling is stranger, and it points at something real. Tool calling is a structured task: the model has to emit a correct function name and arguments, and small inconsistencies break the whole thing. A model that is slightly noisier can produce slightly more varied output, which in this setting may mean it stumbles less on edge cases.

There is a simpler explanation too. If the compression process included fine-tuning on tool-use data, the resulting model is not just smaller but also narrower in the right direction. A specialist can beat a generalist on its specialty while losing on everything else. The honest read is that Saluki is a compressed model that was also steered toward agent work, and the two changes together explain the result.

Intelligence density is the metric that matters here

Underdog frames the improvement as a gain in intelligence per gigabyte, roughly 7.5 times the original. That framing is exactly right for the local-agent case. When you run a model on your own machine, the constraint is not the benchmark score. It is whether the model fits in your memory and leaves room for the applications around it.

A 27B model at full precision needs serious hardware. The same model at about eight gigabytes runs on a laptop, and a laptop is where a lot of agent work actually happens: reading files, running commands, checking results. The value of compression is not aesthetic. It is the difference between a capable model you can run locally and a capable model you can only rent.

Why local agent models are converging on this size

Saluki is one of several small, openly licensed models showing up to do the repetitive work inside larger systems. The pattern is consistent: a model around 7 to 30 billion parameters, aggressively compressed, permissively licensed, aimed at the tight loop of an agent.

The logic is the same each time. An agent calls its model many times per task, so per-call cost and latency dominate the experience. A model small enough to run locally removes both the network round trip and the per-token bill, and that changes how aggressively the agent can be used. You stop metering your own assistant.

There is a second, quieter reason. Locally run models do not send your data anywhere, which matters for anyone working with code or documents they are not allowed to upload. For that audience, compression is not a cost optimization. It is the only way to get a capable model at all.

The case for small files, in one number

It helps to make the size difference concrete. A 27 billion parameter model at 16 bits occupies roughly 54 gigabytes before any runtime overhead. That does not fit on most laptops, and it does not fit in the memory left over after an operating system, a browser, and an editor have taken their share. At two bits, the same parameter count lands near 8 gigabytes, which does fit, and fits alongside the tools the agent is meant to drive.

That is the whole argument for low-bit compression in a local-agent setting. It is not about running a model that could not run before at all, though that is part of it. It is about running a model comfortably enough that the agent can call it repeatedly without the machine noticing, because an agent that makes the whole system sluggish is an agent nobody keeps turned on.

What to check before believing it

Compressed models deserve a specific kind of skepticism. The headline benchmark may hold while the model degrades on the long, messy inputs that real work produces. Quantization damage often shows up first on unusual formatting, long contexts, and rare languages, none of which a short benchmark captures.

It also matters what "two bits" describes. Bit depth per weight, group sizes, and whether some layers are kept at higher precision all change what the number means. Two-bit in one scheme is not the same as two-bit in another, and the difference is not obvious from a model card.

What this adds up to

The frontier argument is about the biggest model anyone can train. A parallel argument, just as consequential, is about the smallest model that is still good enough to be useful on your own hardware. Saluki 27B sits in the second argument, and it makes a specific, testable claim: that two bits is enough for the work agents actually do.

If that claim holds up outside the benchmark, it lowers the floor for what a person can run locally, and a lower floor is usually where the next wave of products gets built. The model is not the story. The shrinking of the model is.

It is worth remembering that this is a young area. The methods that make two-bit weights work are still being written, and each release is as much a demonstration of a compression technique as it is a product. Saluki is interesting less for what it is today than for what it proves is now possible to fit through a narrow pipe. The next model in this line will be smaller, sharper, or both, and the floor will drop again.