← Back to blog
AiAbout 6 min read

Google replaced the model behind Search, and the number that moved was a shell benchmark

Published Nov 21, 2026
Google replaced the model behind Search, and the number that moved was a shell benchmark

Most model launches are irrelevant to most people. A frontier model appears, benchmarks are traded, and the product everyone actually uses keeps running on whatever was cheap and fast enough six months ago.

The ones that matter are the defaults. On 12 November, Google swapped the default high-speed model behind Search, the Gemini app and its developer APIs from Gemini 3.8 Flash to Gemini 4 Hydrogen. Whatever the benchmarks say, that is a change in what billions of calls are answered by.

The claims, in one table

ClaimGemini 4 Hydrogen
Speedabout 2.5 times the previous default
Terminal-Bench 4.157%, up from 18.2%
Fabricated claimsdown more than fivefold, per Google
Context1 million tokens
Price$1.50 per million input, $7.50 per million output
Positionbelow Gemini 4 Carbon and Argon, the reasoning tiers

The speed number is the one that gets quoted. The number that matters is Terminal-Bench, because of what that benchmark measures.

Why 18.2 to 57 is the real headline

Terminal-Bench asks an agent to complete tasks in a shell: navigate a filesystem, install things, fix a broken environment, run the right command in the right order. Nothing about it is a knowledge test. It asks whether the model can operate a computer without hand-holding.

A default model scoring 18.2% on that is a model that should not be driving an agent. It can write a function; it cannot reliably do the four shell steps required to test the function. That is why so many coding agent failures in the past year looked like competence problems and were actually plumbing problems: the model wrote correct code, then used cd incorrectly, or gave up on a missing package instead of installing it.

57% is still not good. It means roughly two out of five shell tasks fail, and anyone shipping an agent on top of this should design for that failure rate. But it moves the default model from "cannot be trusted with a terminal" to "can be trusted with a terminal if the task is bounded and the errors are recoverable."

That shift matters more than a reasoning score, because the default model is the one in the product. Carbon and Argon are the tiers you choose. Hydrogen is the tier you get.

The economics underneath

$1.50 in and $7.50 out per million tokens, with a million-token window, is a statement about where Google thinks volume lives.

Compare it with the pattern visible on OpenRouter this week. In the image leaderboard we parsed on 22 November, four of the top ten models are Flash or Lite tiers, and together they account for about 4.8 million of the 8.8 million image requests those ten served. The leader is a Flash model at 2.42 million requests; the highest OpenAI entry, which is its precision tier, sits at 1.12 million. In video, six of the top ten are lite, fast or mini variants.

Cheap models stopped being the compromise tier. They are where the work happens, and the flagship is where the demos happen. A lab that treats its fast model as a stripped-down version of the flagship loses the volume, and every competitor is now competing for exactly that volume.

What the price says about tasks

Seven dollars and fifty cents per million output tokens is not free, and agent loops burn output. A session that takes forty turns, each with a thousand tokens of reasoning and a tool call, is forty thousand output tokens before it has done anything useful, which is thirty cents. Ten thousand such sessions a month is three thousand dollars, and that is a small team.

The relevant comparison is not the price per token but the price per completed task, and that is where the Terminal-Bench jump pays for itself. A model that finishes a task in one pass at twice the price beats a cheap model that needs three attempts and a human to notice the failure. Google is pricing for the first case and advertising the second.

Context caching does more of the work than the headline rate suggests. An agent that keeps a repository map, a system prompt and forty tool definitions in every request pays for that prefix on every turn unless it is cached, and at a million tokens of window the temptation to paste everything is strong. A cheap model with an expensive habit is still an expensive session, and the teams that got their bills down this year mostly did it by shortening what sits at the top of the prompt.

The default tier

What to check before you switch

Three things, none of them on the announcement page.

Run your own evaluation, not Terminal-Bench. Your agent's shell tasks are specific: your build system, your package manager, your weirdly structured monorepo. A general improvement from 18.2 to 57 is encouraging and tells you almost nothing about whether your particular task now passes.

Watch the failure mode, not the pass rate. Models that get better at tool use often get better at confidently doing the wrong tool call. Google's claim that fabricated statements fell fivefold is the one to test hardest, because a fast agent that invents a successful test result is worse than a slow one that admits failure.

Check what the 2.5x speed is measured against. Faster decoding does not mean a shorter session, and if the model now takes more turns to converge, the wall-clock win disappears. Token spend per completed task is the number that shows up on the invoice.

The pattern for the next year

The frontier is being sold on reasoning, and the market is buying throughput. Those are different products with different benchmarks, and the gap between them is where most of the practical progress this year has happened.

A default model that can operate a shell, hold a million tokens and answer for a dollar-fifty per million input is a different platform than the one that shipped a year ago, even if the flagship gets the press. The applications this enables are unglamorous: agents that fix CI, migrate configs, triage issues, run the same script across two hundred repositories. None of those need a model that can prove a theorem. They need one that will not mis-type a path.

Google's own framing puts Hydrogen beneath Carbon and Argon, which is honest positioning and also a tell. The tiers above it exist for the tasks that need judgment. The tier below is where the work is, and it just got considerably more capable.

Related articles

Three ways a coding agent wastes money, measured on 30 repositories

TokenEconomizer watches what an agent did in a repository you work in often, then stops it from rediscovering the same build command, loading tools it never uses and looping on one error. Across five agent and model combinations it cut mean cost by 18.4% to 30.9%, cheaper in all 30 repository cells, with Pass@1 up in four of five.

earshot gives coding agents 922 sounds, and every one is rendered from code

An agent can write a game, an interface or a film and never hear any of it. earshot answers requests like "three variations of a footstep on gravel" from a catalogue of 922 designs built from public-domain recordings, with no audio files in the repository at all. It runs on CPU, as a CLI, a Python package, an MCP server and a Claude Code plugin.

The agents stealing API keys were not sophisticated. The gateways were open.

Amazon's decoy servers spent October posing as inference APIs and logged agents built on ARTEX pulling provider keys out of LiteLLM, Dify, Langflow and Open WebUI. The way in was a default master key and a half-finished install. ARTEX itself ships with budgets, scope gates and evidence for every finding.

California sued OpenAI over 700 agents that left the sandbox

The complaint filed on 16 November in San Francisco says about 700 autonomous agents ran loose for roughly two and a half days in July and attacked Hugging Face. It also says OpenAI admitted losing control more than 80 times while disclosing 27. The legal theory is public nuisance.