Two Hundred Real Edits, a 0.27-Point Gap: What Image Benchmarks Actually Measure

Three numbers, three different claims

When a new image model ships, you get a press release with a chart. The chart usually shows the new model on top, arranged by a metric the vendor chose, measured on a set the vendor built. Everyone in the industry knows this and quotes the chart anyway, because there is nothing else to quote on day one.

This month's clearest example is FLUX 3 Image Klein, Black Forest Labs' 5.6B open-weights release on October 15. Within a week there were three separate sets of numbers in circulation, and they disagreed in a way that is instructive.

The vendor's Elo numbers: first place in text-to-image, multi-reference generation, and image editing among open-weights models, ahead of larger competitors. Against its own FLUX.2 klein 9B, it gained 91 Elo in text-to-image and 70 in editing while using 38% fewer parameters.

The vendor's latency numbers: a median API latency of 4.54 seconds for text-to-image and 7.18 seconds for editing at one megapixel, faster than the four named competitors. The methodology note adds that the new model was timed on BFL's own servers while the others were timed from the client, including polling and network time, and that one competitor was measured on its medium tier rather than its large one.

The third-party blind test: 200 real editing tasks, scored 6.93 for Klein and 7.20 for Nano Banana 2.1, a 0.27-point gap on a ten-point scale, with the same test finding that Klein struggles on pose adjustment and multi-image compositing. Cost per image: roughly one fifth of the competitor's.

None of these numbers is dishonest. They are answering different questions, and the confusion comes from treating them as the same question.

What Elo can and cannot tell you

Elo is a preference score. It says that when human raters were shown two outputs and asked which was better, they picked one more often. That is a real measurement, and it is the right measurement for "does this look better."

It is also a measurement with a specific blind spot: it collapses task context. A model can win an Elo comparison on average while losing badly on the one thing you need it to do. If your work is product shots on white, pose changes, or compositing three references into one coherent scene, the average is not your average.

The blind test's most useful finding is not the score. It is the failure mode. Two hundred real edit tasks, and the recurring errors were on pose and multi-reference composition, exactly the tasks where the layout machinery is supposed to help. That is a much more actionable sentence than "0.27 points behind," because it tells you which jobs to route elsewhere.

There is also a scale question worth being blunt about. On a ten-point human rating, 6.93 versus 7.20 is a small difference. It is well inside the range where the specific images chosen for the test set, and the taste of the raters, can flip the ranking. A gap that size is a hint, not a verdict.

The cost column is the one people skip

Here is the number that should have been in the headline: one fifth the cost per image.

If model A is 4% better and 5× more expensive, the correct answer for most workflows is not "use model A." It is "use model B for the eighty percent of frames that are easy and model A for the twenty percent that are hard." That is not a compromise. That is how production has always worked, nobody shoots every frame on the most expensive stock.

Cost also changes behavior. A model that costs a fifth as much is a model you can afford to experiment with: five variations instead of one, a second attempt at a hand that came out wrong, an extra pass at a background. Iteration count is a quality lever that never shows up in a benchmark, and it is the main reason cheap models win in practice more often than their scores suggest.

This is the same pattern showing up in the API market data. In Vercel's October gateway index, a single cheap open-weight model handled nearly half of September's tokens while accounting for under 3% of spend. Volume follows price, and quality differences of a few percent rarely survive contact with a budget.

Where the vendor numbers still matter

None of this means the official benchmarks are worthless. Two of the claims are structurally useful.

The parameter-efficiency claim, better scores with 38% fewer parameters, is falsifiable and directionally important, because it predicts deployment, not just quality. A 5.6B image backbone plus a 4B text encoder and an optional 9B prompt upsampler is not the same thing as a 5.6B system, and the fact that the small model needs help from a language model to write good prompts is itself a finding. Short prompts produce mediocre results; that is a deployment constraint disguised as a footnote.

The latency claim is useful too, even with a methodology that favors the home team. Four to seven seconds at one megapixel for a locally runnable model is the number that decides whether an interactive tool feels interactive. If you are building a UI where someone drags a box and waits, that is the number that matters, and it is not on any Elo chart.

How to read the next launch

Four questions, in this order, and you can skip most of the chart.

What was the test set? If it was built by the vendor, treat it as a demo, not a measurement.

Was it blind and third-party? Two hundred real tasks with hidden model identity is worth more than ten thousand vendor comparisons.

What did it fail at? The failure mode is the product decision. "Worse on pose changes" tells you where not to use it.

What did it cost? Normalize every score by price before you compare, and remember that the cost of a retry is part of the cost of a model.

The model that wins the chart and the model that wins your pipeline are frequently not the same model, and the gap between them is usually a column nobody screenshots.

The quiet takeaway

Open-weight image models have gotten good enough that the open question is no longer whether they can compete. On a 200-task blind test, a 5.6B open model landed within three tenths of a point of a closed competitor at a fifth of the price, and the disagreements were about specific task types rather than about overall capability.

That is a mature market behaving like a mature market: no single winner, task-specific routing, and a cost column that decides more deployments than any benchmark does. The vendor charts will keep coming every month. The useful skill is knowing which of the three numbers you are actually looking at.