Interactive Video Generation Finally Has an Exam. It Refuses to Give One Total Score.

StreamEval appeared on October 15 and describes itself as the first benchmark for interactive streaming video generation. The sessions it evaluates run from 15 to 300 seconds, and the unit being measured is a session: how a system behaves when a person keeps talking to it while it generates.
That distinction is the reason the benchmark exists, and it is worth stating plainly. Almost every video model comparison you have read this year scored a five or ten second output against a prompt. Streaming systems do something else: they accept new instructions mid-generation, keep the scene coherent across those instructions, and hold an identity over minutes rather than frames.
Five tasks, because five different things get called video
The benchmark splits into tasks that fail for different reasons.
Digital-human dialogue gives the system a portrait and a persona, plus the user's speech or text turns, and scores quality, audio-video sync, stability, dialogue and identity.
Avatar performance gives a portrait and a script to perform with live directions, and scores quality, sync, stability, following and identity. The difference from the first task is that the user is directing a performance rather than holding a conversation.
World generation takes an opening prompt or image, then live prompts and camera controls, and scores quality, stability, following and continuity, plus a reference metric when the input was an image.
Instruction editing feeds a live source video and text edit instructions, scoring quality, stability, following, edit success and preservation.
Reference editing feeds a live source video plus reference images to apply, patch, remove or reset, and swaps the dialogue metric for a reference metric.
Metrics are scored 0 to 100. A task's overall number is the equal-weight mean of the metrics that apply to it. And then the sentence that makes the whole design legible: tasks are never merged into one total.

Why the missing leaderboard column is the point
Every benchmark that produces a single number eventually gets gamed toward that number. A video benchmark with one total would push teams to trade stability for quality, or continuity for adherence, in whatever ratio the weights happened to favour. Nobody would have to state the trade. It would just appear in the scores.
By refusing the merge, StreamEval forces the trade into the open. A system that nails dialogue and drifts after forty seconds shows up as a dialogue score and a stability score, and a reader has to decide which one they need.
This is a methodological choice with a cost. A five-column table is harder to screenshot than a ranked list, and harder to write a launch post around. The repository takes that cost deliberately, and the paper (arXiv 2610.28402) is where the reasoning lives.
Fully open, down to the hashes
The second design decision is that everything ships. Benchmark data, evaluation code, protocols, scoring rules, all public. Datasets are mirrored to Hugging Face and ModelScope, which matters if you have ever tried to download a benchmark from behind a region block.
The full set lives in `benchmark/v1/full`. Each condition-set folder contains three files: `cases.jsonl` with the input timelines, `oracles.jsonl` with expected outcomes that are for the judge only, and `manifest.json` carrying media paths and SHA-256 hashes.
Those three files describe a benchmark that can be re-run and audited rather than trusted. The oracle separation means the judge sees the answer key and the model does not. The hashes mean the media cannot be swapped between runs. Neither is novel, and both are routinely missing from published evaluations.
A judge model is in the loop, and that is a design choice
Most of the ten metrics are not measured, they are judged. Audio-video sync can be computed; "quality" and "following" cannot, so a model reads the output and scores it.
That is now standard practice and it comes with a known failure: judge models carry preferences, and systems tuned against the same judge learn those preferences rather than the quality they were meant to proxy. The `oracles.jsonl` separation is the mitigation. The judge sees the expected outcome, so the question it answers is closer to "does this match the intent" than "is this good."
The cost is that the benchmark measures judged compliance rather than audience response. For a research harness that is the right trade, because compliance is checkable and audience response is not. For a product team choosing a vendor, it means a high score on following is evidence that the system does what it was told, and nothing about whether what it was told was worth doing.
Running it costs nothing to start
The quick start is deliberately set up so that the first run needs no API key, no GPU and no benchmark download. Python 3.10 or later on Linux or macOS:
git clone https://github.com/StreamEval/StreamEval.git streameval && cd streameval
python -m venv .venv && source .venv/bin/activate
python -m pip install -e ".[runtime,judge,dev]"
python -m streameval.benchmark.synthetic --out .smoke
That last line generates its own media. Then:
streameval run --model streameval.models.mock:MockModel --track instruction_edit \
--case sa.v1.901.instruction_edit --benchmark .smoke/set --data .smoke/data --out .smoke/attempt
A mock model on synthetic media produces a pipeline that works end to end, which is the correct way to ship an evaluation harness. You find out that your judge is misconfigured before you have spent anything, and you find out on data that is reproducible.
The monthly update is a statement about the field
The README says the leaderboard will evaluate new and updated models monthly.
For a benchmark on a stable task, that cadence would be odd. For one on interactive video, it is the only honest option. Between OpenRouter's video leaderboard last week and the week before, the composition changed: Avatar IV entered the top ten, mischief 1 left it, and Alibaba's share of video requests fell by more than half. New systems arrive faster than a yearly benchmark can absorb.
The cost of monthly updates is comparability. If the harness changes, old scores mean less. If it does not, new model classes can be measured against a protocol that predates them. The repository's answer is versioned benchmark folders with hashes, which lets you keep evaluating the same v1 set while adding new tasks beside it.
Where the numbers still do not tell you much
There is a gap between what StreamEval measures and what a production team needs, and it is worth naming.
The benchmark scores identity consistency, sync, stability and following. It does not score whether the output is worth watching. A system can hold a face for five minutes, obey every instruction, and produce something nobody would publish. Quality is in the metric list, but as one of five, and "quality" in a video benchmark still means a judge model's opinion.
The other gap is cost. A 300 second session is a different economic object from a 5 second clip, and the benchmark is silent on latency and price per second. For interactive systems those are usually the constraint that decides whether a product ships.
Neither gap is a flaw in the work. A first benchmark should measure the thing that had no measurement, and interactive streaming video had none. The interesting question is what the leaderboard looks like after three monthly updates, when the same five tasks have been applied to a field that has already moved twice.
Related articles
Chinese text in video always turns to mush. Dreamina handed the typesetting to code
Dreamina launched Motion Studio on November 10. It no longer generates frames; the model writes code that lays out text, shapes and pacing. Chinese titles stop blurring, but whether the content is right is still on you.
A community spreadsheet of taps that turns off the ads inside Chinese Android builds
A no-root tool that switches off the ads Chinese phone makers ship inside the system UI. Its method is a hand-written tap script per phone model, maintained by volunteers every time a ROM moves. It works, and the maintenance is the point.
GPT Image 2.6 held the token price and raised the bill
GPT Image 2.6 kept every token rate from 2.5 and still costs more per image, because more of the work now happens in output tokens. Eleven days into the ramp, the pricing structure says more than the leaderboard does.
Someone rebuilt IDM from the outside and wrote down what they did not touch
A Rust reimplementation of Internet Download Manager claims 5x the speed and 86 languages. The more interesting document is the paragraph explaining how it was built without copying anything.