ByteDance's SeedRealtime takes two frames a second and decides when to speak

Most "AI can see and talk" products are a pipeline wearing a trench coat. Speech recognition turns your voice into text, a vision model describes the frame, a language model decides what to say, and a speech synthesiser says it. Four models, three handoffs, and a latency budget that gets spent in the seams.
ByteDance's Doubao-Seed-Realtime-1.0 is an attempt to remove the seams. Volcengine describes it as an end-to-end realtime audio and video model built on Doubao's realtime voice 3.0, which already ran full duplex. The new part is that the client can push camera or screen frames into the same session, at no more than two frames per second, and the model talks about what it sees.
What two frames per second is for
At two frames per second you get scene awareness. Video understanding needs a much higher rate.
At that rate the model can tell that a kettle is on the stove, that a form has three empty fields, that the person on screen is holding a different object than a minute ago. It cannot see a gesture finish, follow a fast pan, or judge whether a fall looked real. Anyone who has watched a 2 FPS timelapse knows the feeling: the world is legible and nothing moves.
Volcengine's framing is deliberate. Video frames and images enter as context for a conversation, not as a video-editing input. The docs also mention an image search capability, so a frame can trigger a visual lookup rather than only a description.
The honest reading is that this replaces a "describe my screen" prompt with a live narrator, and the ceiling is set by the sample rate. For a support call where someone points a phone at a router, 2 FPS is plenty. For anything with motion, it is a slideshow.
There is a practical consequence for how you build with it. Two frames a second means roughly 500 ms between frames, so the model is never more than half a second behind the room, which is fast enough to comment on a change and slow enough to miss an action. Prompts should ask about state ("is the pan on the heat?") rather than events ("did they pick it up?"). Teams that design for state get useful answers; teams that design for events get confident guesses.
Speaking without being asked
The feature that changes the product is proactive speech. The model does not wait for a question. It keeps reading the scene and the conversation, and when it has something to add it says it: a reminder that the object moved, a warning that the page changed, a suggestion before you ask.
That is a different interaction contract from a voice assistant. A wake word implies the human starts every turn. A model that can open a turn needs a model of when not to. The docs give developers the controls for that: dialog context, additional parameters, location fields, and the ability to send function-call results back into the session.
Full duplex with interruption is the other half. You can talk over it and the system stops its own output and returns to listening. Pipe that through a phone speaker and it feels less like a command line and more like a call.
The plumbing developers actually touch
The endpoint is a WebSocket at wss://openspeech.bytedance.com/api/v3/realtime. Sessions are event-based on both directions: session configuration upstream, audio and text events downstream, plus events for client interruption, function-call results, and session end. Demo clients ship for Go 1.24, Python 3.7 and the web, which tells you the target audience is application developers rather than researchers.
The configuration surface is large and worth reading before you promise a feature. A few examples:
- strict_audit defaults to true, and audit_response lets you define what the model says when a user query trips the safety filter. A custom refusal line instead of silence is a real product decision. - enable_volc_websearch turns on built-in web search, with two service types. The custom one is about 350 ms faster and returns more results per request; the global one covers more of the web and supports adjustable summary length and images from source pages. - visual_search_config wires the frame input to image search, currently only through the global service. - TTS options include filtering bracketed text so stage directions are not read aloud, dialect selection on four specific voices, and loudness normalisation. - aigc_metadata can embed an implicit watermark with producer and propagator identifiers. Given the regulatory direction in China, that is not optional decoration.
The part that is a compliance product
Two settings stand out because they are policy expressed as API parameters.
strict_audit on by default means the safe configuration is the one you get by doing nothing, and the audit response field means a blocked query still produces a usable turn instead of dead air. For a model that speaks proactively, dead air after a refusal is worse than a scripted "I can't help with that".
The AIGC metadata block is the other one. Producer name, production ID, propagator name, propagation ID: the fields exist so generated audio can be traced. In a market where AI content labelling is being written into regulation, shipping the traceability hook in the session config is cheaper than retrofitting it.
Where this will be judged
Five questions.
Latency, which Volcengine does not put a headline number on. End-to-end models win on seam removal, and they still have to beat 91 ms first-audio figures that specialised speech vendors are now quoting.
Always-on cameras. A model that watches your screen and speaks up needs a privacy story beyond "it is your own device". Frame capture at 2 FPS is still frame capture.
Proactive speech fatigue. Every assistant that speaks first gets muted within a week if it is wrong twice. The quality bar for "should I say this" is higher than for "answer this".
Cost per minute, especially for video frames. Two frames a second is 120 frames a minute, and someone pays for them.
Ecosystem gravity. The docs, demos and console are Chinese-first, and the product lives inside Volcengine. That is fine for teams already there and a real integration cost for everyone else.
What it means for the pipeline
If you have been stitching speech, vision and language models together for a live product, this class of model removes the seams and takes away some control. You lose the ability to swap the vision model, and you gain a session where the model has already seen the frame when the user starts talking.
The 2 FPS limit is the tell. ByteDance is not selling video understanding here; it is selling presence, cheap enough to run continuously, opinionated enough to speak first. Whether that is useful depends on whether your product has a scene worth watching that slowly.
Related articles
Six Amateurs, 1,113 Episodes, a Thousand Yuan of Compute an Episode: How an AI Drama Got Hand-Built
The first ten seasons of Armed Into Song run to roughly a thousand episodes and have passed ten billion cumulative views, with six core creators and not one of them film-school trained. Compute runs 1,000–2,000 yuan an episode; a normal shot takes three rolls, a battle twenty-plus, and anything that looks flat is graded a sixty and thrown out.
One server, one AI streamer: inside the code that runs a nightly livestream
open-yoki is the production code behind an AI host who streams nightly on Binance Square, with the persona, voice and money features stripped out. Headless Chrome draws the frame, ffmpeg pushes RTMP, a folder of Markdown decides what she feels and what she posts.
Pinky lets an agent drive your phone, and holds the Send button until you click Allow
Pinky connects any MCP agent to a real iPhone or Android phone and enforces approval in its own gateway rather than in a prompt. Banking apps are invisible, password fields are off limits, and on Android 13 the agent gets a hidden second screen.
Someone put the world's law into git, so you can blame a statute
global-law stores each country's statutes as a git repository where every commit is one law taking effect on a known date. You can run log, blame, show and diff on a legal provision, ask what was in force in every country on a given day, and hand the whole thing to an assistant over MCP.