SeedRealtime - ByteDance's native full-duplex audio and video model

What is SeedRealtime?

SeedRealtime is a native full-duplex audio and video model developed by ByteDance's Seed team. Based on a unified architecture, it natively integrates audio, video, and text to achieve real-time, multi-modal interaction, enabling users to watch, listen, and speak simultaneously. The model boasts three core breakthroughs: joint audio and video understanding, proactive interaction, and smooth dialogue rhythm. End-to-end evaluation shows that dialogue rhythm issues are reduced by half compared to cascaded models. It is currently fully deployed on the Doubao App, becoming the industry's first large-scale deployed full-duplex audio and video AI product.

The main functions of SeedRealtime

• Audio and video joint understanding: It natively and deeply integrates sound, visuals, and temporal information, and can combine visual scenes to resolve homonym ambiguity, accurately understand temporal references, and achieve integrated perception of what is seen, heard, and spoken.
• Proactive interaction capabilities: It possesses continuous environmental awareness and proactive expression capabilities, and can proactively remind users when the screen status changes, and call upon tools to integrate expression, upgrading passive response to proactive collaboration.
• Smooth interactive rhythm: It can sense the user's conversation status and communication rhythm in real time, and naturally respond, pause, and reply at appropriate times. It has strong anti-interference ability and can distinguish between casual conversations and background noise.
• Multi-person scene recognition: In noisy environments with many people, the system can simultaneously recognize faces, distinguish sound sources, and understand content, continuously matching each person's voice with their identity.

The technical principles of SeedRealtime

• End-to-end unified modeling: An end-to-end unified audio and video modeling framework is adopted to integrate perception, understanding, decision-making and expression into the same model and perform them synchronously, avoiding the multi-stage information loss and latency accumulation in the ASR→VLM→TTS system.
• Native full-duplex interaction: Without relying on external VAD rules to determine the turn, the model itself continuously makes decisions on the dialogue state and timing based on multimodal information flow, truly realizing a "speaking and listening" rather than a question-and-answer half-duplex mode.
• Audio and video timing alignment: By unifying the modeling of sound, visuals, and temporal information, and combining the current scene, gestures, gaze, and historical actions to understand user intent, cross-modal disambiguation and referential resolution are achieved.
• Low-latency optimization for engineering: By segmenting audio and video input and streaming generation, combined with efficient quantization and inference optimization, the end-to-end latency of "hearing-understanding-responding" is continuously compressed.

How to use SeedRealtime

• Update Doubao AppUpdate the Doubao App to the latest version.
• Enter voice callAfter opening the app, click the "Make a Call" button in any chat window.
• Switch video modeAfter entering the call interface, enable camera and microphone permissions and switch to video call.
• Start real-time interactionThe system can speak naturally while displaying images, and the model can simultaneously perceive the audio and video streams and respond in real time.
• Use active observationIt can issue continuous observation commands, and the model will actively remind you when the target appears or the scene changes.

SeedRealtime's core advantages

• End-to-end unified architecture: It adopts a single model to natively fuse audio, video and text, eliminating the information loss and latency caused by the serial connection of multiple modules in a cascaded system.
• Native full-duplex interaction: Without relying on external VAD rules, the model autonomously judges the timing of dialogue based on multimodal information flow, truly achieving "speaking and listening at the same time" rather than a question-and-answer session.
• Audio and video joint understanding: By deeply integrating sound, visuals, and temporal information, it can resolve homonym ambiguity by combining visual scenes and accurately interpret cross-modal references such as "this" and "there".
• Active environmental perception: It has continuous visual observation capabilities and can proactively alert users when the screen status changes or a target appears, upgrading the interaction from passive response to proactive collaboration.
• Smooth, interference-free rhythm: It can perceive the user's conversation status in real time and accurately distinguish between casual conversation and formal questions in noisy environments, reducing lag and false triggers by half compared to the cascading model.

SeedRealtime's project address

• Project official website：https://seed.bytedance.com/en/seedrealtime

Comparison of SeedRealtime's similar products

Dimension	SeedRealtime	Gemini Live
Developer	ByteDance Seed Team	Google DeepMind
Architecture	End-to-end unified audio and video modeling, integrating perception, understanding, decision-making, and expression.	Native multimodal, supporting simultaneous input of audio, video, image, and text.
Full-duplex	Native full-duplex, does not rely on external VADs, and autonomously determines the timing of dialogue.	Full-duplex real-time two-way voice communication, supporting active audio output.
Chinese optimization	Deep optimization, enhanced homonym disambiguation and Chinese contextual understanding	General multilingual support (200+ languages), Chinese not specifically optimized.
Active interaction	Continuous environmental awareness, proactively alerting users and invoking tools when the screen changes.	Supports active participation and tool usage, but offers limited visual proactivity.
Anti-interference	It is powerful and can accurately distinguish between casual conversation, background noise, and formal questions.	Built-in noise reduction; performs moderately well in complex and noisy environments.
Visual understanding	Audio and video temporal joint modeling accurately analyzes gestures, references, and dynamic scenes.	Native video stream processing, supporting real-time camera image analysis
Latency performance	End-to-end low latency reduces dialogue pacing issues by half compared to cascaded models.	The initial audio latency is approximately 200-320ms, with industry-leading response speed.
Ecological integration	Deep integration with Doubao App	Google Workspace, Search, Meet, and Android system-level integration

Application scenarios of SeedRealtime

• Smart Guide: Continuously observe the exhibits in the museum, and proactively explain the historical background and craftsmanship details when the target appears.
• Foreign language tutoring: By combining real-time audio correction with visual examples and sentence construction, it keeps the target learner focused even in noisy environments.
• Equipment Operation Instructions: When operating complex equipment, it can correct errors in real time based on changes in visual state and provide adjustment suggestions.
• Travel Information Assistant: Recognize flight information on large screens in noisy airport environments, answer arrival time questions, and provide route guidance.
• Child education and guidance: When accompanying your child to study, observe the screen content in real time and correct pronunciation without being disturbed by background noise.