Bernini - ByteDance's open-source unified video generation and editing framework

What is Bernini?

Bernini is an open-source unified video generation and editing framework from ByteDance, employing a two-stage decoupled architecture of MLLM semantic planning and DiT visual rendering. A large multimodal model understands instructions and plans semantic sketches, while the Diffusion Transformer completes high-quality video generation. The model supports tasks such as text/image video generation, environment/viewpoint/action editing, and multi-reference guidance insertion. The inference code and weights are fully open-source.

Bernini's main functions

• Text/Image to Video GenerationIt supports generating videos directly from text prompts or reference images, and can handle single images, multi-element combination images, and multi-angle reference images.
• Video editingSupports environment editing, perspective editing, focus editing, and action editing.
• Reference Guided Editing: Precisely integrate specified objects, textures, or styles into the video using subject reference images, material reference images, or style reference images.
• Image/video embedding: Fill the screen, signboard, or other areas of the image with posters or videos, and follow the camera movement while maintaining perspective and time stability.
• Multi-element combinationCombine multiple unrelated reference images into a single video character, or generate continuous panning shots based on keyframes of the same scene.

Bernini's technical principles

• Two-phase decoupling architectureThe first stage uses a Multimodal Large Language Model (MLLM) as the Semantic Planner, predicting the target semantic representation in the ViT embedding space. It is responsible for understanding text instructions, source video, and reference materials, and planning the target content. The second stage uses a Diffusion Transformer (DiT) as the Renderer, performing stream matching denoising in the VAE latent space, transforming the semantic planning into high-quality video frames. In the editing task, VAE features from the source video are also injected to preserve details in non-edited areas.
• Segment-Aware 3D RoPE (SA-3D RoPE)When multiple reference images, source videos, and target outputs are concatenated into a unified sequence, tokens from different segments may share the same spatiotemporal coordinates. By assigning a segment index to each visual segment and integrating it into rotational position encoding, the model can distinguish tokens from different sources while preserving the original spatiotemporal modeling characteristics.
• Chain-of-Thought Reasoning and Three-Stage TrainingPlanner performs thought chain reasoning in the latent space, enhancing semantic fidelity in complex editing scenarios. Training consists of three stages: Planner pre-training, Renderer pre-training, and light joint training of Planner and Renderer, retaining the advantages of pre-training while maintaining high efficiency.

How to use Bernini

• Cloning repositoryClone the Bernini project from GitHub to your local machine.
• Install dependenciesRun pip to install the dependency packages listed in requirements.txt.
• Download ModelDownload Bernini-R-Diffusers weights via HuggingFace.
• Configure environmentEnsure you are using Python 3.11.2 and CUDA 12.4+, and a Hopper architecture GPU is recommended.
• Run to generateUse torchrun to execute multi-GPU inference scripts, specifying configuration files and test cases.
• Enhanced configuration promptsConfigure OpenAI-compatible API endpoints to improve generation quality.

Bernini's core advantages

• Understand before generatingMLLM is responsible for semantic planning, and DiT is responsible for visual rendering, solving the pain point of traditional video models "not understanding human language".
• Unified frameworkA single architecture covers multiple tasks such as generation, editing, and reference guidance, without the need to switch models.
• Multiple reference inputsSupports simultaneous input of text, images, and videos, and avoids spatiotemporal confusion through SA-3D RoPE.
• Details preservedInject VAE features from the source video during editing to preserve details in non-edited areas and avoid "changing one thing and everything is changed".
• Fully open sourceThe inference code and weights have been made available, lowering the barrier to research and development.
• Strong generalization abilityIt still performs well on instructions that are not explicitly covered in training (such as causal reasoning and action changes).

Bernini's project address

• Project official websitehttps://bernini-ai.github.io
• GitHub repositoryhttps://github.com/bytedance/Bernini
• HuggingFace model libraryhttps://huggingface.co/ByteDance/Bernini
• arXiv technical paper: https://arxiv.org/pdf/2605.22344

Bernini's Competitive Product Comparison

Bernini's application scenarios

• Advertising CreativityThe model can quickly generate product demonstration videos, and supports product image insertion and style adjustment.
• E-commerce showcaseGenerate highly consistent dynamic display videos based on multi-angle reference images of the product.
• Film and television preview: Generate continuous shots through keyframes to assist in virtual roaming and scene construction.
• Secondary creationIt supports style transfer, weather changes, and motion editing, lowering the barrier to video re-creation.
• Virtual shootingThis allows for the precise insertion of posters or videos into the screen or signage, achieving a seamless blend of virtual and real elements.