BlenderFusion - A generative visual synthesis framework from Google DeepMind

What is BlenderFusion?

BlenderFusion, a generative visual compositing framework from Google DeepMind, combines traditional 3D editing software (Blender) with AI models to achieve precise geometric editing and diverse visual compositing. The framework is based on three steps: first, extracting objects of interest from the source image and converting them into editable 3D elements (object-centric layering); second, performing diverse editing on the objects in Blender (Blender-based editing); and finally, seamlessly blending the edited elements using a generative compositor to generate a final, realistic image (generative compositing). BlenderFusion excels in complex visual compositing tasks, enabling flexible, decoupled, and 3D-aware manipulation of objects, cameras, and backgrounds.

Main functions of BlenderFusion

• Precise 3D geometry controlIt enables precise 3D editing of objects based on Blender, including transformations such as position, rotation, and scaling, as well as modifications to attributes such as color, material, and shape.
• Flexible camera controlIt supports adjusting the camera view independently of object operations, enabling complex viewpoint changes.
• Complex scene compositingIt seamlessly blends edited objects and backgrounds to generate realistic final images, supporting multi-object operations and complex scene editing.
• Decoupled object and camera controlThis allows for highly decoupled control, enabling the manipulation of objects while keeping the camera stationary, or the adjustment of the camera while keeping the object stationary.
• Generalization abilityIt supports applications in unseen scenarios and objects, and supports editing tasks ranging from simple to complex, including progressive multi-step editing.

The technical principles of BlenderFusion

• Object-centric layeringObjects are extracted from the input image using a visual base model (such as SAM2 for segmentation and Depth Pro for depth estimation) and converted into editable 3D elements. Optionally, a complete 3D mesh can be generated using an image-to-3D model (such as Rodin or Hunyuan3D) and aligned with a 2.5D surface mesh for more flexible editing during testing.
• Blender-grounded editingImport the 3D objects obtained from the layering process into Blender, and leverage Blender's powerful features for diverse editing operations, including basic object transformations, attribute modification, and non-rigid deformation. It supports camera control and background replacement, providing precise 3D control signals for the compositing process.
• Generative CompositingThe diffusion-based generative compositor blends the Blender rendering results with the background to generate a final realistic image. Based on a two-stream architecture, the generative compositor simultaneously processes information from the original scene (before editing) and the target scene (after editing), fusing the information from both using a cross-view attention mechanism. Two training strategies—source masking and simulated object jittering—enhance the model's flexibility and decoupling capabilities in complex editing tasks.

BlenderFusion project address

• Project official websitehttps://blenderfusion.github.io/
• arXiv technical paperhttps://arxiv.org/pdf/2506.17450

Application scenarios of BlenderFusion

• Film and television productionUsed in the production of visual effects (VFX) for movies and TV series, it adds virtual objects, adjusts scene layouts, changes backgrounds, and creates realistic composite scenes.
• Game developmentIt helps game developers quickly design and edit game scenes, add and adjust objects in the game, change the camera perspective, and create realistic game environments.
• advertiseIt helps advertising designers create high-quality product display images that highlight product features.
• Architectural DesignArchitects and interior designers can visualize interior designs, add and adjust furniture, decorations, etc., and generate realistic interior renderings.
• Artistic CreationArtists use 3D editing and generative compositing capabilities to create unique digital artworks, achieving creative visualization.