Boximator - A framework from ByteDance for controlling object motion in video generation.

What is a Boximator?

Boximator is a video compositing technology developed by ByteDance's research team. It aims to generate rich and controllable motion to enhance the quality and controllability of video compositing. This technology achieves fine-grained control over the position, shape, or motion path of objects in a video by introducing two types of constraint boxes (hard boxes and soft boxes).

• Official project homepage:https://boximator.github.io/
• Arxiv research paper:https://arxiv.org/abs/2402.01566

How Boximator works

Boximator works based on a video diffusion model, which enhances the precision and controllability of video synthesis by introducing a novel control mechanism. The key steps in Boximator's operation are as follows:

1. Object selection and box definition:
   • Users use hard boxes in the start frame or conditional frame of a video to select and precisely position objects. These hard boxes define the precise boundaries of the objects.
   • For scenarios requiring more flexible control, users can define soft boxes, which provide a general area where an object must exist, allowing the object to move freely within that area.
2. Associating object ID with box:
   • Boximator assigns a unique object ID to each object, represented in the RGB color space, giving each object's bounding box a unique "color." This allows the model to track and control the same object across frames.
3. Video diffusion model integration:
   • Boximator is a plugin that integrates with existing video diffusion models such as PixelDance and ModelScope. During training, the weights of the base model are frozen to preserve its pre-trained knowledge, while only the newly added control modules are trained.
4. Self-tracking technology:
   • To simplify the process of the model learning bounding box-object associations, Boximator introduces a self-tracking technique. During the training phase, the model is trained to generate colored bounding boxes, with the color of these boxes corresponding to the object IDs. The model needs to generate the correct boxes in each frame and align them with the constraints of Boximator.
5. Multi-stage training process:
   • Boximator training is divided into three stages. The first stage uses hard bounding boxes to help the model develop a basic understanding of coordinates and IDs. The second stage introduces soft bounding boxes, which increase the difficulty of training by randomly expanding the hard bounding boxes. The third stage continues to use soft bounding boxes, but instead of generating visible bounding boxes, it allows this relationship to be preserved within the model.
6. Reasoning stage:
   • During the inference (video generation) phase, Boximator inserts soft boxes in frames outside the user-defined bounding boxes. These soft boxes are generated through linear interpolation and relaxation processing, ensuring that objects roughly follow the expected motion trajectory while giving the model enough flexibility to introduce variations.
7. Motion control and quality assessment:
   • Boximator evaluates the accuracy of motion control using an average precision (AP) score, which involves comparing the consistency of detected bounding boxes in the generated video with the ground truth bounding boxes.
   • Video quality is measured by the Fréchet Video Distance (FVD) score and the CLIP Similarity Score (CLIPSIM).

Through these steps, Boximator enables precise control over object motion during video compositing while maintaining high video quality and realism.

Application scenarios of Boximator

• Film and television production:In the post-production of movies and TV series, Boximator can be used to generate or modify scenes, such as adding or removing characters, adjusting action sequences, or creating complex special effects without the need for expensive on-set shooting.
• Game development:Game developers can use Boximator to create dynamic game scenes and character animations, which can greatly save development time and costs, especially when highly customized or rapidly iterating content is required.
• VR and AR content creation:In the fields of virtual reality (VR) and augmented reality (AR), Boximator can be used to generate realistic virtual environments and interactive objects, providing users with an immersive experience.