Kaleido - An open-source multi-agent video generation framework from Zhipu AI

What is Kaleido?

Kaleido is an open-source multi-agent video generation framework from Zhipu AI, which solves the problem of subject consistency and background decoupling in multi-agent video generation. Through an innovative data construction pipeline and the R-RoPE (Reference Rotary Positional Encoding) mechanism, the framework effectively separates subject and background information, maintaining subject feature consistency in multi-agent scenarios. Kaleido achieves state-of-the-art (SOTA) performance in multi-agent video generation, providing a powerful open-source solution for the video generation field and helping developers create high-quality multi-agent videos.

Kaleido's main functions

• Multi-subject consistency generationThe frame can maintain visual consistency among multiple subjects (such as people, objects, etc.) in a video, and can maintain stable subject features in complex scenes and multi-person interactions.
• Background decouplingThe framework effectively separates the subject from the background information, avoiding irrelevant background details from the reference image in the generated video, and making the subject more adaptable to different backgrounds.
• High-quality video generationThrough optimized data construction and training methods, it generates high-quality, high-fidelity video content suitable for various application scenarios, such as advertising and film and television production.
• Flexible condition controlThe framework supports video generation using multiple reference images and text prompts, providing creators with more flexible creative space.

Kaleido's technical principles

• Data building pipelineKaleido employs an innovative data construction pipeline to address the issues of background entanglement and subject consistency in multi-subject video generation. The pipeline forces the model to decouple subject features during training through cross-pair data synthesis (combining subjects and backgrounds from different instances) and background inpainting techniques, avoiding the simple replication of background information from reference images. Multi-stage data augmentation (such as subject localization, segmentation, and quality filtering) ensures data diversity and high quality, enhancing the model's generative capabilities in complex scenes.
• R-RoPE (Reference Rotary Positional Encoding)To accurately distinguish between different reference images and video tokens, Kaleido introduces the R-RoPE mechanism. By providing independent rotational position encoding for the tokens of the reference images, the boundaries between different subjects and video frames are explicitly established during attention computation. R-RoPE ensures that the reference image tokens occupy unique positions in the model's spatiotemporal embedding space through spatial displacement, effectively avoiding feature confusion among multiple subjects and significantly improving the spatiotemporal consistency of multi-subject generation.
• Model architecture and condition injectionKaleido, based on a diffusion model and Diffusion Transformer architecture, combines multiple reference images with the video generation process through a simple yet efficient conditional injection strategy. The model concatenates reference image tokens and video noise representations along the sequence dimension, while using the R-RoPE mechanism to distinguish between image and video information.

Kaleido's project address

• Project official websitehttps://criliasmiller.github.io/Kaleido_Project/
• GitHub repositoryhttps://github.com/zai-org/Kaleido
• HuggingFace model libraryhttps://huggingface.co/zai-org/Kaleido-14B-S2V
• arXiv technical paper: https://arxiv.org/pdf/2510.18573

Kaleido's application scenarios

• Film and television productionKaleido can generate high-quality animated videos based on multiple reference images of a character, helping animators quickly generate initial animation materials and improve production efficiency.
• Advertising CreativityKaleido provides advertisers with flexible material generation solutions through multi-subject video generation, enabling them to quickly generate advertising videos featuring multiple products or people.
• Virtual try-onIn the e-commerce sector, Kaleido can generate videos of users trying on virtual clothing or wearing virtual accessories, enhancing the user experience.
• Virtual character interactionGenerate natural, interactive videos for virtual anchors or virtual characters, enabling them to exhibit richer dynamic effects in scenarios such as live streaming and gaming.
• Education and TrainingIt can be used to generate instructional videos, such as showcasing experimental processes or collaborative scenarios through multi-subject generation, thereby enhancing the intuitiveness and fun of teaching.