OmniShow - A video generation model developed by ByteDance in collaboration with open-source institutions such as the Chinese University of Hong Kong and the University of Hong Kong.

What is OmniShow?

OmniShow is a multimodal human-object interaction video generation model open-sourced by ByteDance in collaboration with the Chinese University of Hong Kong, Monash University, and the University of Hong Kong. As the first end-to-end framework supporting RAP2V (reference image + audio + pose), the model uniformly processes four types of input: text, image, audio, and pose. A single 12.3B parameter model can generate a 10-second high-quality video. The model employs techniques such as gated local contextual attention to achieve precise audio-video synchronization, achieving multiple state-of-the-art (SOTA) results on the HOIVG-Bench benchmark.

OmniShow's main functions

• Full-modal conditional inputAs the first end-to-end framework to fully support RAP2V (Reference+Audio+Pose-to-Video), OmniShow can simultaneously receive four modal inputs: reference images, audio, gesture signals, and text descriptions, enabling comprehensive control over human-object interactive videos.
• Unified generation of multiple tasksThe model can complete a variety of video generation tasks within a single architecture by flexibly combining different input conditions, including reference graph-based video generation (R2V), audio-driven digital human video (RA2V), pose-driven animation (RP2V), and full-modal precise control (RAP2V).
• High-quality long video synthesisOmniShow natively supports generating continuous videos up to 10 seconds long, achieving precise synchronization of lip movements, facial expressions, and body movements with audio while maintaining consistency in character appearance, resulting in industry-class visual quality output.
• Object replacement and video montageUsers can replace objects in the video while preserving the character's movements and poses, or reconstruct poses, objects, and character references from different sources, enabling flexible creative video editing and compositing.

OmniShow's technical principles

• Unified channel condition injectionBy stitching reference images and pose signals along the channel dimension, multimodal conditions are efficiently injected into the pre-trained video generation model, avoiding the destruction of the pre-training prior of the basic model by traditional modulation methods, and ensuring that the original generation quality is maintained while introducing human-object appearance and motion control.
• Gated local context attentionThe design employs masked attention combined with learnable gating vectors, enabling the model to dynamically focus on the correlation between audio features and local facial/body regions. This achieves precise time synchronization between lip movements, facial expressions, and body movements and audio signals, resolving feature conflict issues in multimodal fusion.
• Decoupling-Joint Training StrategyTo address the challenges of heterogeneous and imbalanced data in R2V and A2V tasks, we train dedicated sub-models to learn the independent mapping relationships of each modality, fuse them through weight interpolation, and fine-tune them on joint data. This effectively integrates heterogeneous datasets and improves the consistency of generation under full modality input.

How to use OmniShow

• Get open source codeClone the code repository from the official source to your local machine and configure the runtime environment including dependencies according to the README.
• Preparing multimodal inputBased on the requirements of the generated task, organize reference images (appearance of people or objects), driving audio, pose sequence files, and text descriptions.
• Perform video generationCall the model API or run the inference script, input the combined multimodal conditions, generate a high-quality human-object interaction video of up to 10 seconds and save the output.

Key information and usage requirements for OmniShow

• Project PositioningOmniShow is an industry-grade multimodal human-object interaction video generation model jointly open-sourced by ByteDance, the Chinese University of Hong Kong, Monash University, and the University of Hong Kong. It is designed specifically for complex human-object interaction scenarios.
• Technology scaleThe model adopts a 12.3B parameter architecture, which is the most parameter-efficient solution among similar models (such as HuMo-17B and Phantom-14B), significantly reducing the computational resource requirements while maintaining high performance.
• Core CompetenciesAs the first end-to-end unified framework that fully supports RAP2V (Reference+Audio+Pose-to-Video), it can simultaneously receive four modal inputs—reference image, audio signal, pose sequence, and text description—for joint inference.
• Generate qualityIt natively supports generating continuous videos up to 10 seconds long, and achieves precise synchronization of lip movements, facial expressions, body movements and audio through gated local contextual attention technology, reaching industry-level visual standards.
• PerformanceIn the self-built HOIVG-Bench benchmark test, OmniShow achieved state-of-the-art (SOTA) performance in all four tasks: R2V, RA2V, RP2V, and RAP2V, and is the only leading solution that supports full-modal input.

OmniShow's core advantages

• Full-modal unified architectureAs the industry's first end-to-end framework to fully support RAP2V (reference image + audio + pose → video), OmniShow achieves native unified processing of four conditions: text, image, audio, and pose, and can handle complex multimodal input combinations without the need to stitch together multiple dedicated models.
• Extreme parameter efficiencyThe 12.3B parameter achieves multiple state-of-the-art (SOTA) results on the HOIVG-Bench benchmark, significantly outperforming larger parameter models such as HuMo-17B (17B) and Phantom-14B (14B), while maintaining industry-class generation quality and significantly reducing inference costs.
• Single model with multi-task coverageWith its flexible input combination strategy, a single OmniShow model can handle four types of tasks: R2V (Reference Graph Generation), RA2V (Audio-Driven Digital Human), RP2V (Pose-Driven Animation), and full RAP2V (Full Modal Control), without the need to switch models for different scenarios.
• Precise audio and video synchronizationEmploying an innovative gated local contextual attention mechanism, it achieves a high degree of synchronization between lip movements, facial expressions, body movements, and audio signals through learnable gated vectors and masked attention, reaching a leading level of 8.612 in the Sync-C metric.
• Native generation of long videosUnlike extrapolation methods that rely on sliding windows, OmniShow natively supports generating continuous videos up to 10 seconds long at a time, offering significant advantages in maintaining character appearance, object style, and temporal coherence.

OmniShow project address

• Project official websitehttps://correr-zhou.github.io/OmniShow/
• GitHub repositoryhttps://github.com/Correr-Zhou/OmniShow

Comparison of OmniShow products with similar competitors

Application scenarios of OmniShow

• E-commerce product displayThe reference image replacement function allows for the rapid replacement of clothing, accessories, or merchandise while maintaining the model's movements and poses, enabling efficient product video presentation.
• Digital Human Short Video ProductionBased on RA2V (reference image + audio) capabilities, inputting a person's photo and voice can generate lip-synced speaking or singing videos, suitable for virtual anchors and social media content creation.
• Creative video mashupUsing multimodal recombination capabilities, it extracts pose, object, and person references from different sources for creative synthesis, enabling personalized content editing and secondary creation across videos.
• Interactive entertainment and games: Leveraging the Pose-Driven to Variant (RP2V) function, user motion capture is converted into animations of game characters or virtual avatars in real time, supporting a real-time interactive experience with full-body posture control.
• Advertising and Marketing Content GenerationCombined with full-modal input (RAP2V), it precisely controls the appearance, audio dialogue, and body movements of brand ambassadors, and generates promotional video materials with a unified style in batches.