AB
AB
AiBoss

All Categories

project

MOVA - An end-to-end audio and video model open-sourced by Innovation Academy and Moment Intelligence.

MOVA (MOSS Video and Audio) is China's first high-performance open-source end-to-end audio and video generation model, jointly launched by the OpenMOSS team at Shanghai Innovation Academy and MOSI. The model breaks through the limitations of traditional video 'mute'...

What is MOVA?

MOVA (MOSS Video and Audio) is China's first high-performance open-source end-to-end audio and video generation model, jointly launched by the OpenMOSS team at Shanghai Innovation Academy and MOSI. Breaking through the traditional limitation of "silent" video, the model adopts a heterogeneous dual-tower architecture and bidirectional bridging modules to achieve native cross-modal interaction. The model boasts 32 billion parameters (MoE architecture, 18 billion inference activations) and can simultaneously generate up to 8 seconds of 720p resolution video and accompanying audio, demonstrating outstanding performance in cinematic lip-sync and environmental sound effect fit.

MOVA's main functions

  • End-to-end audio and video generationThe model can output video and accompanying audio simultaneously, eliminating "dumb video".
  • Dual-mode driven generationSupports image + text or plain text input, allowing for flexible control over the generated content.
  • Cinematic-level lip-syncingThe model can accurately match the mouth shape and voice of a person when they speak, and supports multi-person dialogue in Chinese and English.
  • Intelligent ambient sound effectsIt automatically synthesizes and matches background music, action sounds, and ambient sounds based on the scene.
  • Video text renderingThe model can generate clear and readable dynamic text content at a specified location on the screen.
  • High resolution outputThe model supports the generation of audiovisual clips with a maximum resolution of 720p and a duration of 8 seconds.

MOVA's technical principles

  • Heterogeneous dual-tower architectureThe model uses a 14B video diffusion model and a 1.3B audio diffusion model to process visual and auditory information respectively. A bidirectional bridging module is used to achieve deep cross-attention fusion of the two hidden states, so that the sound rhythm can be perceived throughout the image generation.
  • Cross-modal time alignmentThe sampling densities of video and audio differ greatly. The Aligned ROPE mechanism, through precise scaling mapping, unifies the tokens of the two modalities to the same physical time coordinate system, fundamentally eliminating the problem of audio-visual asynchrony.
  • Progressive training strategyThe model is trained in three stages from coarse to fine. First, a low resolution of 360p is used to allow the randomly initialized bridging module to quickly learn audio and video alignment, gradually improving the alignment stability. Finally, it is extended to a high resolution of 720p for image quality refinement.
  • Dual CFG InferenceAddressing the characteristic of audio and video co-generation having two control sources—text commands and modal bridging—it supports independent adjustment of the guidance weights of both, preserving image quality in general scenarios and enhancing lip-sync accuracy in dialogue scenarios.

MOVA's project address

  • Project official websitehttps://mosi.cn/models/mova
  • GitHub repositoryhttps://github.com/OpenMOSS/MOVA
  • HuggingFace model libraryhttps://huggingface.co/collections/OpenMOSS-Team/mova

MOVA application scenarios

  • Film and television productionQuickly generate storyboard previews and voiceover demos, reducing pre-production costs and accelerating creative validation.
  • Short video creationIt provides creators with high-quality story materials with sound effects, improving production efficiency and enriching content formats.
  • Game developmentAutomatically generates cutscenes and character dialogues, achieving an immersive experience with synchronized audio and visuals, and shortening the development cycle.
  • Education and TrainingIt produces multilingual, lip-syncing instructional videos with global content adaptation capabilities, enhancing learning outcomes.
  • E-commerce marketingProduce product demonstration videos with narration and background music to accelerate the iteration of marketing content and enhance conversion capabilities.
Back