AutoMV - An open-source AI music and video generation system developed by M-A-P in collaboration with Beijing University of Posts and Telecommunications and other institutions.

What is AutoMV?

AutoMV is an automated multi-agent system developed by M-A-P, Beijing University of Posts and Telecommunications, and the NJU-LINK Laboratory of Nanjing University, among others. It supports the conversion of songs into coherent, rhythmically aligned music videos. The system extracts the song's beat, structure, and lyrics through music information retrieval. Screenwriter and director agents collaborate to generate scripts and shooting plans, while generation and verification modules ensure video production quality. AutoMV achieves end-to-end generation from audio and lyrics to a complete music video, characterized by high efficiency, low cost, and high quality, providing a novel solution for music video creation.

AutoMV's main functions

• Musical Understanding and AnalysisExtract the song's beat, structure (such as verses and choruses), and lyrics timestamps to provide musical clues for video generation.
• Screenwriting: Generate storyboards based on music and lyrics to ensure that the video content is consistent with the semantics of the lyrics.
• Character and Scene PlanningIt supports creating a character library and designing character images to maintain the continuity and consistency of characters in videos.
• Video generationSupports the generation of narrative shots and performance shots (such as singing and dancing) that are aligned with the rhythm of the music and the lyrics.
• Quality verification and iterationAutomatically checks the audio-visual synchronization, character consistency, and action rationality of the generated video, and reshoots as needed until the quality requirements are met.

AutoMV's technical principles

• Music Information Retrieval (MIR): Use professional tools (such as SongFormer, Whisper, etc.) to analyze the music and extract information such as beat, structure, and lyrics to provide a foundation for subsequent generation.
• Multi-agent collaboration:
  • Screenwriter AI agentResponsible for generating storyboards based on music and lyrics, and designing storylines and scenes.
  • Director (AI agent)Generate specific shooting instructions based on the script, including shot type, character actions, and camera movement.
  • Generation Hub: Call different video generation models (such as diffusion model, lip-sync model) to generate video clips.
  • VerifierCheck the audio-visual synchronization and character consistency of the generated video, and require a reshoot if problems are found.
• Iterative optimizationBy verifying the feedback mechanism of the intelligent agent, the generated video is continuously optimized to ensure that the final output meets high-quality standards.

AutoMV's project address

• Project official websitehttps://m-a-p.ai/AutoMV/
• GitHub repositoryhttps://github.com/multimodal-art-projection/AutoMV
• arXiv technical paper: https://arxiv.org/pdf/2512.12196

AutoMV Application Scenarios

• Film and television productionAutoMV is used by film, television and web series production teams to quickly generate music-related clips for the visual presentation of trailers, opening themes or incidental music, saving time and costs.
• Advertising and MarketingAdvertising companies can generate video content that matches the music in their advertisements, quickly creating engaging video ads that enhance their appeal and reach.
• short video platformContent creators generate short videos that match music and use them on social media platforms (such as TikTok, Kuaishou, Bilibili, etc.) to attract viewers and increase fan interaction.
• Music and Video Production EducationSchools and educational institutions use AutoMV as a teaching tool to help students understand the relationship between music and video, and learn the creation process and techniques of music videos.
• Multimodal learningThrough music videos generated by AutoMV, students can better understand the meaning and emotional expression of lyrics, enhancing their multimodal learning experience with music and literature.