VMB - A framework for enhancing multimodal music generation, jointly launched by the Chinese Academy of Sciences and several universities and institutions.

What is VMB?

VMB (Visuals Music Bridge) is a multimodal music generation framework developed by institutions such as the Institute of Information Engineering, Chinese Academy of Sciences, the School of Cyberspace Security, University of Chinese Academy of Sciences, Shanghai Artificial Intelligence Laboratory, and Shanghai Jiao Tong University. It can generate music from various input modalities, including text, images, and videos. VMB addresses the problems of data scarcity, weak cross-modal alignment, and limited controllability by constructing text and music bridges. The text bridge transforms visual input into detailed musical descriptions, while the music bridge combines broad and targeted music retrieval strategies, providing user control. VMB's explicit conditional music generation framework integrates the two bridges, significantly improving music quality, modal alignment, and custom alignment, surpassing traditional methods.

The main functions of VMB

• Multimodal Music Description ModelIt transforms visual inputs (such as images and videos) into detailed text descriptions, providing a text bridge for music generation.
• Dual-track music retrievalIt combines broad and targeted music retrieval strategies to provide music bridging, allowing users to modify text descriptions or provide reference music to control the output music.
• Explicitly Conditioned Music GenerationMusic is generated based on text bridging and music bridging, integrating the two explicit bridges into a text-to-music diffusion transformer.
• Enhanced modal alignmentImprove the alignment between the input modality and the generated music, making the music more closely match the visual and emotional content of the input.
• Improve controllabilityUsers can use text descriptions or provided music samples to guide the music generation process, enabling more precise control.

VMB's technical principles

• Text bridgingThe Multimodal Music Description Model (MMDM), built on InternVL2, converts visual input into a detailed musical description in natural language, serving as a text bridge for music generation.
• Music bridgingBased on the dual-track music retrieval module, on the one hand, it performs a broad search to identify global alignment of emotions and themes, and on the other hand, it performs a targeted search to focus on specific music attributes (such as rhythm, instruments, and genres).
• Explicit Conditional Music GenerationCombining text and music bridging, a Diffusion Transformer (DiT) is used to convert text descriptions into music. The model integrates fine-grained control for broad retrieval using a Music ControlFormer, and handles overall conditions for targeted retrieval using a Stylization Module.
• Search Enhancement Generation (RAG)This is the first time that RAG technology has been explored in music generation. It dynamically combines music knowledge and uses bridging modal gaps to improve cross-modal generation performance and increase controllability.
• Control signal fusionDuring the generation process, the hidden states of the main branch and the ControlFormer branch are combined using element-level addition to ensure structural and semantic alignment is established in the early stages of generation.
• Stylized moduleThis approach combines retrieved music with text descriptions, integrates conditional representations into noisy music based on a cross-attention mechanism, focuses on style cues in music and text data, and improves the alignment between generated music and specified attributes.

VMB project address

• GitHub repository:https://github.com/wbs2788/VMB
• arXiv technical paper:https://arxiv.org/pdf/2412.09428

VMB application scenarios

• Film and video productionIt can automatically generate background music for movies, TV series, commercials, documentaries, etc., to enhance the emotional expression and atmosphere of visual content.
• Game developmentThe game generates background music in real time based on scene changes, enhancing player immersion and gaming experience.
• Virtual Reality (VR) and Augmented Reality (AR)It provides music that is compatible with virtual environments and augmented reality experiences, allowing music to synchronize with the user's visual experience and enhancing interactivity.
• Social media content creationIt helps users generate personalized music based on their video content (such as travel logs, life records, etc.), thereby enhancing the appeal of the content.
• Music education and creative supportIt assists musicians and music lovers in creating new compositions, providing inspiration and creative tools, especially when exploring different musical styles and structures.