MM-StoryAgent - A multi-agent story picture book video generation framework jointly open-sourced by Shanghai Jiao Tong University and Alibaba.
MM-StoryAgent is an open-source, multimodal, multi-agent framework jointly developed by the X-LANCE Lab at Shanghai Jiao Tong University and Alibaba Group. It's used to generate immersive audio storybook videos. Based on a large-scale language model...
What is MM-StoryAgent?
MM-StoryAgent is an open-source, multimodal, multi-agent framework jointly developed by the X-LANCE Lab at Shanghai Jiao Tong University and Alibaba Group for generating immersive audio storybook videos. Based on a combination of large language models (LLMs) and multimodal generation tools (such as text, images, and audio), it enhances the story's appeal and immersion through a multi-stage writing process and modality-specific prompting and revision mechanisms. The framework supports flexible modular design, allowing for the replacement of different generation models and APIs. MM-StoryAgent improves story quality, achieving better alignment between images, speech, music, and sound effects, providing an efficient, flexible, and expressive solution for the automated creation of children's storybooks.
Main functions of MM-StoryAgent
- High-quality story generationBased on multi-agent collaboration and a multi-stage writing process, it generates engaging, educational, and emotionally resonant story content.
- Multimodal content generationIt combines multiple modalities such as text, images, voice, music, and sound effects to bring users a rich and immersive experience.
- Role ConsistencyIn image generation, based on character extraction and cue revision, we ensure that the characters in the story are visually consistent.
- Modal alignmentBased on prompt revision and contrastive learning models, the alignment between text, images, and audio is optimized to improve the overall coherence of the story.
- Flexible modular designIt supports flexible replacement of generation modules (such as different text generation models, image generation models, etc.), making it easy for developers to customize and optimize according to their needs.
Technical principles of MM-StoryAgent
- Multi-agent collaborative architectureThis feature simulates a dialogue between amateur and expert writers to generate story outlines and chapter content. The dialogue process mimics human brainstorming, providing richer creativity and greater appeal to the story. It transforms the textual story into prompts suitable for different modalities such as images and audio. Based on multi-round collaboration between "revisers and reviewers," the quality of the prompts is optimized.
- Multimodal generation technology:
- Text generationGenerate story text based on large language models (LLMs).
- Image generationUse diffusion models (such as StoryDiffusion) to generate images consistent with the story content, and use character extraction to ensure that characters remain consistent across multiple frames.
- Audio generationIt generates narration based on a text-to-speech (TTS) model and generates sound effects and background music using models such as AudioLDM2 or MusicGen.
- Video compositingThe generated images, audio, and text content are combined to form a complete audio story video.
- Modal alignment optimizationUse contrastive learning models (such as CLIP and CLAP) to evaluate the alignment between the generated content and the text, and optimize the generation effect based on the prompt revision mechanism.
- Modular designThe framework supports flexible replacement of different generation models and APIs, allowing developers to choose more advanced models to improve generation quality according to their needs.
MM-StoryAgent project address
- GitHub repository:https://github.com/X-PLUG/MM_StoryAgent
- arXiv technical paper:https://arxiv.org/pdf/2503.05242
- Experience the demo online:https://huggingface.co/spaces/wsntxxn/MM-StoryAgent
Application scenarios of MM-StoryAgent
- Children's Education and EntertainmentGenerate fun and educational audio story videos to accompany children's learning and growth.
- Digital content creationIt enables creators to quickly generate multimodal story content, reducing creation costs and improving efficiency.
- Online EducationIt serves as a teaching aid, explaining knowledge through stories to enhance the fun of learning.
- Multimedia PublishingAutomated generation of audio picture books helps publishers quickly produce multimedia content.
- Smart device integrationIt is used in devices such as smart speakers and tablets to provide personalized story generation services.