StoryWeaver - A unified story visualization AI model jointly launched by Xiamen University and NetEase Fuxi.

What is StoryWeaver?

StoryWeaver, an AI model developed by the Key Laboratory of Multimedia Trusted Perception and High-Efficiency Computing of the Ministry of Education at Xiamen University and the NetEase Fuxi AI Lab, enables high-quality story visualization through knowledge-enhanced character customization. StoryWeaver uses a novel Character Graph to richly represent characters, attributes, and relationships within a story. It employs Customization via Character Graph (CCG) and Knowledge-Enhanced Space Guidance (KE-SG) techniques to precisely inject character semantics, generating image sequences that match the textual narrative. The system excels in maintaining alignment between character identity and textual semantics, effectively improving the accuracy and vividness of story visualization.

StoryWeaver's main functions

• Character customization and visualizationGenerates a series of visual story images based on textual narration and character images, precisely customizing the appearance of a given character.
• Semantic alignmentThe system can ensure that the generated image sequence is semantically consistent with the text prompts, that is, the image content matches the text description.
• Knowledge graph applicationsThe system can understand and represent the detailed attributes of the characters in the story and the relationships between them.
• Multi-role interactionStoryWeaver can handle multi-character story scenarios, keeping each character's identity clear and showcasing natural interactions between characters.
• Cross-attention allocation: Optimize attention allocation in multi-character stories to avoid identity mixing issues.

The technical principles of StoryWeaver

• Character Graph (CG): Construct a knowledge graph CG, which consists of object nodes (characters), attribute nodes (attributes related to characters), and events (relationships between characters), which together define the core elements of the story scene.
• Customization via Character Graph (CCG)Based on CCG, the structured knowledge in CG is transformed into enhanced scene descriptions, improving the consistency of character identity preservation and event semantic alignment.
• Knowledge Enhancement Space Guidance (KE-SG)A knowledge encoder is introduced to extract features of different characters. The initial position prior is adjusted according to the character features, and the erroneous cross-attention map is corrected to ensure that the character knowledge accurately focuses on the corresponding area in the story scene.
• Attention mechanism modificationThis approach enhances the attention map by strengthening areas relevant to the characters and reducing the attention given to areas unrelated to the characters, thereby improving the visual quality of multi-character stories.
• Unified frameworkStoryWeaver provides a unified framework that can handle both single-character and multi-character story visualization tasks.

StoryWeaver project address

• GitHub repository:https://github.com/Aria-Zhangjl/StoryWeaver
• arXiv technical paper:https://arxiv.org/pdf/2412.07375

Application scenarios of StoryWeaver

• Educational SupportUse StoryWeaver to generate story images to help children learn language and literature and increase their interest in reading.
• Comic creationAutomatically generates frame images for comic book stories, speeding up the creative process for comic artists.
• Interactive gamesIn role-playing games, storylines and visual content are dynamically generated based on player choices.
• Personalized advertisingCustomize personalized story ads for different products to enhance the appeal and effectiveness of advertising.
• Movie PreviewTransforming film scripts into visual scenes helps directors with pre-production planning and previewing.