StoryMem - A video generation framework jointly developed by ByteDance and Nanyang Technological University

What is StoryMem?

StoryMem is an open-source video generation framework jointly released by ByteDance and Nanyang Technological University, addressing the long-standing issue of cross-shot consistency in AI video generation. Through an explicit visual memory mechanism, it transforms single-shot video diffusion models into multi-shot narrative tools. Its core is the Memory-to-Video (M2V) module, which maintains a dynamically updated keyframe memory library and injects memories into the single-shot model, ensuring cross-shot consistency.

StoryMem's main functions

• Multi-camera long video generationIt can generate coherent multi-camera long video stories, support narratives across multiple scenes, and maintain consistency between characters and scene elements.
• Dynamic memory mechanismBy maintaining a keyframe memory library, the memory is injected into the single-shot video diffusion model to ensure that the video maintains logical and visual coherence over a long period of time.
• Cinematic picture quality and high aesthetic qualityIt inherits the high image quality and aesthetic characteristics of the single-lens video generation model, while maintaining an accurate understanding and adherence to user prompts.
• Flexible camera control and transitionsIt supports shot-level control, enabling smooth shot transitions and customized narrative applications.
• Cross-lens consistency optimizationIn multi-camera video generation, it significantly improves consistency across shots, showing a clear advantage over other methods.
• Customized story generationIt allows users to generate customized stories that meet specific needs by using reference images as initial memories.

StoryMem's technical principles

• Memory maintenanceAfter generating the first shot, the system extracts keyframe information and stores it in the dynamic memory library for reference when generating subsequent shots.
• Memory-to-Video (M2V) moduleThe visual features in the memory bank are injected into the single-shot video diffusion model, and the memory and generated content are fused through latent space stitching and negative RoPE offset.
• Semantic keyframe selection strategyAfter generating each shot, information-rich and reliable memory frames are obtained through semantic keyframe selection strategy and aesthetic preference filtering to achieve long-term consistency across shots.
• Iterative lens compositingBy iteratively generating shots and dynamically updating the memory library, a coherent long video story is gradually constructed.
• Lightweight LoRA tuningA pre-trained single-shot video diffusion model can be transformed into a multi-shot narrative tool simply by fine-tuning with LoRA (Low-Rank Adaptation).

StoryMem project address

• Project official website: https://kevin-thu.github.io/StoryMem/
• Github repository: https://github.com/Kevin-thu/StoryMem
• Huggingface model libraryhttps://huggingface.co/Kevin-thu/StoryMem
• arXiv technical paper: https://arxiv.org/pdf/2512.19539

StoryMem Application Scenarios

• Advertising and MarketingQuickly generate coherent ad videos, reduce the cost of storyboard visualization, and enhance the creativity and appeal of ad content.
• Film and television productionIt provides independent creators and small film and television teams with technical support for efficiently generating coherent narrative short films, accelerating the creative process.
• Content creationIt helps video creators generate high-quality narrative videos in a short time, improving the efficiency and diversity of content creation.
• Education and TrainingGenerate educational video stories for use in teaching and training scenarios, enhancing the fun and interactivity of learning.
• Entertainment and GamesIt provides game developers and the entertainment industry with customized story generation tools to enrich game plots and entertainment content.
• social mediaIt provides social media creators with tools to quickly generate engaging video stories, increasing the reach and user engagement of their content.