Yume 1.5 - An interactive world generation model jointly developed by Shanghai AI Lab and Fudan University.

What is Yume 1.5?

Yume 1.5, developed by the Shanghai Artificial Intelligence Laboratory and Fudan University, is an interactive world generation model capable of generating realistic, continuous, and explorable virtual worlds from single images or text prompts. The model addresses the limitations of existing models in terms of versatility, latency, and text control capabilities through joint spatiotemporal channel modeling (TSCM) and real-time acceleration strategies. Yume 1.5 supports three modes: text-to-world, image-to-world, and text-based event editing, allowing users to control character and camera movement via the keyboard. Achieving 12 FPS real-time rendering on a single GPU, Yume 1.5 significantly improves interactive performance and provides new possibilities for virtual environments and simulation systems.

Main features of Yume 1.5

• Text-to-WorldIt supports users in generating corresponding virtual worlds by inputting text descriptions.
• Image-to-World: Generate a dynamic, explorable virtual world based on a single static image.
• Text-based event editingUsers can trigger specific events through text commands, such as "a ghost suddenly appears" or "a rainstorm begins," and observe the events unfolding in the virtual world in real time.
• Real-time interaction and controlIt supports real-time exploration of the virtual world via keyboard input (such as using the WASD keys to control character movement and the arrow keys to control camera direction). Users can move freely in the generated world, observe scene changes from different perspectives, and achieve an immersive experience.
• Long video generation and coherenceThe model supports the generation of infinitely long video content while maintaining temporal and visual coherence. By combining spatiotemporal channel modeling (TSCM) and self-forcing techniques, it addresses the common quality degradation issues in long sequence generation, ensuring that the video remains stable over extended periods.

Technical Principles of Yume 1.5

• Joint Spatiotemporal Channel Modeling (TSCM)Yume 1.5 employs a joint spatiotemporal channel modeling technique, reducing the GPU memory usage of historical frames through spatiotemporal compression and channel compression. The model downsamples historical frames in both temporal and spatial dimensions while compressing the channel dimension, and then fuses features using linear attention. This approach significantly reduces computational costs while maintaining generation quality, making long video generation more efficient.
• Real-time acceleration strategyYume 1.5 combines self-forcing and enhanced text embedding to reduce error accumulation during inference. The model uses its own generated historical frames as conditions for prediction, improving robustness to errors. By optimizing the training process, it achieves fast inference and significantly improves real-time interactive performance.
• Text control event generationYume 1.5 decomposes text prompts into "event descriptions" and "action descriptions," processing them separately to reduce computational overhead. Through a mixed dataset training strategy, the model can trigger specific events based on text commands, achieving text-based event generation capabilities and enhancing the interactivity and dynamism of the virtual world.
• System optimization and alternating training paradigmYume 1.5 employs a hybrid dataset training strategy, combining an alternating training paradigm to simultaneously achieve world generation and editing capabilities. During training, the model alternates between text-to-video and image-to-video tasks, improving overall performance across different tasks and providing stronger support for the generation of complex real-world scenes.

Yume 1.5 project address

• Project official website: https://stdstu12.github.io/YUME-Project/
• GitHub repository: https://github.com/stdstu12/YUME
• HuggingFace model libraryhttps://huggingface.co/stdstu123/Yume-5B-720P
• arXiv technical paperhttps://arxiv.org/pdf/2512.22096

Application scenarios of Yume 1.5

• Virtual Reality and Augmented Reality (VR/AR)Yume 1.5 can generate immersive virtual worlds, allowing users to freely explore and interact with them through VR/AR devices, and is suitable for virtual tourism, education, and entertainment experiences.
• Game developmentYume 1.5 enables game developers to generate and edit dynamic game scenes in real time, supports text commands to trigger specific events, and enhances the diversity and playability of games.
• Film and television productionThe model can be used to quickly generate virtual scenes and special effects in film and television post-production, generating specific environments through text descriptions, thus reducing production costs and time.
• Architectural Design and Urban PlanningYume 1.5 can generate dynamic urban environments based on design drawings or text descriptions, helping architects and planners to conduct virtual tours and evaluate solutions.
• educateYume 1.5 can generate virtual content such as historical scenes and scientific experimental environments, providing students with an immersive learning experience.