Lucy 2 - A real-time video generation model from Decart AI
Lucy 2.0 is a real-time world transformation model launched by Decart AI, enabling high-fidelity video editing to leap from offline rendering to an interactive real-time experience.
What is Lucy 2?
Lucy 2.0, a real-time world transformation model from Decart AI, enables high-fidelity video editing to leap from offline rendering to an interactive, real-time experience. As a pure diffusion model, it doesn't rely on 3D geometry or depth maps; its physical behavior is entirely learned autonomously through observing video evolution, allowing for continuous image generation at near-zero latency (30fps) at 1080p resolution. Based on Smart History Augmentation technology, the model can self-correct quality drift over long periods, supporting hours of uninterrupted, coherent generation. Deeply optimized for AWS Trainium 3, the model can be used for real-time character replacement, virtual try-on, and other visual effects, providing a real-time data augmentation and physically consistent simulation environment for robot training.
Lucy 2's main functions
- Real-time visual transformationThe model continuously generates footage at 30fps and 1080p resolution with near-zero latency, supporting real-time editing during live streams without the need for pre-rendering.
- Multi-dimensional editingIt enables character replacement, clothing change, product placement, motion control, and full environment transformation through text prompts and reference images.
- Persistent operation capabilityWith the help of Smart History Augmentation technology, it can run continuously for hours without identity collapse, geometric distortion or texture degradation.
- Robot Data AugmentationAs a real-time simulation engine, it changes materials, lighting, and environment in real time while maintaining physical consistency, expanding a single realistic demonstration into thousands of training variants.
The technical principles of Lucy 2
- Pure diffusion architectureIt does not rely on depth maps, 3D meshes, or explicit physics engines; all visual dynamics evolve autonomously through observing videos.
- Emergent physical understandingThe model implicitly learns the world structure (such as finger geometry, fabric wrinkles, and object separation) from the data without requiring manual definition of topological rules.
- Smart History AugmentationDuring the training phase, the model is exposed to its own imperfect output and penalized, enabling it to learn to recognize drift and actively correct it, rather than blindly following the previous frame.
- Hardware co-optimizationIt is deeply customized for AWS Trainium3, using mega-kernels to reduce startup overhead, on-chip SRAM to avoid high bandwidth memory latency, and custom WebRTC pipelines to achieve end-to-end real-time transmission.
Lucy 2 project address
- Project official websitehttps://decart.ai/publications/lucy-2-introducing-sota-video-generation-in-realtime
- Experience the demo onlinehttps://lucy.decart.ai/
Application scenarios of Lucy 2
- Live interactive broadcastThe host can instantly switch characters, virtually try on clothes, or dynamically insert products during the live stream, presenting diverse visual effects in real time without post-production.
- On-site content creationCreators can adjust the style and atmosphere of the scene in real time on set through text prompts, bringing the post-production process forward to achieve an instant visual preview that is exactly what you see.
- Robot Training EnhancementThe R&D team can change the material, lighting, and background environment of objects in real time, expanding a single real operation video into thousands of physically consistent and diverse training samples, thus solving the problem of data scarcity.
- Virtual production simulationProduction teams can leverage the model's ability to run stably for extended periods to generate interactive dynamic backgrounds and environmental simulations in real time, directly replacing traditional offline rendering production processes.