Self Forcing - A video generation model developed by Adobe in collaboration with the University of Texas.

What is Self Forcing?

Self Forcing, a novel autoregressive video generation algorithm jointly developed by Adobe Research and the University of Texas at Austin, addresses the exposure bias problem of traditional generative models during training and testing. By simulating a self-generation process during training, it generates subsequent frames based on previously generated frames, rather than relying on real frames, thus bridging the gap between training and testing distributions. Self Forcing introduces a rolling key-value caching mechanism, supporting theoretically unlimited video generation and achieving real-time generation capabilities of 17 FPS on a single H100 GPU with latency less than one second. This breakthrough opens new possibilities for live streaming, gaming, and real-time interactive applications, such as real-time generation of virtual backgrounds or effects. The efficiency and low latency of Self Forcing make it an important tool for future multimodal content creation.

The main functions of Self Forcing

• High-efficiency real-time video generationSelf Forcing enables efficient real-time video generation on a single GPU, achieving a frame rate of 17 FPS with a latency of less than one second.
• Unlimited long video generationThrough a rolling key-value caching mechanism, Self Forcing supports theoretically unlimited video generation. Video content can be continuously generated without interruption due to length limitations, providing powerful support for dynamic video creation.
• Bridging the training and testing gapSelf-Forcing simulates a self-generation process during the training phase, generating subsequent frames based on the generated frames, rather than relying on real frames. This effectively solves the exposure bias problem in autoregressive generation, bridges the distribution differences between the training and testing phases, and improves the quality and stability of the generated videos.
• Low resource demandSelf Forcing optimizes the use of computing resources, enabling streaming video generation on a single RTX 4090 graphics card, reducing reliance on hardware resources and making it easier to deploy and use on ordinary devices.
• Support multimodal content creationSelf Forcing's efficiency and real-time capabilities enable it to support multimodal content creation, such as generating backgrounds or effects in real time during game streaming, or dynamically generating visual content in virtual reality experiences, providing creators with a wider range of applications.

The technical principle of Self Forcing

• Self-regression unfolding and overall loss monitoringSelf-Forcing simulates the autoregressive generation process during inference during the training phase, meaning that each frame is generated based on previously generated frames by the model, rather than real frames. It supervises the entire generated sequence through a video-level overall distribution matching loss function, not just frame-by-frame evaluation. The model can learn directly from its own prediction errors, effectively mitigating exposure bias.
• Rolling key-value caching mechanismTo support long video generation, Self Forcing introduces a rolling key-value (KV) caching mechanism. This mechanism maintains a fixed-size buffer that stores the KV embeddings of the most recent frames. When a new frame is generated, the buffer removes the oldest entry and adds the new embedding.
• Few-step diffusion model and gradient cutoff strategyTo improve training efficiency, Self Forcing employs a few-step diffusion model combined with a stochastic gradient truncation strategy. Specifically, during training, the model randomly selects the number of denoising steps and performs backpropagation only on the final denoising step.
• Dynamic condition generation mechanismWhen generating each frame, Self Forcing dynamically combines two types of conditional inputs: clear frames generated in past time steps and noisy frames in the current time step. Generation is completed through iterative denoising, ensuring the consistency and naturalness of the generation process.

Self Forcing project address

• Project official websitehttps://self-forcing.github.io/
• Github repository: https://github.com/guandeh17/Self-Forcing
• arXiv technical paper: https://arxiv.org/pdf/2506.08009

Application scenarios of Self Forcing

• Live streaming and real-time video streamingSelf Forcing enables real-time video generation at 17 FPS on a single GPU with latency of less than one second. It is suitable for live streaming scenarios, such as generating virtual backgrounds, effects, or dynamic scenes in real time during a live stream, providing viewers with a completely new visual experience.
• Game developmentIn game development, Self Forcing can generate game scenes and effects in real time, eliminating the need for pre-producing large amounts of video resources. It generates dynamic environmental changes and effects based on player actions in real time, enhancing the game's immersion and interactivity.
• Virtual Reality and Augmented RealitySelf Forcing's low latency and high-efficiency generation capabilities enable real-time visual content for virtual reality (VR) and augmented reality (AR) applications. It allows for the real-time generation of realistic virtual scenes in VR experiences or the real-time overlay of virtual elements in AR applications.
• Content creation and video editingSelf Forcing can be used as a short video creation tool to help creators quickly generate high-quality video content.
• World Simulation and TrainingSelf Forcing can be used for world simulation and training scenarios, such as generating realistic natural environments or urban landscapes for military training, urban planning, or environmental simulation.