Light-A-Video - A video relighting method developed by Shanghai AI Lab in collaboration with Shanghai Jiao Tong University and other universities.
Light-A-Video is a training-free video relighting method jointly developed by Shanghai Jiao Tong University, University of Science and Technology of China, Chinese University of Hong Kong, Hong Kong University of Science and Technology, Stanford University, and Shanghai AI Lab. It is based on a progressive...
What is Light-A-Video?
Light-A-Video is a training-free video relighting method jointly developed by Shanghai Jiao Tong University, University of Science and Technology of China, Chinese University of Hong Kong, Hong Kong University of Science and Technology, Stanford University, and Shanghai AI Lab. It achieves high-quality, temporally consistent video lighting control based on progressive light fusion technology. Light-A-Video implements two key modules: Consistent Light Attention (CLA) and Progressive Light Fusion (PLF). CLA stabilizes the generation of background lighting sources by enhancing inter-frame interaction; PLF gradually integrates the relighting effect into the video based on the motion prior of the video diffusion model, ensuring smooth lighting transitions. Light-A-Video supports relighting the entire video and relighting foreground sequences to generate the background.
Light-A-Video's main functions
- Time ConsistencyThis is based on enhancing the consistency of lighting between frames to avoid video flickering and lighting discontinuity issues.
- High-quality relightingThe pre-trained image relighting model is used to adjust the lighting of each frame in the video while maintaining image quality.
- Foreground and background separation processingSupports relighting the foreground of the video and automatically generates a background that matches the lighting conditions.
- Zero-shot generationNo additional training or optimization is required; it directly generates videos that meet the lighting conditions based on text prompts.
- High compatibilityIt is compatible with a variety of popular video generation models (such as AnimateDiff, CogVideoX, etc.) and has wide applicability.
The technical principles of Light-A-Video
- Consistent Light Attention (CLA):
- The CLA module stabilizes the generation of illumination sources by enhancing inter-frame interaction. It introduces cross-frame information into the self-attention layer of the image relighting model and suppresses high-frequency jitter in illumination based on temporal averaging features to generate stable background illumination.
- CLA employs a dual-stream attention fusion strategy: one stream processes the original frame information, preserving details; the other stream is processed based on temporal averaging to suppress jitter. Finally, the outputs of the two streams are fused through a weighted average.
- Progressive Light Fusion (PLF):
- The PLF module uses motion priors from the Video Diffusion Model (VDM) to gradually integrate the relighting effect into the video. Based on linear fusion, it combines the appearance of the relighting image with the appearance of the original video to ensure a smooth lighting transition.
- PLF gradually adjusts the illumination target during the denoising process of the video diffusion model. Based on the dynamic adjustment of fusion weights, it gradually guides the direction of video denoising and achieves a temporally consistent relighting effect.
- Overall processThe input video is first encoded as a noise signal in the latent space, which is then gradually restored through the VDM denoising process. In each denoising step, the CLA and PLF modules work together to progressively inject relighting information into the video.
Light-A-Video's project address
- Project official website:https://bujiazi.github.io/light-a-video.github.io/
- GitHub repository:https://github.com/bcmi/Light-A-Video/
- arXiv technical paper:https://arxiv.org/pdf/2502.08590
Application scenarios of Light-A-Video
- Post-production of film and televisionQuickly adjust lighting conditions, such as changing from day to night, saving costs and time.
- Game developmentDynamically change scene lighting to enhance immersion.
- Video creationQuickly change the style of your videos to meet your creative needs.
- VR/ARReal-time lighting adjustment enhances the fusion of virtual and reality.
- videoconference: Optimize video lighting to improve the remote collaboration experience.