RelightVid - A video relighting model developed by Shanghai AI Lab in collaboration with Fudan University and other universities.
RelightVid is a temporal consistency diffusion model for video relighting, developed by Shanghai AI Lab, Fudan University, Shanghai Jiao Tong University, Zhejiang University, Stanford University, and the Chinese University of Hong Kong. It supports rendering based on text cues, background visuals, etc.
What is RelightVid?
RelightVid, developed by the Shanghai AI Lab, Fudan University, Shanghai Jiao Tong University, Zhejiang University, Stanford University, and the Chinese University of Hong Kong, is a temporally consistent diffusion model for video relighting. It supports fine-grained and consistent scene editing of input videos based on text cues, background video, or HDR environment maps, and supports full-scene relighting and foreground-preserving relighting. The model generates high-quality video relighting data pairs based on a custom enhancement pipeline, combining real video and 3D rendered data. It inserts trainable temporal layers into a pre-trained Image Lighting Editing Diffusion Framework (IC-Light) to enhance the video relighting effect. RelightVid has significant advantages in maintaining temporal consistency and lighting detail, providing new possibilities for the field of video editing and generation.
RelightVid's main functions
- Text Conditional RelightingBased on the text description entered by the user, the video is relit, such as "sunlight shines through the leaves, creating dappled light and shadow" or "soft morning light, golden hour".
- Background video conditional heavy lightingUse background video as lighting conditions to dynamically adjust the lighting of the foreground object, so that the lighting of the foreground and background are consistent.
- HDR environment map conditional heavy lighting: Use HDR environment mapping to precisely control lighting and achieve high-quality heavy lighting effects.
- Full-scene relightingRelight the foreground and background to match the lighting conditions to the entire scene.
- Foreground Retention RelightingThis function relights the foreground while preserving the background, making it suitable for scenarios where the foreground object needs to be highlighted.
RelightVid's technical principles
- Diffusion model extensionRelightVid is based on a pre-trained image relighting diffusion model (such as IC-Light), and its extended architecture supports video input. It introduces a temporal layer to capture the temporal dependencies between frames, ensuring the temporal consistency of relighting.
- Multimodal conditional joint training:The model can use background video, text prompts, and HDR environment maps as conditions simultaneously. The conditions are embedded into the model based on the encoder, and collaborative editing is achieved based on the cross-attention mechanism.
- Illumination-Invariant Ensemble (IIE)The input video is enhanced with brightness and the predicted noise is averaged to improve the robustness of the model under different lighting conditions and prevent changes in albedo.
- Data Augmentation Pipeline (LightAtlas)High-quality relighting data pairs are generated based on real video and 3D rendering data, providing the model with rich prior knowledge of lighting and enhancing its adaptability to complex lighting scenes.
RelightVid project address
- Project official website:https://aleafy.github.io/relightvid/
- GitHub repository:https://github.com/Aleafy/RelightVid
- arXiv technical paper:https://arxiv.org/pdf/2501.16330
- Experience the demo online:https://huggingface.co/spaces/aleafy/RelightVid
Application scenarios of RelightVid
- Film and television productionAdjust the lighting effects of scenes in movies or TV series to match different plot requirements or the director's creative ideas without having to reshoot.
- Game developmentDynamically change scene lighting in the game to enhance immersion and visual effects, adapting to different time and weather conditions.
- Augmented Reality (AR)In AR applications, the lighting of virtual elements can be adjusted in real time to match the lighting of the real environment, thereby improving the user experience.
- Video advertising and marketingIt can quickly generate various lighting styles for advertising videos to meet the promotional needs of different brands or events and enhance visual appeal.
- Video content creationIt helps video bloggers or content creators easily change the lighting and atmosphere of their videos, such as simulating different weather or time of day scenes, to enrich their creative content.