LCVD - A lighting-controlled portrait animation generation framework developed by Sichuan University

What is LCVD?

LCVD (Lighting Controllable Video Diffusion Model) is a high-fidelity, lighting-controlled portrait animation generation framework developed by Sichuan University. LCVD separates the intrinsic features (such as identity and appearance) and extrinsic features (such as pose and lighting) of a portrait, using a reference adapter and a shadow adapter to map these features to different subspaces. During animation generation, LCVD combines the feature subspaces and uses a multi-condition classifier-guided mechanism to finely control the lighting effects, preserving the portrait's identity and appearance. The model is based on a stable video diffusion model (SVD), generating high-quality portrait animations that are consistent with the driving video pose and meet the target lighting conditions. LCVD significantly outperforms existing methods in terms of lighting realism, image quality, and video consistency, providing strong technical support for fields such as virtual reality, video conferencing, and film and television production.

Main functions of LCVD

• Portrait AnimationConvert static portraits into dynamic videos, matching and driving head movements and expressions in the videos.
• Light controlDuring animation generation, the portrait is relit based on the lighting conditions specified by the user or a reference image.
• Identity and appearance preservedDuring animation and relighting, preserve the identity and appearance of the portrait to avoid loss of identity information.
• High-quality video generationThe generated videos perform exceptionally well in terms of lighting realism, image quality, and video consistency, making them suitable for scenarios such as virtual reality, video conferencing, and film and television production.

LCVD Technical Principles

• Feature separationThe Reference Adapter maps the intrinsic features (identity and appearance) of a reference portrait to the feature space. The Shading Adapter maps the extrinsic features (lighting and pose) to the feature space. Based on the separation of intrinsic and extrinsic features, the model independently controls lighting and pose during animation.
• Light-controlled diffusion modelBased on a stable video diffusion model, a classifier-free guidance method is used to adjust lighting effects. By modifying the guidance intensity (such as the weight ω), the influence of lighting cues can be enhanced or weakened, achieving fine-grained lighting control.
• Motion alignment and long video generationBased on the motion alignment module, the generated portrait is ensured to match the pose of the driving video. A diffusion model sampling method is used to generate videos of arbitrary length, and an overlap strategy ensures smooth transitions between video segments.
• Training and optimizationDuring the training phase, self-supervised learning optimizes the adapter and diffusion model to ensure consistency in lighting, pose, and identity across generated videos. Loss functions (such as LPIPS, FID, etc.) are used to evaluate and optimize the quality of the generated videos.

LCVD project address

• arXiv technical paper:https://arxiv.org/pdf/2502.19894

Applications of LCVD

• Virtual Reality (VR) and Augmented Reality (AR)Create realistic virtual characters that blend naturally into virtual or real-world scenes.
• videoconferenceIt generates high-quality portrait animations in real time, reducing bandwidth requirements and improving user experience.
• Film and television productionQuickly generate portrait animations that fit different lighting conditions for use in special effects and virtual scenes.
• Game developmentGenerate realistic virtual character animations to enhance the realism and immersion of the game.
• Social media and content creationIt supports users in generating personalized dynamic avatars or short videos, enriching the forms of content creation.