SynthLight - A portrait relighting technology developed in partnership with Yale University and Adobe.

What is SynthLight?

SynthLight is a portrait relighting technology developed jointly by Yale University and Adobe Research, based on a diffusion model. Trained on synthetic data simulating different lighting conditions, it can re-render portrait photos into images with entirely new lighting effects, such as adding highlights, shadows, or adjusting the overall lighting atmosphere. Its core lies in a physically based rendering engine that generates synthetic datasets. Through multi-task training and a classifier-guided diffusion sampling strategy, it bridges the gap between synthetic data and real images, achieving high-quality relighting of realistic portraits.

SynthLight's main functions

• Portrait relightingThe portrait is relit using an ambient lighting map to generate realistic lighting effects, such as natural highlights, shadows, and diffuse reflections.
• Multi-task trainingMulti-task training based on real human images without lighting labels improves the model's generalization ability and enables it to handle various real-world scenarios.
• Inference time samplingThe algorithm employs a classifier-free diffusion sampling procedure to preserve the details of the input portrait during the inference stage, ensuring that the generated lighting effects are natural and delicate.
• Physical rendering engineIt uses a physically based rendering engine to generate synthetic datasets, simulates lighting transitions under different lighting conditions, and provides high-quality lighting effects.
• UniversalityDespite being trained using only synthetic data, SynthLight generalizes well to real-world scenes, including half-body and full-body portraits.

SynthLight's technical principles

• Treating relighting as a re-rendering issueSynthLight defines image relighting as a re-rendering process that adjusts the rendering effect of pixels by changing the ambient lighting conditions.
• Physically based rendering engines and synthetic datasetsThe technology uses a physically-based rendering engine to generate synthetic datasets. By rendering a 3D head model under different lighting conditions, it simulates pixel changes under realistic lighting conditions.
• Multi-task training strategySynthLight employs a multi-task training approach, using real human images without lighting labels for training. By combining training with real and synthetic images, it helps the model better adapt to real-world scenes and reduces the domain gap between synthetic data and real images.
• Classifier-free guided diffusion samplingDuring the inference phase, SynthLight uses a classifier-free diffusion sampling procedure. It leverages the details of the input portrait to better preserve the texture and features of the image, while generating realistic lighting effects.
• Advantages of diffusion modelsThe diffusion model generates images by progressively adding noise and learning a denoising process. SynthLight leverages this characteristic to generate high-quality lighting effects, including specular highlights, shadows, and subsurface scattering.

SynthLight's project address

• Project official website:https://vrroom.github.io/synthlight
• arXiv technical paper:https://arxiv.org/pdf/2501.09756

Application scenarios of SynthLight

• Portrait photography post-processingSynthLight can relight realistic portrait photos, generating lifelike lighting effects such as natural highlights, shadows, and diffuse reflections.
• Virtual scene rendering and compositingBy simulating different lighting conditions, SynthLight can seamlessly blend real-life portraits with virtual scenes to generate high-quality composite images.
• Game development and character designIn game development, SynthLight can be used to quickly adjust the lighting effects of characters to adapt to different scenes and environments, saving time and costs.
• Advertising and Commercial Image ProductionIn advertising and commercial image production, SynthLight can quickly generate images under various lighting conditions, helping designers quickly explore different visual effects and improve work efficiency.