Diffusion Self-Distillation - A zero-shot custom image generation technology from Stanford University

What is Diffusion Self-Distillation?

Diffusion Self-Distillation (DSD) is a zero-shot custom image generation technique developed by Jiajun Wu's team at Stanford University. It automatically generates datasets using a pre-trained text-to-image diffusion model and fine-tunes them to perform text-conditional image-to-image tasks. Based on generated image grids and visual-language model filtering, Diffusion Self-Distillation creates high-quality paired datasets, enabling custom image generation that preserves the identity of any instance in any context without human intervention. This method excels in maintaining identity consistency, competing with instance-by-instance tuning techniques, and requires no test-time optimization.

The main functions of Diffusion Self-Distillation

• Custom image generation that preserves identityGenerate images of specific instances in different contexts while preserving the identity characteristics of those instances.
• Zero-shot learning: No training data for specific instances is required; pre-trained models can be directly applied to generate tasks.
• Automated data pairingIt automatically creates high-quality training data pairs based on self-generated datasets and the assistance of visual language models.
• Image-to-image conversionIt supports converting input images into output images with specific text conditions, such as changing lighting, style, or other visual attributes.
• Wide applicabilityIt is suitable for a variety of text-conditional image generation tasks, including personalization, relighting, depth control, and instruction following.

The technical principle of Diffusion Self-Distillation

• Context generation capability of pre-trained modelsImage meshes are created based on the context generation capabilities of a pre-trained text-to-image diffusion model.
• Data matching and filteringUsing visual language models (VLMs) to assist in screening, a large-scale, high-quality pairing dataset is created from the generated image grid.
• Fine-tuning diffusion modelThe pre-trained text-to-image diffusion model was fine-tuned using the selected paired dataset, transforming it into an image generation model that supports text + image conditions.
• Parallel processing architectureThis paper introduces a parallel processing architecture that treats the input image as the first frame of a video sequence and generates two video frames as output. The first frame reconstructs the input image, and the second frame is the edited output, achieving effective information exchange.
• Information exchange and editingEstablish identity mapping and conditional editing goals between two video frames, enabling the model to capture complex semantics and perform complex edits.

Diffusion Self-Distillation project address

• Project official website:primecai.github.io/dsd
• arXiv technical paper:https://arxiv.org/pdf/2411.18616

Application scenarios of Diffusion Self-Distillation

• Artistic CreationArtists create works in different styles and contexts while maintaining consistency in the characters or objects within the artwork.
• Game developmentIn game design, this involves quickly generating multiple variations of game characters or items with consistent identity characteristics to adapt to different game scenarios.
• Film and animation productionFilmmakers maintain consistency in character portrayal across different scenes, or re-render scenes under varying lighting conditions.
• Advertising and MarketingMarketers customize advertising images to ensure brand image remains consistent across various advertising media.
• Personalized productsIt generates personalized products based on user-uploaded images, such as customized T-shirts, mugs, and phone cases, while maintaining consistency in brand elements.