Hyper-SD - A high-efficiency image compositing framework launched by ByteDance
Hyper-SD is a high-efficiency image generation framework developed by researchers at ByteDance. Through Trajectory Segmentation Consistent Distillation (TSCD) technology, combined with human feedback learning and fractional distillation, it significantly improves the performance of diffusion models with fewer...
What is Hyper-SD?
Hyper-SD, developed by researchers at ByteDance, is a high-efficiency image synthesis framework designed to address the high computational cost of existing diffusion models during multi-step inference. Hyper-SD utilizes Trajectory Segmentation Consistent Distillation (TSCD) to maintain data consistency across different time periods, effectively preserving the original ODE (Ordinary Differential Equation) trajectory. Furthermore, it incorporates human feedback learning to optimize model performance in low-step inference scenarios and leverages fractional distillation to further enhance image quality during single-step inference. This framework significantly reduces necessary inference steps while maintaining high image quality, enabling rapid generation of high-resolution images and further advancing the field of generative AI.
Hyper-SD official website entrance
- Official project homepage: https://hyper-sd.github.io/
- Hugging Face model address:https://huggingface.co/ByteDance/Hyper-SD
- arXiv research paper:https://arxiv.org/abs/2404.13686
- Hyper-SD T2I version Demo:https://huggingface.co/spaces/ByteDance/Hyper-SDXL-1Step-T2I
- Hyper-SD Doodle Demo:https://huggingface.co/spaces/ByteDance/Hyper-SD15-Scribble
How Hyper-SD works
- Trajectory Segmentation Consistent Distillation (TSCD):The training time step range [0, T] is divided into k uniform time intervals.Consistent distillation is performed within each time period, using the original model as the teacher, and the student model gradually learns the behavior of the teacher model.By gradually reducing the number of time intervals (e.g., 8 → 4 → 2 → 1), the student model is trained to approximate the global behavior of the teacher model.
- Human Feedback Learning (ReFL):The model is optimized by leveraging human feedback on image preferences.Train the reward model so that it can recognize and reward images that are more in line with human aesthetics.The student model is fine-tuned by iterative denoising and direct prediction, combined with feedback from the reward model.
- Fractional distillation:Use scoring functions based on the true and false distributions to guide the single-step reasoning process.The single-step generation performance of students is optimized by minimizing the KL divergence between two distributions.
- Low-rank adaptation (LoRA):Using LoRA technology to adapt and train student models makes it a lightweight plugin that can be quickly deployed and used.
- Training and loss function optimization:Define a loss function that combines consistency loss, human feedback loss, and fractional distillation loss.The student model is trained using optimization algorithms such as gradient descent, while the LoRA plugin is updated.
- Reasoning and image generation:After training, the student model is used for the inference process of image generation.Choose an appropriate number of inference steps based on the needs of the application scenario to balance generation quality and efficiency.
- Performance evaluation:Use quantitative metrics (such as CLIP score, aesthetic score) and qualitative metrics (such as user research) to evaluate the quality of generated images.Based on the evaluation results, the model parameters were further adjusted and optimized.