AB
AB
AiBoss

All Categories

project

DanceGRPO - A unified visual generative reinforcement learning framework jointly developed by ByteDance Seed and the University of Hong Kong.

DanceGRPO is the first unified visual generative reinforcement learning framework jointly launched by ByteDance Seed and the University of Hong Kong. It applies reinforcement learning to the field of visual generation, covering two major generative paradigms (diffusion and rectified generation)...

What is DanceGRPO?

DanceGRPO is the first unified reinforcement learning framework for visual generation, jointly launched by ByteDance Seed and the University of Hong Kong. It applies reinforcement learning to the field of visual generation, covering two major generation paradigms (diffusion and rectified flow), three tasks (text-to-image, text-to-video, and image-to-video), four basic models (SD, HunyuanVideo, FLUX, and SkyReels-I2V), and five reward models (image-video aesthetics, image-text alignment, video dynamic quality, and binary reward). DanceGRPO addresses the limitations of existing RLHF solutions in visual generation tasks, achieving seamless adaptation across multiple generation paradigms, tasks, basic models, and reward models. It significantly improves model performance, reduces memory pressure, is adaptable to training on large prompt datasets, and can be transferred to rectified flow and video generation models.

DanceGRPO's main functions

  • Improve visual generation qualityTo make the generated images and videos more aesthetically pleasing, realistic, and natural.
  • Unifying multiple generation paradigms and tasksSuitable for various tasks such as text-to-image, text-to-video, and image-to-video.
  • Adaptable to multiple models and rewardsIt is compatible with multiple basic models and reward models to meet diverse needs.
  • Improve training efficiency and stabilityReduce memory pressure, improve training efficiency, and enhance training stability.
  • Enhancing human feedback learning abilityTo enable models to learn better from human feedback and generate content that better meets human expectations.

The technical principles of DanceGRPO

  • The denoising process is modeled as a Markov decision process.The denoising process of diffusion models and rectified flows is modeled as a Markov decision process (MDP), with the prompt as part of the state and each step in the denoising process as an action, providing a basic framework for the application of reinforcement learning.
  • based on SDE sampling equationTo meet the requirements of GRPO for stochastic exploration, the sampling process of the diffusion model and rectified flows is uniformly expressed in the form of stochastic differential equations (SDEs). For the diffusion model, the forward SDE describes the process of gradually adding noise to the data, and the corresponding inverse SDE generates the data. For rectified flows, the randomness of the inverse process is achieved by introducing SDEs, providing the necessary stochastic exploration mechanism for reinforcement learning.
  • Optimization using the GRPO objective functionThis approach borrows the GRPO strategy from Deepseek-R1. Given a prompt, it generates a set of output samples and optimizes the policy model based on the objective function of maximizing GRPO. The objective function considers the reward signal and the advantage function among different samples. During training, the model can better learn how to adjust the generation strategy according to the reward signal, improving the quality of the generated results and their consistency with human preferences.
  • Initialization noise and time step selection strategyIn the DanceGRPO framework, initialization noise is a key factor. To avoid reward hacking, DanceGRPO assigns shared initialization noise to samples from the same text prompt. DanceGRPO uses a time step selection strategy to reduce computation and improve training efficiency without sacrificing performance by choosing an appropriate optimization time step.
  • Integration of multi-reward models and aggregation of advantage functionsTo ensure training stability and high-quality generated results, DanceGRPO uses multiple reward models in practice. Since different reward models may have different dimensions and distributions, DanceGRPO uses a dominance function aggregation method to better balance the contributions of different reward models. This allows the model to comprehensively consider multiple evaluation metrics during optimization, generating visual content that better meets human expectations.

DanceGRPO's project address

Application scenarios of DanceGRPO

  • Text to Image GenerationGenerate high-quality images based on text descriptions, which can be used in advertising design, game development and other fields to improve creative efficiency.
  • Text to video generationIt generates smooth and coherent videos based on text, suitable for video advertising and educational video production, reducing labor costs.
  • Image to video generationIt converts static images into dynamic videos, which can be used in animation production and virtual reality to enrich the visual experience.
  • Multimodal content creationIt combines text, images, and videos to generate diverse content, which can be applied in multimedia education, interactive entertainment, and other fields to enhance immersion.
  • Creative design and artistic creationIt helps artists and designers quickly generate creative inspiration and artworks, stimulates more creativity, and improves creative efficiency.
Back