IterComp - A text-to-image generation framework jointly developed by Tsinghua University, Peking University, Oxford University, and other institutions.

What is IterComp?

IterComp is a text-to-image generation framework jointly developed by researchers from Tsinghua University, Peking University, LibAI Lab, University of Science and Technology of China, Oxford University, and Princeton University. Based on an iterative feedback learning mechanism, it aggregates the combined generation preferences of multiple diffusion models, comprehensively improving the model's overall ability to handle complex combinatorial tasks. IterComp first builds a graph library containing multiple open-source models, each demonstrating its strengths in different aspects such as attribute binding, spatial relationships, and non-spatial relationships. A reward model is trained based on these model preferences, and an iterative optimization strategy is used to gradually improve the combined generation capabilities of the base diffusion models. This approach not only improves the quality and accuracy of generated images but also does not incur additional computational overhead, allowing IterComp to surpass existing state-of-the-art methods in multi-class object combination and complex semantic alignment.

Main functions of IterComp

• Model Preference AggregationIterComp aggregates the preferences of multiple open-source diffusion models, and each model exhibits its own advantages in different combinations of generation.
• Dataset ConstructionBased on model preferences, a dataset containing a large number of image ranking pairs is constructed and used to train a combined perception reward model.
• Iterative feedback learningBased on the iterative feedback learning method, the basic diffusion model and reward model are gradually optimized to improve the model's performance in multi-class object combinations and complex semantic alignment.
• Self-optimizationIterComp can self-optimize in a closed loop, continuously improving the quality and accuracy of generated images through multiple iterations.

IterComp's technical principles

• Model library constructionSelect multiple open-source diffusion models that perform well in different combinations and generate them to form a model library.
• Preference data collectionFor key combinatorial indicators such as attribute binding, spatial relationships, and non-spatial relationships, preference data is collected from the model library to construct a dataset.
• Reward Model TrainingThe collected dataset is used to train reward models for different combinatorial metrics, and the models will guide the optimization of the base diffusion model.
• Iterative optimizationBased on an iterative feedback learning framework, the basic diffusion model and reward model are gradually optimized to achieve continuous self-improvement of the model in combinatorial generation tasks.
• Effect verificationBased on extensive experiments, it is verified that IterComp improves combinatorial generation capabilities and is compared with state-of-the-art methods.

IterComp's project address

• GitHub repository:https://github.com/YangLing0818/IterComp
• HuggingFace model library:https://huggingface.co/comin/IterComp
• arXiv technical paper:https://arxiv.org/pdf/2410.07171

Application scenarios of IterComp

• Artistic CreationArtists and designers use IterComp to generate visual artworks with specific styles and complex combinations of elements, such as fantasy scenes, character designs, or concept art.
• Game developmentIn game design, IterComp quickly generates prototype images of game environments, characters, and items, accelerating the creation process of game assets.
• Advertising and Brand MarketingMarketers design advertising images that precisely combine multiple elements based on text descriptions to attract the target audience.
• Education and trainingIn the field of education, IterComp creates illustrations for teaching materials, such as visual explanations of scientific concepts or recreations of historical events.
• Media and EntertainmentContent creators generate concept art for video games, movies, or television programs, which is used as a visual element in storytelling.