SynCD - An open-source training dataset for synthesizing textural images from Meta and Carnegie Mellon University.

What is SynCD?

SynCD (Synthetic Customization Dataset) is a high-quality synthetic training dataset from Carnegie Mellon University and Meta, designed to enhance the customization capabilities of text-to-image models. SynCD contains images of the same object under different lighting, backgrounds, and poses. It ensures object consistency across different images using a masked shared attention mechanism and 3D asset guidance (such as the Objaverse). SynCD uses a language model (LLM) to generate detailed object descriptions and background scenes, combined with a deep-guided text-to-image model to generate coupled images. SynCD addresses the challenge of large-scale collection of multi-view, multi-background object images from the real world, providing rich training resources for tuning-free model customization and significantly improving image quality and identity preservation capabilities when generating specific objects in new scenes.

SynCD's main functions

• Provide diverse training samples: By generating images from multiple perspectives and backgrounds, the model's visual understanding of objects is enhanced.
• Enhance object consistencyBased on a shared attention mechanism and 3D asset guidance, it ensures that objects remain consistent across different images, avoiding the drift of object features in the generated images.
• Improve generation quality: Improve the image quality and identity preservation capabilities of models in customized tasks using high-quality synthetic data.
• Supports customization without tuningProvides data support for tuning-free methods, avoiding expensive optimization steps for each new object.

SynCD Technical Principles

• Language Model-Assisted Prompt Generation:
  • Detailed object and background scene descriptions are generated using a language model (such as LLama3). For rigid objects, the object description provided by Cap3D is used; for deformable objects, detailed descriptions are generated directly from the category name.
  • Multiple background descriptions are generated based on LLM, and object descriptions are combined with background descriptions to generate multi-image prompts.
• Masked Shared Attention (MSA):
  • When generating multiple images, the features of the foreground object region are shared based on the Masked Shared Attention mechanism to ensure the consistency of objects in different images.
  • In the attention module of the diffusion model, each image's features not only focus on itself, but also on object features in other images, ignoring background regions based on a mask.
• 3D Asset Guidance:
  • For rigid body objects, use 3D assets in Objaverse for multi-view rendering to generate depth maps and corresponding images.
  • Based on depth guidance and multi-view correspondence, the 3D consistency of objects is further enhanced. It supports pixel-level cross-view correspondence, "distorting" features in one image into other images to ensure that the shape and color of objects are consistent across different viewpoints.
• Data filtering and quality control:
  • Low-quality or inconsistent images are filtered using aesthetic scores and object similarity (calculated via the DINOv2 feature space) to ensure the quality of the final dataset.
  • Based on automated filtering steps, images that do not meet the requirements are removed, ultimately generating a high-quality synthetic dataset.

SynCD's project address

• Project official website:https://www.cs.cmu.edu/~syncd
• GitHub repository:https://github.com/nupurkmr9/syncd
• arXiv technical paper:https://arxiv.org/pdf/2502.01720

SynCD application scenarios

• Personalized content generationUsers can upload photos of their personal items or pets, and the system will generate new images of them in different scenes or styles based on text prompts to meet their personalized needs.
• Creative Design and Artistic CreationDesigners and artists can quickly generate concept images, validate design ideas, or create artworks with a specific style, improving the efficiency of creative expression.
• Virtual scene constructionIn virtual reality (VR) and augmented reality (AR), specific objects in virtual scenes are generated, such as placing virtual characters in different environments to enhance immersion.
• Advertising and MarketingBrands can generate images of their products in different usage scenarios or target audience environments for use in advertising to enhance appeal and persuasiveness.
• Education and TrainingIn the education field, teaching materials can be generated, such as placing historical artifacts in ancient settings, to help students better understand knowledge and improve learning outcomes.