D-DiT - A multimodal diffusion model developed by Yale University in collaboration with Byte Seed and other institutions.

What is D-DiT?

D-DiT (Dual Diffusion Transformer) is a multimodal diffusion model developed by Carnegie Mellon University, Yale University, and ByteDance Seed Lab, unifying image generation and understanding tasks. The model combines continuous image diffusion (stream matching) and discrete text diffusion (mask diffusion), simultaneously training image and text modalities based on a bidirectional attention mechanism. D-DiT can perform bidirectional tasks of text-to-image and image-to-text generation, supporting various applications such as visual question answering and image caption generation. Based on the multimodal diffusion Transformer architecture and trained jointly with diffusion targets, the model exhibits multimodal understanding and generation capabilities comparable to autoregressive models, providing a new direction for the development of visual language models.

Main functions of D-DiT

• Text to Image GenerationGenerate high-quality images based on the input text description.
• Image to text generationGenerate descriptive text, such as image descriptions, titles, or answers to visual questions, based on the input image.
• Visual Q&AIt combines images and question text to generate accurate answers.
• Multimodal understandingIt supports a variety of visual language tasks, such as image description, visual instruction understanding, and long text generation.
• Bidirectional generation capabilityIt supports both text-to-image and image-to-text generation tasks, offering high flexibility.

The technical principle of D-DiT

• Two-branch diffusion modelD-DiT combines continuous image diffusion and discrete text diffusion. Continuous image diffusion uses flow matching to generate the image through a reverse diffusion process. Discrete text diffusion uses masked diffusion to gradually denoise and generate the text.
• Multimodal Transformer architecture:
  • Image BranchProcess image data and output the diffusion target of the image.
  • Text branchProcess text data and output the text's dissemination target.
• Joint training objectivesThe model simultaneously trains image and text modalities based on a joint diffusion objective. Image diffusion loss is based on flow matching loss, optimizing the reverse diffusion process of image generation. Text diffusion loss is based on mask diffusion loss, optimizing the reverse diffusion process of text generation. Through joint training, the model learns the joint distribution between images and text.
• Bidirectional attention mechanismD-DiT uses a bidirectional attention mechanism, allowing the model to flexibly switch between images and text, and supports unordered processing of input modalities. This enables the model to fully utilize information from both images and text during the generation process, improving the performance of multimodal tasks.

D-DiT project address

• Project official website:https://zijieli-jlee.github.io/dualdiff.github.io/
• GitHub repository:https://github.com/zijieli-Jlee/Dual-Diffusion
• arXiv technical paper:https://arxiv.org/pdf/2501.00289

Application scenarios of D-DiT

• Text to Image GenerationGenerates high-quality images based on text descriptions, suitable for creative design, game development, advertising production, and education.
• Image to text generationGenerate descriptive text for images to assist visually impaired individuals, provide content recommendations, and create smart photo albums.
• Visual Q&AIt combines images and questions to generate accurate answers, which can be used in smart assistants, educational tools, and customer support.
• Multimodal dialogue systemIt can generate detailed answers by combining images in the dialogue, and is suitable for intelligent customer service, virtual assistants and educational tutoring.
• Image Editing and EnhancementRepair, transform, or enhance images based on text descriptions; used in image restoration, style transfer, and image enhancement.