AB
AB
AiBoss

All Categories

project

Jodi - A unified model for visual understanding and generation developed by the Chinese Academy of Sciences

Jodi is a diffusion model framework developed by the Institute of Computing Technology of the Chinese Academy of Sciences and the University of Chinese Academy of Sciences. It unifies visual generation and understanding based on joint modeling of the image domain and multiple label domains.

What is Jodi?

Jodi is a diffusion model framework developed by the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences. It unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Jodi utilizes a linear diffusion Transformer and a role-switching mechanism to perform three tasks: joint generation (generating images and multiple labels simultaneously), controlled generation (generating images based on label combinations), and image perception (predicting multiple labels from images). Jodi is trained on the Joint-1.6M dataset, which contains 200,000 high-quality images and labels from seven visual domains. Jodi performs exceptionally well in both generation and understanding tasks, demonstrating strong scalability and cross-domain consistency.

Jodi's main functions

  • Joint generationIt generates images and multiple labels simultaneously, such as depth maps, normal maps, and edge maps, ensuring that the generated images and labels are consistent in both semantics and spatiality.
  • Controllable generationGenerates an image based on a given combination of labels. The user specifies certain labels as conditional inputs to control specific attributes or features of the generated image.
  • Image perceptionIt can predict multiple labels from a given image simultaneously, enabling multi-dimensional understanding and analysis of the image, such as performing tasks like depth estimation, edge detection, and semantic segmentation at the same time.

Jodi's technical principles

  • Joint modelingModeling is based on the joint distribution of the image domain and multiple label domains. Based on the learned joint distribution p(x, y1, y2, …, yM), the marginal and conditional distributions required for generating and understanding tasks are derived.
  • Character switching mechanismDuring training, each domain is randomly assigned one of three roles: generating target ([G]), conditional input ([C]), or ignored ([X]). The model can simultaneously learn different types of probability distributions, including joint generation, controlled generation, and image perception.
  • Linear Diffusion TransformerUsing a linearly diffusing Transformer as the backbone network, the computational complexity is effectively reduced based on a linear attention mechanism, enabling the model to efficiently handle multiple visual domains. Masked linear attention and domain-invariant position embeddings are introduced to enhance consistency and alignment between different visual domains.
  • Dataset ConstructionTo support joint modeling across multiple visual domains, Jodi constructed the Joint-1.6M dataset, which contains 200,000 high-quality images and corresponding automatic labels for the seven visual domains, providing rich data support for model training.

Jodi's project address

Jodi's application scenarios

  • Creative content generationIt provides inspiration for artists and designers, quickly generating images with specific styles or elements.
  • Multimodal data augmentationGenerate high-quality multimodal data to enhance the training dataset for machine learning models.
  • Image editing and restorationRepair or edit images based on user input, generate missing parts or adjust styles.
  • Visual understanding and analysisPredicting multiple visual labels to assist in image understanding tasks, such as medical image analysis.
  • Virtual Reality and Augmented RealityGenerate realistic virtual scenes and annotation information to enhance the experience of VR and AR applications.
Back