DoraCycle - National University of Singapore launches unified generative model for multimodal domain adaptation

What is DoraCycle?

DoraCycle, developed by the Show Lab at the National University of Singapore, is a unified generative model for multimodal domain adaptation. It achieves information transformation and alignment between different modalities through two multimodal loops (text-to-image-to-text and image-to-text-to-image). Based on bidirectional mappings learned by the unified generative model, it is trained on unpaired data for domain adaptation, eliminating the need for large amounts of labeled data. The model is optimized through cross-entropy loss at the loop endpoints, promoting self-evolution and adapting to specific domains.

DoraCycle's main functions

• Domain adaptation without paired dataThrough cycle consistency learning, DoraCycle has achieved domain adaptation of generative models using unpaired data for the first time, significantly reducing data acquisition costs.
• Flexible task adaptabilityDoraCycle can handle tasks that do not require pairwise knowledge (such as stylization) and can effectively combine a small amount of paired data to complete tasks that require new knowledge (such as identity generation).

DoraCycle's technical principles

• Multimodal Cyclic Consistency LearningDoraCycle integrates two multimodal cycles: text-to-image-to-text (T cycle) and image-to-text-to-image (I cycle). These two cycles utilize pre-trained unified generative models (such as vision-language alignment models) for cross-modal mapping.
  • T cycleStarting with the input text sequence, the model first converts it into an image representation, then converts the generated image back into a text sequence, and optimizes the model by calculating the cross-entropy loss between the generated text and the original text.
  • I cycleStarting with the input image, it is first converted into a text description, and then the text description is converted back into an image. The model is optimized by calculating the cross-entropy loss between the generated image and the original image.
• Self-supervised learning for cross-modal alignmentDoraCycle is a bidirectional mapping between vision and language learned using a unified generative model. Through these two cycles, data can be preserved within the same modality, imposing constraints on biases introduced during the process. This enables the model to achieve cross-modal alignment between vision and language through self-supervised learning.
• Enhanced training stabilityTo avoid gradient explosion during multi-step inference, DoraCycle employs the following techniques:
  • Gradient clippingThis avoids conflicting optimization directions between the two loops, thereby improving training stability.
  • EMA modelMaintain a slowly updated exponential moving average (EMA) model for inference to generate pseudo-data, thereby enhancing the stability of pseudo-data generation.

DoraCycle's project address

• Github repository:https://github.com/showlab/DoraCycle
• arXiv technical paper:https://arxiv.org/pdf/2503.03651

Application scenarios of DoraCycle

• Stylized designDoraCycle can be used to generate images and text content that conform to a specific style.
• Virtual character generationIn virtual character design, DoraCycle can combine a small amount of paired data with a large amount of unpaired data to generate virtual characters with specific identities and styles.
• Personalized advertising contentDoraCycle can generate personalized ad images and copy based on brand style and target audience.
• Personalized learning materialsDoraCycle can generate personalized learning materials based on students' learning styles and preferences.