AB
AB
AiBoss

All Categories

project

Hunyuan-DiT - Tencent Hunyuan's open-source text-based graph diffusion model Hunyuan-DiT

Hunyuan-DiT is a high-performance text-to-image diffusion Transformer model developed by Tencent's Hunyuan team. It possesses fine-grained Chinese and English understanding capabilities and can generate multi-resolution images based on text prompts...

What is Hunyuan DiT?

Hunyuan-DiT is a high-performance text-to-image diffusion Transformer model open-sourced by Tencent's Hunyuan team. It possesses fine-grained Chinese and English understanding capabilities and can generate high-quality images at multiple resolutions based on text prompts. Hunyuan-DiT employs an innovative network architecture, combining bilingual CLIP and multilingual T5 encoders. Trained and optimized through a carefully designed data pipeline, it supports multi-turn dialogue and can generate and refine images based on context. In the field of Chinese-to-image generation, Hunyuan-DiT has achieved a leading level among open-source models.

The main functions of Hunyuan DiT

  • Bilingual text-to-image generationThe Hunyuan DiT can generate images based on Chinese or English text prompts, which makes it a promising tool for cross-language image generation tasks.
  • Fine-grained understanding of Chinese elementsThe model has been specifically optimized for Chinese, enabling it to better understand and generate elements related to traditional Chinese culture, such as ancient Chinese poems, traditional Chinese clothing, and Chinese festivals.
  • Long text processing capabilitiesIt supports text input with up to 256 tags, enabling DiT to understand and generate images that match complex, long text descriptions.
  • Multi-size image generationThe Hunyuan-DiT can generate high-quality images at various scales to meet the needs of different applications, from social media posts to large-format printing.
  • Multi-turn dialogue and contextual understandingBy engaging in multi-turn dialogues with users, Hunyuan DiT can generate and iterate images based on dialogue history and contextual information, which enhances interactivity and creativity.
  • High consistency between images and textThe images generated by Hunyuan-DiT are highly consistent with the input text prompts in terms of content, ensuring that the images accurately reflect the intent and details of the text.
  • Artistic and creativeHunyuan DiT can not only generate common images, but also capture creative descriptions in text to generate artistic and creative image works.

Hunyuan DiT's official website entrance

The technical architecture of Hunyuan DiT

  • Dual text encoderHybrid DiT combines bilingual CLIP and multilingual T5 encoders to enhance the understanding and encoding capabilities of input text. The CLIP model was chosen for its strong ability to correlate images and text, while the T5 model was chosen for its capabilities in multilingual and text understanding.
  • Variational Autoencoder (VAE)Using a pre-trained VAE to compress images into a low-dimensional latent space helps the diffusion model learn the data distribution. The latent space of the VAE has a significant impact on the generation quality.
  • diffusion modelBased on the diffusion Transformer, Hunyuan DiT uses a diffusion model to learn the data distribution. This model combines textual conditions with the diffusion model through a cross-attention mechanism.
  • Improved generatorThe Diffusion Transformer has several improvements over the baseline DiT, such as using Adaptive Layer Normalization (AdaNorm) to enhance the execution of fine-grained text conditions.
  • Location codingMixed-Yuan DiT uses Rotational Position Embedding (RoPE) to simultaneously encode absolute and relative position dependencies, supporting multi-resolution training and inference.
  • Multimodal Large Language Model (MLLM): Used for reconstructing the original captions of image-text pairs to improve data quality. MLLM is fine-tuned to generate structured captions that incorporate world knowledge.
  • Data PipelineThis includes data acquisition, interpretation, stratification, and application. The validity of new data is checked through an iterative process known as the "data fleet."
  • Post-training optimizationOptimizations are performed during the inference phase to reduce deployment costs, including ONNX graph optimization, kernel optimization, and operation fusion.

Comparison of Hybrid DiT with other text-based graph models

To comprehensively compare the generative capabilities of HunyuanDiT with other models, the Hunyuan team constructed a test set with four dimensions, which was evaluated by more than 50 professional evaluators, including text-image consistency, elimination of AI artifacts, subject clarity, and aesthetics.

Model open source Text-image consistency (%) AI artifacts excluded (%) Subject clarity (%) Aesthetic appreciation (%) Overall score (%)
SDXL ✔ 64.3 60.6 91.1 76.3 42.7
PixArt-α ✔ 68.3 60.9 93.2 77.5 45.5
Playground 2.5 ✔ 71.9 70.8 94.9 83.3 54.3
SD 3 ✘ 77.1 69.3 94.6 82.5 56.7
Midjourney v6 ✘ 73.5 80.2 93.5 87.2 63.3
DALL-E 3 ✘ 83.9 80.3 96.5 89.4 71.0
Hunyuan-DiT ✔ 74.2 74.3 95.4 86.6 59.0
Back