LDGen - Li Auto's multilingual text-to-image generation technology

What is LDGen?

LDGen is an innovative text-to-image synthesis technology that improves the quality and semantic consistency of text-to-image generation by combining Large Language Models (LLMs) with diffusion models. Through hierarchical title optimization and human-instruction techniques, it extracts precise semantic information from the text and achieves efficient alignment and interaction between LLMs and image features based on a lightweight adapter. LDGen supports zero-shot multilingual generation and can generate high-quality images from text descriptions in multiple languages, significantly outperforming traditional methods.

LDGen's main functions

• Multilingual zero-shot generationBy combining a large language model (LLM) with existing text-to-image diffusion models, LDGen enables zero-shot multilingual text-to-image generation. Even with only English cues during training, the model can generate images with descriptions in multiple languages, significantly improving cross-lingual generation capabilities.
• Language representation optimizationBy employing layered caption optimization and manual instruction techniques, more accurate semantic information is extracted, enhancing the semantic alignment between text and images. This improves the semantic consistency of generated images and avoids false information caused by erroneous instructions.
• Improve generation qualityIn multiple experiments, LDGen outperformed baseline models and other enhancement methods, such as PixArt- and ELLA, in terms of instruction compliance and image aesthetic quality.

LDGen's technical principles

• Language representation strategiesLDGen employs layered caption optimization and human instruction techniques to extract more accurate semantic information. By generating captions of varying lengths and combining them with human instruction optimization, the model can better capture the hierarchical structure of image content while avoiding false information caused by erroneous instructions.
• LLM Alignment ModuleTo align the features of LLM with those of existing diffusion models, LDGen designed a lightweight adapter. The adapter achieves efficient feature alignment by adjusting the feature space of the LLM output to match the feature space of text encoders such as T5.
• Cross-modal refinerLDGen introduces a cross-modal refiner module to enhance the interaction between LLM features and image features. This module optimizes LLM feature representations through self-attention and cross-attention mechanisms, further improving semantic alignment between text and images.
  • Self-attention mechanism: Optimize the internal representation of LLM features.
  • Cross-Attention MechanismUsing LLM features as queries and image features as keys and values, it promotes deep interaction between text and images.
  • Learnable Scaling FactorsDuring training, the original features and optimized features are dynamically balanced to ensure a seamless transition from pre-trained weights to new features.
• High-efficiency training strategies and efficiencyLDGen significantly reduces computational requirements through staged training.
  • Feature alignment stageThe LLM alignment module was trained using approximately 80 million text data points.
  • Fine-tuning stageFine-tuning was performed using 24 million text-image pairs at 512 resolution.
  • High-resolution training phaseTraining continued using 14 million data points at 1024 resolution. The entire training process required only about 120 A100 GPU days, reducing computational resources by about 74% compared to PixArt-α.

LDGen's project address

• Project official website:https://zrealli.github.io/LDGen/
• Github repository:https://github.com/zrealli/LDGen
• arXiv technical paper:https://arxiv.org/pdf/2502.18302

Application scenarios of LDGen

• Artistic Creation and DesignArtists and designers can use LDGen to generate high-quality images from creative descriptions, accelerating the creative process. It quickly transforms text descriptions into visual content, helping creators explore different design directions.
• Advertising and MarketingIn the advertising and marketing field, LDGen can quickly generate engaging ad images and social media posts based on brand style or market trends. Generating images directly from text descriptions enhances the appeal and personalization of promotional materials.
• Media and EntertainmentLDGen can be used for concept art creation in film, game, and animation production, generating initial visual representations of scenes and characters. In film production, it can generate initial sketches of special effects scenes, helping directors and designers quickly preview scene layouts.
• educateIn education, LDGen can help students and teachers create teaching materials, such as historical scene recreations or visualizations of scientific concepts. Generating images from text descriptions allows for a more intuitive presentation of complex concepts.
• e-commerceOnline retailers can use LDGen to generate visual displays of their products, showcasing how clothing looks in different environments or on different models. This helps merchants quickly generate high-quality product images, enhancing the user experience.