ERNIE-Image - A text-based image model open-sourced by Baidu Wenxin.

What is ERNIE-Image?

ERNIE-Image is an open-source 8B-parameter text-to-image model from the Baidu Wenxin team. Based on the Diffusion Transformer architecture, it emphasizes high controllability and accurate rendering of long text. The model can accurately generate complex visual content such as bilingual (Chinese and English) posters, comics, and infographics, solving the problem of blurred text illusions. ERNIE-Image offers two versions: a standard version (50 high-quality steps) and a Turbo version (8 quick steps). It only requires 24GB of video memory and can run locally, providing designers and developers with an out-of-the-box Chinese scene image generation solution.

Main functions of ERNIE-Image

• Highly controllable generationThe model supports complex structured layout control and can accurately arrange the spatial relationships between multiple objects, achieving a high score of 0.8856 in the GENEval benchmark test.
• Long text renderingSpecifically optimized for accurate rendering of long bilingual (Chinese and English) text in images, it achieved a score of 0.9733 in the LongTextBench test, making it particularly suitable for poster and comic creation.
• Dual version modeIt offers two inference modes: a standard version (50 steps of high-quality rendering) and a Turbo version (8 steps of rapid distillation), which respectively meet the needs of meticulous creation and rapid iteration.
• End-to-end processingIt features a complete built-in image processing workflow, covering four stages: generation, editing, compositing, and enlargement, supporting end-to-end creation from sketch to finished product.
• Enhanced intelligent promptsEquipped with a lightweight Prompt Enhancer module, it can automatically expand short user input into a well-structured and detailed generated description.

ERNIE-Image's technical principles

• Single-stream DiT architectureIt adopts Diffusion Transformer to uniformly encode text and image signals, and achieves cross-modal depth alignment with an 8B parameter scale, thereby improving generation consistency and the ability to understand complex scenes.
• Chinese typography optimizationIt is specifically trained for Chinese character structure and typesetting logic, improves attention mechanism and positional encoding, and solves the problems of text illusion, ambiguity and structural error in traditional diffusion models.
• Distillation Acceleration TechnologyThe Turbo version transfers the capabilities of the teacher model to the lightweight 8-step student model through knowledge distillation, significantly improving inference speed while maintaining usable quality.
• Consumer hardware adaptationIt only requires 24GB of video memory to run locally and natively supports 1024×1024 resolution, lowering the deployment threshold.

How to use ERNIE-Image

• Environmental preparationEnsure your local environment has an NVIDIA graphics card with 24GB or more of video memory, and that Python 3.8+ is installed and configured. transformers>=4.50.0,torch,diffusers Dependency libraries, etc.
• Load the standard version model:pass DiffusionPipeline.from_pretrained("Baidu/ERNIE-Image", torch_dtype=torch.float16, trust_remote_code=True) Load the 8B parameter model and move it to the CUDA device for a 50-step high-quality generation scenario.
• Load Turbo version model:pass DiffusionPipeline.from_pretrained("Baidu/ERNIE-Image-Turbo", torch_dtype=torch.float16, trust_remote_code=True) Load the distilled version of the model, supporting 8-step rapid reasoning to meet the needs of draft iteration.
• Execution of basic text-based diagrams: call pipe(prompt="描述文本", num_inference_steps=50, guidance_scale=7.5, height=1024, width=1024) Pass in the text prompt and parameters, and call the returned image object. .images[0] Get the results and save them.
• Enhance with intelligent promptsThe system's built-in Prompt Enhancer automatically expands short inputs into structured descriptions, improving generation quality without the need to manually write complex prompts.
• Call the image editing function:use pipe.edit(prompt="修改指令", image=原图, mask=遮罩) The API performs partial redrawing of a specified area, enabling precise content modification based on a mask.
• Enable low video memory modeAdd when video memory is less than 24GB. device_map="auto" Or call pipe.enable_sequential_cpu_offload() The implementation layer is offloaded, and inference is performed on a consumer-grade graphics card.

Key information and usage requirements for ERNIE-Image

• Basic SpecificationsBased on an 8B parameter single-stream Diffusion Transformer architecture, it supports the generation of 1024×1024 resolution images.
• Dual version designThe standard version is a 50-step SFT model for final high-quality rendering; the Turbo version is an 8-step distillation model for rapid draft iteration.
• Core CompetenciesAccurate rendering of long bilingual text (LongTextBench score 0.9733), highly controllable layout generation (GENEval score 0.8856), and support for a four-stage workflow of generation/editing/compositing/enlarging.
• Open source licenseIt uses the Apache-2.0 license, supports commercial applications and secondary development, and can be deployed locally with consumer-grade graphics cards (24GB video memory).
• Hardware configurationA graphics card with 24GB or more of video memory (such as the RTX 4090) is required to meet the inference requirements of the 8B parameter model.
• Software environmentRequires Python 3.8 or later, and installation. transformers>=4.50.0,torch,diffusers Core dependency libraries, etc.

ERNIE-Image's core advantages

• Accurate rendering of long Chinese textsSpecifically optimized for accurate rendering of long bilingual (Chinese and English) text in images, achieving a LongTextBench score of 0.9733. It completely solves problems such as text blurring and structural illusion in traditional diffusion models, making it particularly suitable for creative scenarios requiring precise layout, such as posters, comics, and infographics.
• Highly controllable structured generationIt supports precise control over complex page layouts and spatial relationships between multiple objects, achieving a GENEval benchmark score of 0.8856. It can accurately arrange element positions based on structured descriptions and generate multi-object combinations that strictly adhere to instructions.
• Flexible adaptation to dual versionsIt offers a standard version (50-step SFT) for final high-quality rendering and a Turbo version (8-step distillation) for rapid draft iteration. Users can flexibly choose the inference mode according to their quality or speed requirements.
• Consumer-grade hardware can be deployedIt requires only 24GB of video memory to run 8B parameter models locally on a single consumer-grade graphics card (such as the RTX 4090), and provides optimization solutions such as layer unloading, significantly reducing the barrier to entry for enterprise-level text generation capabilities.

ERNIE-Image project address

• Project official website: https://ernie.baidu.com/blog/posts/ernie-image/
• HuggingFace model library:
  • https://huggingface.co/baidu/ERNIE-Image
  • https://huggingface.co/Baidu/ERNIE-Image-Turbo

Comparison of ERNIE-Image products with similar competitors

Application scenarios of ERNIE-Image

• Commercial poster designWith precise long text rendering capabilities, it generates Chinese and English posters containing brand slogans, promotional information, and product parameters. It is suitable for e-commerce promotions, event promotions, outdoor advertising, and other scenarios. The text is clear and readable and requires no post-processing.
• Comic and illustration creationIt supports complex panel layouts and multi-character scene control, and can generate comic pages with dialogue bubbles, narration text, and onomatopoeia, realizing one-stop production from draft to final version.
• Infographics and Data VisualizationTransform structured data descriptions into visual infographics that include charts, labels, and explanatory text, suitable for creating report covers, teaching materials, long-form social media posts, and other content.
• E-commerce product imagesHigh controllability allows for precise arrangement of the spatial position of the product body, background elements, price tags, and selling point copy, quickly generating product main images and detail pages that conform to platform specifications.
• Publishing and Printing MaterialsAccurate Chinese character rendering ensures that the text on printed materials such as book covers, magazine pages, and brochures is accurate, avoiding the text garbled problems of traditional AI-generated images.