CogView-4 - A text-to-image generation model launched by Zhipu AI

What is CogView4?

CogView4, an open-source text-to-image generation model developed by Zhipu, boasts 6 billion parameters and supports native Chinese input and Chinese text generation. The model ranked first in the DPG-Bench benchmark test, achieving state-of-the-art (SOTA) performance among open-source text-to-image models. CogView4 is the first image generation model to comply with the Apache 2.0 license, supporting image generation at any resolution and capable of generating high-quality images from complex text descriptions.

Main functions of CogView4

• Supports Chinese and English bilingual inputCogView4 is the first open-source text-to-image model that supports the generation of Chinese characters, and can generate high-quality images based on Chinese or English prompts.
• Arbitrary resolution image generationThis model supports generating images with resolutions between 512×512 and 2048×2048, meeting the creative needs of different scenarios.
• powerful semantic alignment capabilitiesIn the DPG-Bench benchmark test, CogView4 ranked first in overall score, demonstrating its superior performance in complex semantic alignment and instruction following.
• Chinese character paintingCogView4 has been specially optimized for Chinese text generation, which can naturally integrate Chinese characters into images, making it suitable for creative fields such as advertising and short videos.
• Memory optimization and efficient inferenceBy employing techniques such as model CPU offloading and text encoder quantization, CogView4 significantly reduces GPU memory usage and improves inference efficiency.

CogView4's technical principles

• Architecture DesignCogView4 employs an architecture combining a diffusion model and a Transformer. The diffusion model generates images by progressively removing noise, while the Transformer handles the joint representation of text and images. The model uses a 6B parameter configuration, supporting text input of arbitrary length and image generation at arbitrary resolution.
• Text Encoder and TokenizerCogView4 uses a bilingual (Chinese and English) GLM-4 encoder, enabling it to handle complex semantic alignment tasks. Text is transformed into embedding vectors via a tokenizer, which are then combined with the latent representation of the image.
• Image encoding and decodingThe image is encoded into a representation of the latent space using a Variational Auto-Encoder (VAE), and the final image is generated through progressive denoising using a diffusion model. This enables the model to efficiently handle image generation tasks.
• Diffusion process and denoisingThe core of the diffusion model is to gradually generate an image through a series of denoising steps. CogView4 uses FlowMatch Euler Discrete Scheduler to control the denoising process, and users can adjust the number of denoising steps (num_inference_stepsTo balance the generation quality and speed.
• Multi-stage training strategyCogView4 employs a multi-stage training strategy, including base resolution training, general resolution training, high-quality data fine-tuning, and human preference alignment training. This ensures the high quality and aesthetic appeal of the generated images.
• Optimization and EfficiencyTo improve training and inference efficiency, CogView4 employs memory optimization techniques, such as model CPU offloading and text encoder quantization. The model supports the Apache 2.0 license, facilitating further development by the open-source community.

CogView4 project address

• Github repository:https://github.com/THUDM/CogView4
• HuggingFace model library:https://huggingface.co/THUDM/CogView4-6B

Official case study of CogView4

• Extra Long Story (Four-Panel Comic):Please generate a four-panel comic strip containing four scenes, using an anime illustration style. The main characters are: Xiao Ming: A human boy with a brave heart, wielding a sword and wearing simple warrior clothing. Princess: A beautiful and elegant human woman, wearing magnificent princess attire, imprisoned in the monster's lair. King: A dignified and benevolent human man, wearing magnificent royal robes and sitting on the kingdom's throne. Flame Dragon: A monster covered in flame-like scales, breathing fire, and enormous in size. Dark Lord: A gigantic monster shrouded in darkness, possessing powerful magical abilities.

• CogView4 can seamlessly integrate Chinese and English characters into the visuals, making poster and copywriting illustration creation more convenient.

• They are good at understanding and following Chinese prompts, such as being able to draw the artistic conception in classical Chinese poems.

Application Scenarios of CogView4

• Advertising and Creative DesignCogView4 can naturally integrate Chinese and English characters into the screen, generating high-quality posters, copywriting and illustrations, etc., to meet the needs of the advertising and creative design fields.
• Educational resource generationThe model can generate teaching illustrations, scientific illustrations, etc. for the education field, helping students to better understand and absorb knowledge.
• Children's picture book creationCogView4 can generate illustrations suitable for children's picture books, stimulating children's imagination.
• E-commerce and content creationGenerate high-quality product images, advertising posters, etc., to help businesses quickly create attractive visual content.
• Personalized customizationGenerate customized image content based on user needs to enhance user experience.