PixArt-Σ - Huawei's text-based image model capable of generating 4K high-definition images
PixArt-Σ is a text-based graph model developed by researchers from Huawei Noah's Ark Lab, Dalian University of Technology, and the University of Hong Kong. It is based on the Diffusion Transformer architecture (DiT) and is specifically designed for extracting text from prompts...
What is PixArt-Σ?
PixArt-Σ is a text-to-image model based on the Diffusion Transformer (DiT) architecture, developed by researchers from Huawei Noah's Ark Lab, Dalian University of Technology, and the University of Hong Kong. It is specifically designed to directly generate high-quality images up to 4K resolution from text prompts. Building upon PixArt-α, this model integrates advanced elements and employs a weak-to-strong training method, resulting in a more powerful model that not only improves the fidelity of generated images but also enhances the alignment between the image and the text prompt. PixArt-Σ's generated images achieve aesthetic quality comparable to top-tier text-to-image products such as DALL·E 3 and Midjourney V6, and excel in adhering to text prompts.
PixArt-Σ Official Website Entry
- Official project homepage:https://pixart-alpha.github.io/PixArt-sigma-project/
- GitHub repository:https://github.com/PixArt-alpha/PixArt-sigma(Source code and model to be released)
- arXiv research paper:https://arxiv.org/abs/2403.04692
PixArt-Σ Features
- 4K resolution image generationPixArt-Σ can directly generate high-resolution images of 3840×2160 from text prompts without the need for further processing or other software intervention.
- High-fidelity text-to-image conversionWhen generating images, the model can accurately follow the content of the text description, ensuring a high degree of consistency between the image and the text.
- High-efficiency trainingPixArt-Σ employs a "weak-to-strong training" strategy, significantly improving training efficiency by introducing high-quality data and efficient token compression technology.
- Smaller model sizeAlthough PixArt-Σ can generate high-resolution images, its model has a relatively small number of parameters (0.6B parameters), which makes the model more efficient and easier to deploy.
How PixArt-Σ works
PixArt-Σ works based on the DiT (Diffusion Transformer) architecture, a deep learning method that combines diffusion models and Transformer architectures, specifically designed to convert text descriptions into corresponding images. The following is a detailed explanation of how PixArt-Σ works:
1. Pre-training phase:
During the pre-training phase, PixArt-Σ uses a large number of text-image pairs to learn the associations between text and images. This process typically includes the following steps:
- Text EncodingFirst, the input text description is transformed into an embedded representation of a series of tokens using a text encoder (such as Flan-T5). In PixArt-Σ, to handle more complex descriptions, the token length of the text encoder is extended to approximately 300 words.
- Conditional Feature ExtractionNext, these text embeddings, along with the conditional features of the image, are fed into a diffusion model. The conditional feature extractor is typically a pre-trained neural network used to extract features relevant to image generation from the text embeddings.
- diffusion processThe diffusion model learns the distribution of data by simulating a process of gradually adding noise to the data distribution. During training, the model learns how to recover a sharp image from noisy data.
- Iterative refinementThrough an iterative process, the model gradually learns how to generate high-quality images based on text descriptions. This process involves adjusting the model parameters so that the generated images become increasingly closer to real images.
2. Training from weak to strong:
PixArt-Σ employs a weak-to-strong training strategy, meaning the model is first trained on lower-quality datasets and then gradually introduced with higher-quality datasets and more complex training strategies. This approach helps the model effectively utilize new data and algorithms to enhance performance with limited resources.
3. High-quality training dataset (Internal-Σ):
PixArt-Σ uses a new, high-quality dataset containing high-resolution images and dense, accurate descriptions. This data provides the model with rich visual and textual information, helping to improve the quality of the generated images and their alignment with the textual descriptions.
4. Efficient token compression and weight initialization:
To generate high-resolution images, PixArt-Σ introduces key-value token compression techniques and a specially designed weight initialization scheme. These techniques enable the model to efficiently handle the generation of high-resolution images while reducing computational resource requirements.
5. Fine-tuning techniques:
During the fine-tuning phase, PixArt-Σ further improved the model's performance by replacing the more powerful variational autoencoder (VAE), expanding from low resolution to high resolution, and evolving the model from not using KV compression to using KV compression.
6. Generation Phase:
During the generation phase, the user inputs a text description, and the model iteratively generates an image that matches the description based on the learned text-image associations. The generated image is adjusted according to the content, style, and details of the text description to ensure that the final output image is both aesthetically pleasing and accurately reflects the intent of the text.