AB
AB
AiBoss

All Categories

project

DiT - A diffusion model based on the Transfomer architecture

DiT (Diffusion Transformers) is a novel diffusion model proposed by William Peebles (one of the research leaders at Sora) and Xie Saining, an assistant professor at New York University. It combines a denoising diffusion probability model (...

What is DiT?

DiT (Diffusion Transformers) is a novel diffusion model proposed by William Peebles (one of the lead developers of Sora) and Xie Saining, an assistant professor at New York University. It combines denoising diffusion probabilistic models (DDPMs) with the Transformer architecture. A diffusion model is a generative model that generates new samples by simulating a progressive denoising process of data. The core idea of DiT is to use a Transformer as the backbone network of the diffusion model, instead of traditional convolutional neural networks (such as U-Net), to handle the latent representations of images. Recently, with the popularity of OpenAI's video generation model Sora, DiT has received widespread attention as one of the technological foundations behind Sora.

In DiT, images are first compressed into smaller latent representations using an autoencoder (such as a variational autoencoder, VAE), and then a diffusion model is trained in this latent space. This reduces the computational cost required to train the diffusion model directly in the high-resolution pixel space. The DiT model processes these latent representations through the Transformer's self-attention mechanism, enabling it to capture long-range dependencies in the image and thus generate high-quality images.

DiT's official website entrance

DiT's technical principles

  1. Data preparation:
    • The input image is encoded into a representation in the latent space using a pre-trained variational autoencoder (VAE). This latent representation is typically a low-dimensional representation of the image; for example, encoding a 256×256×3 RGB image into a 32×32×4 latent vector.
    • This latent representation was then used as input to the DiT model.
  2. Patchification:
    • The latent representation of the input is first divided into a series of small patches through a process called "patchify," with each patch corresponding to an input token for the Transformer model. This process is similar to segmenting an image into small pieces so that the model can process them piece by piece.
    • Each segment is transformed into a fixed-dimensional vector through linear embedding, and then positional embeddings are added so that the model can understand the position of the segment in the image.
  3. Transformer Blocks module:
    • The input labeled sequence is processed through a series of Transformer blocks. These blocks include components such as self-attention layers, feedforward neural networks, and layer normalization.
    • In DiT, researchers experimented with different Transformer block designs, including Adaptive Layer Normalization (adaLN), Cross-Attention, and In-Context Conditioning, to handle conditional information such as timesteps and class labels.
  4. Conditional diffusion process:
    • During training, the DiT model learns the backdiffusion process, which is the process of recovering a sharp image from noisy data. This process involves the statistical properties of the predicted noise, such as the mean and covariance.
    • Using variational lower bound (VLB) to train the model involves minimizing the mean squared error (MSE) between the predicted noise and the actual noise.
  5. Sample generation:
    • After training, new images can be generated using the DiT model. First, a latent representation is sampled from a standard normal distribution. Then, noise is gradually removed through the DiT model's reverse diffusion process, and finally, the image is decoded back into pixel space to obtain the generated image.
  6. Scalability:
    • The scalability of the DiT model is reflected in its ability to increase computational cost (Gflops) by adding layers, width, or the number of input labels to the Transformer, thereby improving the quality of the generated images. This scalability allows the DiT model to generate high-quality images at different resolutions and complexities.

The DiT model leverages the powerful expressive capabilities of the Transformer and the generative capabilities of the diffusion model to achieve efficient and high-quality output in image generation tasks.

Key features of DiT

  • Transformer-based architecture:DiT uses Transformer as its core architecture, which enables the model to handle sequential representations of images and capture long-distance dependencies in images through a self-attention mechanism.
  • Latent space operations:DiT is trained in a latent space, which is generally more efficient than training directly in pixel space. By using a variational autoencoder (VAE) to encode images into the latent space, DiT reduces computational complexity.
  • Scalability:DiT exhibits excellent scalability, significantly improving the quality of generated images by increasing the computational cost of the model (measured in Gflops). This scalability allows DiT to generate images at different resolutions and complexities.
  • Conditional generation capability:DiT supports conditional generation, enabling it to generate images of specific categories based on given category labels. This capability makes DiT extremely useful for domain-specific image generation tasks.
  • Adaptive Layer Normalization (adaLN):DiT uses adaptive layer normalization, a normalization method used in the Transformer block, which learns to adjust the parameters of layer normalization, thereby improving the model's expressive power and training efficiency.
  • Multiple Transformer Block Designs:DiT explores different Transformer block designs, including Adaptive Layer Normalization (adaLN), Cross-Attention, and In-Context Conditioning, to handle conditional information.
  • Efficient training process:DiT exhibits high stability during training, consistently achieving high performance even without learning rate warm-up and regularization techniques.
  • Diversity and quality of generated images:DiT can generate images with high visual quality and diversity. By adjusting the guidance strength of class-conditional generation, a trade-off can be struck between sharpness and diversity in the generated images.
  • High computational efficiency:When generating images, DiT can achieve high computational efficiency while maintaining high image quality. This makes DiT attractive even in resource-constrained environments.
  • Application potential:DiT has broad application potential in the field of image generation, including artistic creation, game development, virtual reality, data augmentation, and especially in scenarios that require the generation of high-quality images.
Back