AB
AB
AiBoss

All Categories

project

HART - An autoregressive visual generative model developed by MIT

HART (Hybrid Autoregressive Transformer) is an autoregressive visual generative model developed by a research team at MIT. It can directly generate high-resolution images of 1024×1024 pixels, with quality comparable to diffusion models. H...

What is HART?

HART (Hybrid Autoregressive Transformer) is an autoregressive visual generative model developed by a research team at MIT. It can directly generate high-resolution images of 1024×1024 pixels, with quality comparable to diffusion models. HART is based on a hybrid tokenizer technique, decomposing the continuous latent representation of the autoencoder into discrete and continuous tokens. Discrete tokens capture the main structure of the image, while continuous tokens focus on details. HART's lightweight residual diffusion module uses only 37 million parameters, significantly improving computational efficiency. On the MJHQ-30K dataset, HART reduced the reconstructed FID from 2.11 to 0.30 and the generated FID from 7.85 to 5.38, a 31% improvement. In terms of throughput, it is 4.5-7.7 times higher than existing diffusion models, and the MAC is reduced by 6.9-13.4 times.

HART's main functions

  • High-resolution image generationIt can directly generate high-resolution images of 1024×1024 pixels to meet the needs of high-quality visual content.
  • Image quality improvementBased on hybrid tokenizer technology, HART surpasses traditional autoregressive models in image reconstruction and generation quality, and is comparable to diffusion models.
  • Computational efficiency optimizationWhile maintaining high image quality, it significantly improves computational efficiency and reduces training costs and inference latency.
  • Autoregressive modelingBased on the autoregressive method, images are generated step by step, supporting more precise control over the generation process.

HART technology principles

  • Hybrid TokenizerThe core of HART is the Hybrid Tokenizer, which decomposes the continuous latent representation of the autoencoder into discrete tokens and continuous tokens. Discrete tokens are responsible for capturing the main structure of the image, while continuous tokens focus on the details.
  • Discrete Autoregressive ModelThe discrete part is modeled by a discrete autoregressive model with scalable resolution, which supports the model in generating images at different resolutions.
  • Lightweight residual diffusion moduleThe continuous part is learned by a lightweight residual diffusion module with only 37 million parameters, which helps improve the efficiency of the model.
  • Efficiency and performance balanceHART outperforms existing diffusion models in FID and CLIP scores, improves throughput by 4.5-7.7 times, and reduces MAC by 6.9-13.4 times, achieving a good balance between efficiency and performance.
  • Autoregressive generationHART is based on an autoregressive method that generates images step by step, with each step based on the output of the previous step, allowing the model to gradually refine image details during the generation process.

HART project address

HART application scenarios

  • Digital art creationArtists and designers generate high-quality digital artworks, including illustrations, concept art, and visual effects.
  • Game developmentIn game design, this involves generating high-resolution images of game assets such as characters, environments, and props.
  • Film and video productionGenerate movie posters, concept art, or backgrounds and effects for video content.
  • Advertising and MarketingThe marketing team can quickly generate attractive advertising images and marketing materials.
  • Social media contentUsers generate personalized images and visual content for social media platforms.
Back