SimpleAR - An image generation model jointly developed by Fudan University and ByteDance's Seed team.
SimpleAR is a pure autoregressive image generation model jointly developed by the Vision and Learning Lab at Fudan University and the ByteDance Seed team. Employing a concise autoregressive architecture, it achieves high-quality image generation by optimizing the training and inference processes...
What is SimpleAR?
SimpleAR is a pure autoregressive image generation model jointly developed by the Vision and Learning Lab at Fudan University and the ByteDance Seed team. Employing a concise autoregressive architecture, it achieves high-quality image generation by optimizing the training and inference processes. SimpleAR can generate 1024×1024 resolution images with only 500 million parameters, achieving excellent results in benchmark tests such as GenEval. Training employs a three-stage approach of "pre-training – supervised fine-tuning – reinforcement learning," significantly improving text following ability and generation quality. SimpleAR is compatible with existing acceleration technologies, reducing inference time to less than 14 seconds.
SimpleAR's main functions
- High-quality text-to-image generationSimpleAR is a purely autoregressive visual generation framework that can generate high-quality images with a resolution of 1024×1024 using only 500 million parameters, achieving an excellent score of 0.59 in benchmark tests such as GenEval.
- Multimodal fusion generationIt treats text and visual tokens equally, integrating them into a unified Transformer architecture, supporting multimodal modeling, and enabling better text-guided image generation.
SimpleAR's technical principles
- Autoregressive generation mechanismSimpleAR employs a classic autoregressive generation approach, progressively generating image content through "next token prediction." This mechanism decomposes the image into a series of discrete tokens, then predicts each token individually to construct the complete image.
- Multimodal fusionSimpleAR integrates text encoding and visual generation into a decoder-only Transformer architecture. This improves parameter utilization efficiency, better supports joint modeling between text and visual modalities, and enables the model to more naturally understand and generate images corresponding to text descriptions.
- Three-stage training method:
- Pre-training: Learn general visual and language patterns through large-scale data pre-training.
- Supervised fine-tuning (SFT)Based on pre-training, supervised learning is used to further improve the generation quality and instruction following ability.
- Reinforcement Learning (GRPO)Post-training based on a simple reward function (such as CLIP) optimizes the aesthetics and multimodal alignment of the generated content.
- Inference acceleration technologySimpleAR optimizes the inference process through techniques such as vLLM, significantly reducing image generation time. For example, a model with 0.5B parameters can generate a high-quality image with a resolution of 1024×1024 in 14 seconds.
- Visual tokenizer selectionSimpleAR uses Cosmos as its visual tokenizer, which has limitations in low-resolution images and detail reconstruction, and there is still room for improvement.
SimpleAR's project address
- Github repository:https://github.com/wdrink/SimpleAR
- HuggingFace model library:https://huggingface.co/papers/2504.11455
- arXiv technical paper:https://arxiv.org/pdf/2504.11455
SimpleAR Application Scenarios
- Creative DesignSimpleAR can help designers quickly generate high-quality images for advertising design, poster production, art creation, and more.
- Virtual scene constructionIt generates virtual scenes through text descriptions, providing material for game development, virtual reality (VR), and augmented reality (AR) applications.
- Multimodal machine translationSimpleAR's multimodal fusion capabilities can be used to combine image information with text translation, improving the accuracy and richness of the translation.
- Video description generation: By combining image generation with video content, it generates detailed descriptive text for the video.
- Augmented Reality (AR) and Virtual Reality (VR)SimpleAR can generate virtual images that blend seamlessly with real-world scenes, suitable for applications such as industrial maintenance, educational demonstrations, and tourist navigation. It creates high-quality virtual environments and objects for virtual reality applications, enhancing the user experience.
- Image enhancement and restorationSimpleAR can be used to enhance the details of low-resolution images and improve image quality. It can also be used to repair images by generating image content of missing or damaged parts.