GigaTok - HKU and ByteDance jointly launch a visual word segmenter for autoregressive image generation

What is GigaTok?

GigaTok is a visual word segmenter for autoregressive image generation with 3 billion parameters. Through semantic regularization, it aligns the segmenter's features with the semantic features of a pre-trained visual encoder (such as DINOv2), effectively constraining the potential space complexity and resolving the trade-off between reconstruction and generation quality when the visual word segmenter scales up. GigaTok employs a one-dimensional word segmenter architecture to improve scalability, prioritizes decoder expansion for efficient allocation of computational resources, and introduces entropy loss to stabilize the training of large-scale models.

GigaTok's main functions

• High-quality image reconstructionGigaTok has successfully scaled up the visual word segmenter to 3 billion parameters, significantly improving image reconstruction quality. Through semantic regularization, it aligns the word segmenter features with the semantic features of a pre-trained visual encoder (such as DINOv2), preventing excessive potential spatial complexity during the scaling process.
• Improve downstream production performanceGigaTok excels in downstream autoregressive generation tasks, resolving the trade-off between reconstruction and generation quality in traditional methods. Through semantic regularization and optimized extension strategies, GigaTok achieves higher quality and better generalization capabilities in generation tasks.
• Optimize representation learningGigaTok significantly improves the representation learning quality of downstream autoregressive models by scaling up the visual word segmenter and incorporating semantic regularization. Experiments show that models trained using GigaTok achieve a significant improvement in linear detection accuracy.
• Innovative expansion strategiesGigaTok proposes a one-dimensional word segmenter architecture, which offers better scalability compared to traditional two-dimensional word segmenters. It prioritizes expanding the decoder and introduces entropy loss to stabilize the training of large-scale models.

GigaTok's technical principles

• Hybrid architecture designGigaTok employs a hybrid architecture combining CNNs and Transformers to achieve efficient feature extraction and latent space encoding. The encoder progressively downsamples the image using CNN blocks, then generates discrete latent codes through Transformer layers and vector quantizers. The decoder reconstructs the image from the latent codes using Transformer layers and a CNN decoder. It supports both one-dimensional (1D) and two-dimensional (2D) word segmenters, with the 1D segmenter offering superior scalability.
• Semantic regularizationTo address the issue of excessive latent space complexity when scaling up the word segmenter, GigaTok introduces semantic regularization techniques. This constrains the complexity of the latent space by aligning the features of the word segmenter with semantically consistent features of a pre-trained visual encoder (such as DINOv2). Specifically, through a contrastive learning framework, the intermediate features of the word segmenter are forced to align with the semantic space of the pre-trained model, maintaining generation quality while scaling the model.
• Asymmetric expansion strategyGigaTok prioritizes expanding the decoder when scaling both the encoder and decoder. This allows for more efficient allocation of computing resources while avoiding potential spatial instability caused by overly complex encoders.
• Entropy lossGigaTok introduces entropy loss to stabilize the training of large-scale word segmenters. Entropy loss ensures model stability during training by encouraging higher codebook utilization, avoiding training crashes caused by increased complexity.

GigaTok's project address

• Project official website:https://silentview.github.io/GigaTok/
• Github repository:https://github.com/SilentView/GigaTok
• arXiv technical paper:https://arxiv.org/pdf/2504.08736

GigaTok Application Scenarios

• Image generation and synthesisGigaTok excels in autoregressive image generation, producing high-quality images. It can be used in fields such as artistic creation, game development, and virtual reality to help users quickly generate image content that meets their needs.
• Image Editing and EnhancementGigaTok can be used for image editing tasks, such as seamlessly integrating foreground objects into background images.
• Data Augmentation and Pre-trainingGigaTok provides high-quality pre-training data for machine learning models through its efficient image segmentation and reconstruction capabilities.
• Multimodal learningGigaTok's semantic regularization technology enables it to be combined with text generation models to generate images from text. This multimodal capability can be applied to areas such as intelligent content creation and virtual assistants.
• Medical Image ProcessingGigaTok's high-fidelity image reconstruction capabilities can be applied to medical image generation and processing, such as generating high-quality medical images for diagnosis or research.