AB
AB
AiBoss

All Categories

project

UniTok - A unified visual word segmenter launched by ByteDance in collaboration with the University of Hong Kong and Huazhong University of Science and Technology.

UniTok is a unified visual word segmenter developed by ByteDance in collaboration with the University of Hong Kong and Huazhong University of Science and Technology. It supports both visual generation and understanding tasks. Based on multi-codebook quantization technology, it segments visual features into multiple small blocks, each using...

What is UniTok?

UniTok is a unified visual word segmenter developed by ByteDance in collaboration with the University of Hong Kong and Huazhong University of Science and Technology. It supports both visual generation and understanding tasks. Based on multi-codebook quantization technology, it segments visual features into multiple small blocks, each quantized using an independent sub-codebook, greatly expanding the representational power of discrete word segmentation and resolving the contradiction between detail capture and semantic understanding in traditional word segmenters. UniTok achieves a zero-shot classification accuracy of 78.6% on ImageNet with a reconstruction quality index (rFID) of only 0.38, significantly outperforming existing word segmenters. The multimodal large language model (MLLM) built on UniTok performs excellently in visual question answering and image generation tasks, demonstrating its powerful potential in multimodal tasks.

UniTok's main functions

  • Unified visual representationEncode images into discrete visual tokens. These tokens can be used in image generation tasks (such as text-to-image processing) as well as visual understanding tasks (such as visual question answering).
  • High-quality image reconstructionEfficient image reconstruction while preserving image details.
  • Semantic alignmentBy combining contrastive learning and reconstruction loss, we ensure that the generated visual tokens are aligned with the text descriptions, thereby improving visual understanding capabilities.
  • Supports Multimodal Large Language Model (MLLM)As the visual input module for multimodal large language models, it supports unified processing and generation of models in multimodal tasks.

UniTok's technical principles

  • Multicodebook QuantizationUniTok divides visual tokens into multiple small blocks, each quantized using an independent sub-codebook. For example, a 64-dimensional visual feature vector is divided into eight 8-dimensional blocks, each quantized based on a sub-codebook of 4096 codewords. In this way, UniTok's theoretical vocabulary can grow exponentially, greatly expanding the representational power of discrete tokens.
  • Attention decompositionUniTok replaces the traditional linear projection layer with a multi-head attention module for token decomposition, better preserving the semantic information in the original tokens and improving the expressive power of the decomposed features. UniTok uses causal attention to ensure compatibility with autoregressive generation tasks.
  • Unified training objectivesThe VQVAE-based reconstruction loss ensures accurate reconstruction of image details. The loss includes pixel-level reconstruction error, perceptual loss, discriminator loss, and vector quantization loss. UniTok introduces a CLIP-like contrastive loss to ensure that the generated visual tokens align with the text descriptions, improving visual understanding. The final total loss is a weighted sum of the reconstruction and contrastive losses; based on this approach, UniTok can simultaneously optimize both generation and understanding tasks.
  • Ensemble of Multimodal Large Language Models (MLLM)The generated visual tokens are mapped to the token space of a multimodal large language model based on an MLP projection layer, achieving unified processing of vision and language. To simplify the input of MLLM, UniTok merges tokens generated by multiple subcodebooks into a single visual token and inputs it into MLLM. When visual token prediction is needed, MLLM autoregressively predicts multiple subcodebook tokens corresponding to the next position, achieving efficient visual generation.

UniTok's project address

UniTok application scenarios

  • Visual input for multimodal modelsAs the visual module of a multimodal large language model, it helps the model process text and image information simultaneously, improving overall performance.
  • High-quality image generationIt generates highly detailed images based on text descriptions, making it suitable for creative design, advertising production, and other fields.
  • Visual Question Answering and ComprehensionIt assists models in understanding image content and answering vision-related questions, and can be used in education, medical image analysis, and other fields.
  • Multimodal content creationQuickly generate graphic and text content for use in news reports, social media, etc., improving creation efficiency.
  • Cross-modal retrieval and recommendation: To improve the user experience of e-commerce platforms and multimedia platforms by performing searches and recommendations based on text or images.
Back