AB
AB
AiBoss

All Categories

project

BAGEL - ByteDance's open-source multimodal foundation model

BAGEL is an open-source multimodal base model developed by ByteDance, boasting 14 billion parameters, 7 billion of which are active. It employs a Hybrid Transformer Expert (MoT) architecture, capturing pixel-level and semantic parameters of images through two independent encoders...

What is BAGEL?

BAGEL is an open-source multimodal foundational model developed by ByteDance, boasting 14 billion parameters, 7 billion of which are active. It employs a Hybrid Transformer Expert (MoT) architecture, capturing pixel-level and semantic-level features of images through two independent encoders. BAGEL is trained following a "next-tag group prediction" paradigm, using massive amounts of multimodal labeled data for pre-training, including language, image, video, and network data. In terms of performance, BAGEL surpasses top-tier open-source visual language models such as Qwen2.5-VL and InternVL-2.5 in multimodal understanding benchmarks. Its text-to-image generation quality is comparable to SD3, and it outperforms many open-source models in image editing scenarios. BAGEL can perform free-form image editing, future frame prediction, 3D manipulation, and world navigation tasks.

BAGEL's main functions

  • Image and text fusion understandingBAGEL can deeply understand the relationship between images and text, accurately combining image content with text descriptions.
  • Video content comprehensionBAGEL can process video data, understand the dynamic information and semantic content within the video, and capture key information from the video for effective analysis.
  • Text to Image GenerationUsers can generate corresponding images by entering text descriptions. BAGEL can generate high-quality images that match the descriptions based on the text content.
  • Image editing and modificationBAGEL supports editing and modifying existing images. BAGEL can generate modified images based on instructions, enabling free-form image editing.
  • Video frame predictionBAGEL can predict future frames in a video. Based on the first few frames of the video, the model can generate the content of subsequent frames, restoring the integrity of the video.
  • 3D scene understanding and manipulationBAGEL can understand and manipulate 3D scenes. It can identify, locate, and manipulate 3D objects, such as moving objects in a virtual environment and changing their properties.
  • World NavigationBAGEL has world navigation capabilities, enabling path planning and navigation in virtual or real 3D environments.
  • Cross-modal retrievalBAGEL enables cross-modal retrieval, such as retrieving images or videos that match text descriptions, or retrieving related text information based on image content.
  • Multimodal fusion taskIn multimodal fusion tasks, BAGEL can effectively fuse data from different modalities (such as images, text, and speech) to generate comprehensive results.

BAGEL's technical principles

  • Dual encoder designBAGEL employs a Hybrid Transformer Expert (MoT) architecture, which comprises two independent encoders. One encoder processes pixel-level features of the image, while the other focuses on semantic-level features. This dual-encoder design allows the model to simultaneously capture both low-level details and high-level semantic information of the image.
  • Expert hybrid mechanismIn the MoT architecture, each encoder contains multiple expert modules. These expert modules can be viewed as small subnetworks, each expert focusing on processing a specific type of feature or task. By dynamically selecting the most suitable combination of experts during training, the model can handle complex multimodal data more efficiently.
  • TokenizationBAGEL transforms input multimodal data (such as images and text) into a series of tokens. For images, the model segments them into multiple patches, each of which is treated as a token; for text, each word or sub-word is also treated as a token.
  • Prediction taskThe training objective of the model is to predict the next set of tags. During training, the model will see a sequence of tags and attempt to predict the next tag.
  • Compression and LearningThrough this prediction task, the model can learn the inherent structure and relationships of multimodal data. The process of predicting the next label forces the model to compress and understand key information from the input data, improving its ability to understand and generate multimodal data.
  • Massive dataTo train BAGEL, ByteDance used trillions of multimodal tags from language, image, video, and web data for pre-training. The data covered a wide range of scenarios and domains, enabling the model to learn a broad range of multimodal features and patterns.
  • Optimization strategyDuring training, BAGEL employs advanced optimization strategies, such as mixed precision training and distributed training, to improve training efficiency and model performance.

BAGEL's project address

Application scenarios of BAGEL

  • Content creation and editingUsers can generate high-quality images by inputting text descriptions. BAGEL can also edit and modify existing images.
  • 3D scene generationBAGEL can generate 3D scenes, providing rich visual content for VR and AR applications.
  • Visual learningBAGEL can present complex concepts through images or videos, helping students to understand them better.
  • Creative Ad GenerationAdvertisers can use BAGEL to generate engaging advertising images and videos. For example, they can create creative advertising posters or short videos based on product features.
  • User interaction experienceOn e-commerce platforms, BAGEL can generate 3D models and virtual displays of products, enhancing the user's shopping experience.
Back