BLIP3-o - A multimodal model developed by Salesforce Research and other organizations.

What is BLIP3-o?

BLIP3-o is an innovative multimodal model developed by Salesforce Research and other institutions. It combines the inference and instruction-following capabilities of an autoregressive model with the powerful generative capabilities of a diffusion model. Based on semantically rich CLIP image features, the model does not rely on traditional VAE features or raw pixels, demonstrating excellent performance in both image understanding and generation. BLIP3-o employs a sequential pre-training strategy, first training for image understanding and then for image generation, preserving image understanding capabilities while developing powerful image generation abilities. The model has achieved excellent results on multiple image understanding and generation benchmarks and is completely open-source, including the code, model weights, pre-training dataset, and instruction tuning dataset.

Main functions of BLIP3-o

• Text to textGenerate descriptive text related to the image.
• Image to TextIt can understand the input image and generate descriptive text, supporting a variety of image understanding tasks, such as visual question answering (VQA) and image classification.
• Text to ImageGenerate high-quality images based on the input text description.
• Image to ImageEdit and modify the input image to generate a new image.
• Hybrid trainingIt supports hybrid training for image generation and understanding tasks, improving the overall performance of the model.

Technical principles of BLIP3-o

• Combination of autoregressive and diffusion modelsThe autoregressive model generates intermediate visual features, capturing semantic information from the text description. The diffusion model generates the final image. The diffusion model generates images based on progressive noise removal, producing high-quality and diverse images.
• CLIP Feature DiffusionThe CLIP model is used to encode images, generating semantically rich feature vectors. These feature vectors are more compact and information-rich than traditional VAE features. A diffusion model is used to model CLIP features, generating feature vectors similar to those of the target image, thus achieving high-quality image generation.
• Sequential pre-training strategyFirst, the model is pre-trained for an image understanding task to ensure it possesses strong image understanding capabilities. Based on this pre-training, the weights of the autoregressive model are frozen, and only the diffusion model is trained to achieve efficient image generation.
• Stream matching loss functionUsing the flow matching loss function to train a diffusion model allows the loss function to better capture the distribution of image features, generating higher-quality images. The flow matching loss function introduces randomness, enabling the model to generate diverse images, not just a single output.
• Instructions to adjust datasetBased on the diverse cues generated by GPT-4o, a dataset containing 60k high-quality cue image pairs was created to fine-tune the model, improving instruction compliance and visual aesthetic quality.

BLIP3-o project address

• GitHub repository:https://github.com/JiuhaiChen/BLIP3o
• HuggingFace model library:https://huggingface.co/BLIP3o
• arXiv technical paper:https://arxiv.org/pdf/2505.09568

Application scenarios of BLIP3-o

• Image generation and editingGenerate or modify images based on text descriptions to assist in design and creative work.
• Visual Q&AIt can understand image content and answer related questions, and can be used for education and intelligent customer service.
• Multimodal dialogueCombining images and text for dialogue enhances the interactive experience.
• Image annotation and classificationAutomatically generate image tags and classify them to optimize image management.
• Art and CreativityGenerate artistic images, inspire creativity, and meet personalized needs.