xGen-MM - An open-source multimodal AI model launched by Salesforce

What is xGen-MM?

xGen-MM is an open-source multimodal AI model launched by Salesforce. It has the ability to process interleaved data and can simultaneously understand and generate multiple data types such as text and images. By learning from a large amount of image and text information, xGen-MM not only demonstrates powerful performance in visual language tasks, but also promotes continuous improvement of model capabilities through open-source models, datasets, and fine-tuning code libraries.

Main functions of xGen-MM

• Multimodal understandingxGen-MM can process and understand image and text information simultaneously, and supports answering questions about visual content.
• Large-scale data learningThrough training with a large amount of diverse data, xGen-MM can capture rich visual and language patterns.
• High-performance generationxGen-MM can not only understand input information, but also generate text, such as writing descriptions or answers based on an image.
• Open source and accessiblexGen-MM's models, datasets, and code are open source, and researchers and developers are free to access and use these resources to build their own applications.
• Fine-tuning capabilityUsers can fine-tune xGen-MM according to their specific needs to adapt to different application scenarios.

xGen-MM project address

• GitHub repository:https://github.com/salesforce/LAVIS/tree/xgen-mm
• Hugging Face Model Library:https://huggingface.co/Salesforce/xgen-mm-phi3-mini-instruct-interleave-r-v1.5
• arXiv technical paper:https://arxiv.org/pdf/2408.08872

xGen-MM's technical principles

• Multimodal learningxGen-MM can be trained to understand both image and text data simultaneously, achieving the fusion of visual and linguistic information.
• Large-scale datasetsThe model is trained on a large-scale, diverse dataset containing a wealth of images and corresponding descriptions.
• Visual token samplerxGen-MM uses efficient visual token samplers (such as the Perceiver architecture) to process image data, enabling the model to handle images of different resolutions in a scalable manner.
• pre-trained language modelIt combines a pre-trained large-scale language model (such as the Phi-3 model), which has been trained on a large amount of text data and has a strong language understanding ability.
• Unified training objectivesIt simplifies the training process by training the model using a single autoregressive loss function, focusing on predicting text tokens in multimodal contexts.
• Command fine-tuningThe model can be fine-tuned through instructions to better understand and execute user queries, and additional training can be performed on the pre-trained model for specific tasks.
• Post-training optimizationThis includes Direct Preference Optimization (DPO) and Safety Fine-tuning, which improve the usefulness of the model, reduce the illusion effect, and enhance safety.
• Open source and customizabilityThe code, models, and datasets of xGen-MM are all open source, allowing community members to customize and further develop them according to their own needs.

Application scenarios of xGen-MM

• Image description generationAutomatically generates descriptive text for images, suitable for social media, photo album management, etc.
• Visual Q&AAnswer questions about image content, such as providing product information in the education or e-commerce sectors.
• Document UnderstandingIt parses and understands images and text in documents, suitable for automated document processing and information retrieval.
• Content creationIt assists users in the creative process, such as automatically generating storyboards and designing concept art.
• Information retrievalBy combining images and text, the relevance and accuracy of search results can be improved.