Migician - A multimodal visual positioning model jointly developed by Beijing Jiaotong University, Tsinghua University, and Huazhong University of Science and Technology
Migician is a multimodal large language model (MLLM) jointly developed by research teams from Beijing Jiaotong University, Huazhong University of Science and Technology, and Tsinghua University. It is specifically designed for free-form multi-image grounding (MIL)...
What is Migician?
Migician is a multimodal large language model (MLLM) jointly developed by research teams from Beijing Jiaotong University, Huazhong University of Science and Technology, and Tsinghua University. It is specifically designed for free-form multi-image grounding (MIG) tasks and utilizes the large-scale training dataset MGrounding-630k. Based on free-form queries (such as text descriptions, images, or a combination of both), it identifies and precisely locates relevant visual regions in multiple images. Migician is trained on the large-scale instruction-tuning dataset MGrounding-630k using a two-stage training method, combining multi-image understanding and single-image localization capabilities to achieve end-to-end multi-image localization. The design and training methods of Migician provide new insights for the application of multimodal models in complex visual scenes, promoting the integration of multi-image understanding and fine-grained visual localization.
Migician's main functions
- Cross-image localizationFind objects or regions related to the query in multiple images and provide their precise locations (e.g., coordinate boxes).
- Flexible input methodsIt supports text, images, or a combination of both as queries, such as "find objects in Figure 2 that are similar to those in Figure 1, but with different colors".
- Multitasking supportIt can handle a variety of tasks related to multiple images, such as object tracking, difference recognition, and common object localization.
- Efficient ReasoningBased on end-to-end model design, inference is performed directly in multi-image scenarios, avoiding the multi-step inference and error propagation problems of traditional methods.
Migician's technical principles
- End-to-end multi-image localization frameworkThis approach directly handles multi-image localization tasks based on an end-to-end model architecture, avoiding the complexity and efficiency issues of traditional methods that decompose the task into multiple sub-tasks (such as generating text descriptions before localization). It simultaneously understands the content of multiple images and directly outputs the location of the target object based on the query.
- Large-scale instruction tuning dataset (MGrounding-630k)This dataset contains over 630,000 data points related to multi-image localization tasks. It covers various task types (such as static difference localization, common object localization, and object tracking), and combined with free-form instructions, the model learns diverse localization capabilities.
- Two-stage training method:
- Phase 1The model is trained on various multi-image tasks to learn basic multi-image understanding and localization capabilities.
- Phase TwoBased on free-form instruction tuning, improve the model's ability to locate complex queries and maintain adaptability to diverse tasks.
- Multimodal fusion and reasoningIt combines visual and linguistic modal information to understand and locate complex queries based on multimodal fusion, and processes abstract visual semantic information, such as locating target objects through comparison, similarity, or functional association.
- Model merging techniquesBased on model merging technology, the weights of different training stages are averaged to optimize overall performance.
Migician's project address
- Project official website:https://migician-vg.github.io/
- GitHub repository:https://github.com/thunlp/Migician
- HuggingFace model library:https://huggingface.co/Michael4933/Migician
- arXiv technical paper:https://arxiv.org/pdf/2501.05767
Migician's application scenarios
- autonomous drivingIt can quickly locate targets around the vehicle (such as pedestrians and obstacles) and supports multi-view perception and dynamic target tracking.
- Security monitoringMulti-camera linkage identifies abnormal behavior or targets, and analyzes abnormal situations such as crowd gathering and rapid movement.
- Robot InteractionIt accurately locates target objects, enabling robots to perform tasks such as grasping and navigation in complex environments.
- Image editingAnalyze the content of multiple images to achieve object replacement, deletion, or creative content generation.
- Medical imagingIt integrates multimodal images to quickly locate lesion areas or abnormal tissues and supports dynamic monitoring.