MMedAgent - A multimodal AI agent designed specifically for the medical field, managing a variety of medical tasks.

What is MMedAgent?

MMedAgent is a multimodal AI agent designed specifically for the medical field, managing various medical tasks by integrating various open-source medical models. The system includes a multimodal large language model (MLLM) for instruction tuning, acting as an action planner and outcome aggregator, and a suite of medical tools tailored to the agent, each for specific medical tasks. MMedAgent can handle multiple medical imaging modalities, including MRI, CT, and X-rays, and supports various data types encountered in clinical practice. By understanding user commands and medical images, it generates formatted instructions to invoke specific tools, aggregating tool outputs to provide accurate and comprehensive responses to the user. MMedAgent outperforms existing open-source methods and even surpasses the closed-source model GPT-4o on multiple medical tasks.

Main functions of MMedAgent

• Multimodal task processingMMedAgent can handle multiple language and multimodal tasks, including grounding, segmentation, classification, medical report generation (MRG), and retrieval augmentation generation (RAG).
• Medical imaging supportThe system supports multiple medical imaging modes, such as MRI, CT and X-ray, to adapt to various data types encountered in clinical practice.
• Tool integration and invocationMMedAgent integrates multiple tools covering seven representative medical tasks and can select and invoke the appropriate tools based on user instructions.
• Command fine-tuningMMedAgent trains a multimodal large language model (MLLM) as an action planner by creating instruction-adjusted datasets to understand and execute user instructions.
• Results aggregationMLLM acts as a result aggregator, combining the tool's output with user instructions and images to generate the final answer.
• End-to-end trainingMMedAgent uses an autoregressive objective to train the generated sequences end-to-end, ensuring that the model can use the correct tools and answer questions based on the tool results.

Technical principles of MMedAgent

• System ArchitectureMMedAgent consists of two main parts:
  • A multimodal large language model (MLLM) with instruction adjustment serves as an action planner and outcome aggregator.
  • A collection of medical tools tailored for agents, each tool designed for specific tasks within the medical field.
• WorkflowThe MMedAgent workflow consists of four steps:
  • Users provide instructions and medical images.
  • MLLM understands instructions and images, and generates formatted instructions to invoke specific tools.
  • Execute the tool and return the results.
  • MLLM combines the tool's output with user commands and images to generate the final answer.
• Command fine-tuningMMedAgent employs a standardized dialogue format to ensure its role as an action planner and results aggregator. Upon receiving user input, MMedAgent generates three parts:
  • Thought: Determine whether external tools are needed.
  • API Name and API Params: The name and parameters of the API call.
  • Value: The tool output and natural language response aggregated by MLLM.
• Autoregressive target trainingMMedAgent uses an autoregressive objective to train the generated sequences end-to-end, ensuring that the model can use the correct tools and answer questions based on the results of those tools.

MMedAgent project address

• Github repository:https://github.com/Wangyixinxin/MMedAgent
• arXiv technical paper:https://arxiv.org/pdf/2407.02483

Application scenarios of MMedAgent

• Visual Question Answering (VQA)MMedAgent can handle questions related to medical imaging, providing answers based on image content and supporting multiple imaging modalities such as MRI, CT, X-ray, histology, and gross pathology.
• Classification tasksBy using the BiomedCLIP tool, MMedAgent can perform zero-sample and fine-grained medical image classification.
• Location and segmentation tasksMMedAgent integrates Grounding DINO and MedSAM tools for localization and segmentation tasks in medical images, including segmentation based on bounding box cues and segmentation based on text cues (G-Seg).
• Medical Report Generation (MRG)Using the ChatCAD tool, MMedAgent can generate accurate medical reports from chest X-ray images.
• Search Enhancement Generation (RAG)MMedAgent, through the ChatCAD+ tool, can retrieve the most relevant information from external data sources, supporting the medical search process.
• Cross-modal medical task processingMMedAgent can seamlessly leverage various medical tools to handle a wide range of medical tasks across different imaging modalities.