HealthGPT - A medical visual language model developed by Zhejiang University in collaboration with Alibaba and other institutions.
HealthGPT is an advanced medical visual language model (Med-LVLM) jointly developed by Zhejiang University, University of Electronic Science and Technology of China, Alibaba, and other institutions. It utilizes heterogeneous knowledge adaptation technology to achieve medical visual understanding and generation tasks...
What is HealthGPT?
HealthGPT is an advanced medical visual language model (Med-LVLM) jointly developed by Zhejiang University, University of Electronic Science and Technology of China, Alibaba, and other institutions. It achieves a unified framework for medical visual understanding and generation tasks through heterogeneous knowledge adaptation technology. Employing innovative Heterogeneous Low-Rank Adaptation (H-LoRA) technology, it stores knowledge for visual understanding and generation tasks in independent "plugins," avoiding conflicts between tasks. HealthGPT offers two versions: HealthGPT-M3 (3.8 billion parameters) and HealthGPT-L14 (14 billion parameters), based on the Phi-3-mini and Phi-4 pre-trained language models, respectively. The model introduces Hierarchical Visual Perception (HVP) and a three-stage learning strategy (TLS) to optimize visual feature learning and task adaptation capabilities.
Main functions of HealthGPT
- Medical image analysis and diagnostic assistanceHealthGPT can process various medical images (such as X-rays, CT scans, MRI scans, etc.), helping doctors interpret image results and provide diagnostic suggestions.
- Visual Q&AThe model can answer related questions based on medical images, such as explaining abnormalities or the location of lesions in the images.
- Medical text understanding and generationHealthGPT can process and generate medical texts, such as medical record summaries and diagnostic reports, helping doctors quickly organize and record patient information.
- Multimodal fusionBy combining visual and textual information, HealthGPT can gain a more comprehensive understanding of complex medical scenarios and provide more accurate diagnostic and treatment recommendations.
- Personalized treatment plan recommendationsBased on the patient's medical history and medical images, HealthGPT can generate personalized treatment plans to assist doctors in making clinical decisions.
HealthGPT's technical principles
- Heterogeneous low-rank adaptation (H-LoRA)HealthGPT introduces a heterogeneous low-rank adaptation technique, separating the learning processes for visual understanding and generation tasks to avoid conflicts between tasks. By introducing a low-rank matrix to update the weights, H-LoRA significantly reduces the number of parameters required for training while maintaining the model's expressive power.
- Hierarchical visual perception (HVP)The technology separates visual detail learning from the visual transformer (ViT), addressing the different visual granularity requirements of visual understanding and generation tasks respectively. This enables the model to process complex medical image data more efficiently.
- Three-stage learning strategy (TLS)HealthGPT employs a three-stage learning strategy to progressively train the H-LoRA plugin, enabling the model to quickly adapt to various downstream medical tasks. It performs exceptionally well even with limited data, achieving or exceeding the performance of state-of-the-art models across multiple metrics.
HealthGPT project address
- Project official website:https://llsuzy.github.io/HealthGPT.github.io/
- Github repository:https://github.com/DCDmllm/HealthGPT
- HuggingFace model library:https://huggingface.co/lintw/HealthGPT-M3
- arXiv technical paper:https://arxiv.org/pdf/2502.09838
Application scenarios of HealthGPT
- Medical image generationHealthGPT can generate high-quality medical images, such as for super-resolution tasks or image reconstruction, to aid in medical diagnosis and research.
- Medical Education and ResearchHealthGPT can be used in medical education to help students better understand medical images and diagnostic procedures. It also provides powerful tools for medical research, supporting the analysis and processing of multimodal data.
- Smart Health AssistantHealthGPT can serve as a smart health assistant, helping users query health data and providing daily health management suggestions.