Lingshu - A multimodal language model for healthcare launched by Alibaba.

What is Lingshu?

Lingshu is a large-scale multimodal language model developed by Alibaba DAMO Academy, focusing on the medical field. The model supports over 12 medical imaging modalities, including X-ray, CT scan, and MRI, demonstrating outstanding performance in tasks such as multimodal question answering, text-based question answering, and medical report generation. Based on multi-stage training, Lingshu gradually embeds medical expertise, significantly improving its reasoning and problem-solving capabilities in the medical field. Two parameter versions, 7B and 32B, are available, with the 32B version outperforming proprietary models like GPT-4.1 in multiple medical multimodal question answering tasks. The Lingshu project also introduces the MedEvalKit evaluation framework, integrating mainstream medical benchmarks to promote the standardized evaluation and development of medical AI models.

Lingshu's main functions

• Multimodal medical Q&AIt supports visual question answering tasks for various medical imaging modalities (such as X-ray, CT, MRI, etc.). It generates accurate answers based on the input medical images and related questions.
• Medical report generationLingshu can generate high-quality medical reports, such as radiology reports, and generate detailed "Discovery" and "Impression" sections based on the input medical images, providing valuable references for clinicians.
• Medical Knowledge Q&ALingshu can answer plain text medical questions, covering a wide range of medical knowledge, providing accurate information for medical students, clinicians, and researchers.
• Medical reasoning and diagnostic supportLingshu possesses powerful medical reasoning capabilities, enabling it to perform complex reasoning and diagnostic support based on input medical images and text information.
• Medical Image Understanding and AnnotationLingshu can understand and annotate key features in medical images, such as the location, type, and severity of lesions, and supports the generation of detailed image descriptions to help doctors better understand the image content.

Lingshu's technical principles

• Data processing:
  • Data collectionCollect medical imaging data, medical text data, and general domain data from multiple sources.
  • Data cleaningBased on image filtering, deduplication, and text cleaning, we ensure the quality and relevance of the data.
  • Data SynthesisGenerate high-quality medical titles, visual question answering (VQA) samples, and inference trajectories to enrich training data.
• Model ArchitectureLingshu is based on the Qwen2.5-VL model architecture and includes three key components: a large language model (LLM) to process text input and generate text output, a visual encoder to extract visual features from medical images, and a projector to map the visual features to the representation space of the language model.
• Multi-stage training:
  • Medical superficial alignmentFine-tuning with a small number of medical image text pairs allows the model to accurately encode medical images and generate corresponding descriptions.
  • Medical Deep AlignmentWe introduce a larger-scale, higher-quality, and semantically richer medical image-text pair dataset to further integrate medical knowledge.
  • Medical directive adjustment: Improve the ability to execute specific task instructions by optimizing models based on large-scale medical instruction data.
  • Medical-oriented reinforcement learningEnhance the medical reasoning ability of the model using the reinforcement learning and verifiable reward (RLVR) paradigm.
• Evaluation FrameworkIntroducing the MedEvalKit evaluation framework, which integrates multiple multimodal and text-based medical benchmarks and supports various question formats, including multiple-choice questions, closed-ended questions, open-ended questions, and medical report generation. The framework provides standardized data preprocessing formats and post-processing protocols, as well as consistent model deployment and inference interfaces, supporting rapid integration and one-click evaluation.

Lingshu's project address

• Project official websitehttps://alibaba-damo-academy.github.io/lingshu/
• HuggingFace model library: https://huggingface.co/collections/lingshu-medical-mllm/lingshu-mllms-6847974ca5b5df750f017dad
• arXiv technical paper: https://arxiv.org/pdf/2506.07044

Application scenarios of Lingshu

• Medical image diagnosisIt can handle visual question answering tasks in multiple medical imaging modalities, analyze anomalies in images and provide diagnostic suggestions, generate detailed image descriptions and label key features to assist doctors in making accurate diagnoses.
• Medical report generationIt generates high-quality radiology and pathology reports based on medical images, covering "discovery" and "impression" sections, providing important references for clinicians and improving the efficiency and quality of report writing.
• Medical Knowledge Q&AIt provides accurate medical knowledge to help medical students, professionals, and clinicians obtain the information they need to make more informed decisions.
• Medical researchIn medical research, assist in the organization and analysis of medical images and text data.
• public healthLingshu can process public health data, support epidemiological studies, and calculate attribution risks, among other things.