SlideChat - A visual language assistant launched by Shanghai AI Lab in collaboration with multiple universities and institutions.
SlideChat, developed by the Shanghai AI Lab, Xiamen University, East China Normal University, and other institutions, is the first visual language assistant capable of understanding gigapixel-level full-slice images. SlideChat can generate detailed descriptions of full-slice images...
What is SlideChat?
SlideChat, developed by the Shanghai AI Lab, Xiamen University, East China Normal University, and other institutions, is the first visual language assistant capable of understanding gigapixel-level whole-slice images. SlideChat generates detailed descriptions of whole-slice images and provides context-sensitive, complex command responses for diverse pathological scenarios. Based on training, SlideChat has demonstrated superior performance in multiple clinical tasks, including microscopy and diagnosis. SlideChat utilizes the large-scale multimodal command dataset SlideInstruction and the evaluation benchmark SlideBench, which contains multiple subsets covering 21 different clinical tasks.
SlideChat's main functions
- Full-slice image understandingProcesses and understands gigapixel-level whole-slice pathological images (WSIs), providing in-depth analysis of the images.
- Multimodal dialogue capabilitiesIt supports multimodal dialogue with users, understands natural language commands, and provides responses by combining visual information.
- Complex command responseIt responds to and executes complex visual queries and pathology-related instructions.
- Clinical task coverageIt demonstrates superior performance in a variety of clinical settings, such as microscopy and diagnosis, covering 21 different clinical tasks.
SlideChat's technical principles
- Image segmentationThe whole slice image is divided into small patches of 224×224 pixels to facilitate calculation and processing.
- Local encoderEach image patch is converted into a visual embedding based on a local encoder, capturing local features.
- Slide-level encoder: Process the output of the local encoder with a slide-level encoder to generate a context embedding that includes global context information.
- Multimodal projectionMultimodal projectors map visual features to a unified space aligned with a large language model (LLM).
- Two-stage training:
- Cross-domain alignmentIn the first stage, the model learns to align the word embeddings of the LLM with the visual features extracted from the WSI.
- Visual instruction learningIn the second phase, the model learns how to accurately respond to domain-specific questions specific to WSI.
SlideChat's project address
- Project official website:uni-medical.github.io/SlideChat.github.io
- HuggingFace model library:https://huggingface.co/datasets/General-Medical-AI/SlideBench
- arXiv technical paper:https://arxiv.org/pdf/2410.11761
SlideChat Application Scenarios
- Pathological diagnostic aidIt helps pathologists analyze and interpret whole-slice images, aiding in the diagnosis of various diseases, including serious pathological conditions such as cancer.
- Education and trainingIn medical education, it serves as a teaching tool to help students and residents learn how to interpret pathology slides and improve their diagnostic skills.
- Research and developmentResearchers are exploring new biomarkers to classify disease subtypes and predict disease progression and patient prognosis.
- Clinical decision supportIt integrates into clinical workflows, providing real-time pathological analysis to help doctors make more accurate treatment decisions.
- Quality control and standardizationIn pathology laboratories, ensuring consistency and accuracy in diagnoses relies on automated analysis to reduce human error.