VisionFM - A general-purpose ophthalmology AI model with the ability to diagnose multiple diseases with few samples.

What is VisionFM?

VisionFM is a multimodal, multi-task vision foundational model designed specifically for general ophthalmic artificial intelligence. It is pre-trained on 3.4 million ophthalmic images from 560,457 individuals, covering a wide range of ophthalmic diseases, imaging modalities, devices, and population statistics. VisionFM can handle eight common ophthalmic imaging modalities, including fundus photography, optical coherence tomography (OCT), and fluorescein fundus angiography (FFA), and is applied to various ophthalmic AI tasks such as ophthalmic disease identification, disease progression prediction, disease phenotypic segmentation, and systemic biomarker and disease prediction. It surpasses ophthalmologists with basic and intermediate levels in diagnosing 12 common ophthalmic diseases and outperforms powerful baseline deep neural networks on large-scale ophthalmic disease diagnostic benchmark databases. VisionFM also demonstrates strong generalization ability to novel ophthalmic modalities, disease lineages, and imaging devices.

VisionFM's main functions

• Disease screening and diagnosisVisionFM can screen and diagnose a variety of eye diseases, including but not limited to diabetic retinopathy, glaucoma, and age-related macular degeneration.
• Disease prognosisThe model can also predict the development trend and prognosis of diseases.
• Disease phenotype segmentationVisionFM can perform subclassification of disease phenotypes, including segmentation of lesions, blood vessels and layers, as well as landmark detection.
• Whole-body biomarkers and disease predictionIn addition to eye diseases, VisionFM can also predict systemic biomarkers and diseases from eye images.
• Multimodal processing capabilityVisionFM can handle eight common ophthalmic imaging modalities, including fundus photography, optical coherence tomography (OCT), and fluorescein fundus angiography (FFA).
• Modality-independent diagnosisVisionFM supports modality-independent diagnostics, meaning that a single decoder can be used to diagnose multiple ophthalmic diseases in different imaging modalities.
• Few-shot learningVisionFM demonstrates the ability to learn with few samples, enabling it to diagnose new diseases with high accuracy, even with only a small number of labeled samples.
• Strong generalization abilityThe model demonstrates strong generalization ability for new ophthalmic modalities, disease spectrums, and imaging devices.
• Synthetic data reinforcement learningVisionFM can also leverage synthetic ophthalmic imaging data to enhance its representation learning capabilities, thereby achieving significant performance improvements in downstream ophthalmic AI tasks.

VisionFM's technical principles

• Large-scale pre-trainingVisionFM is a deep learning-based vision foundation model that is pre-trained on 3.4 million ophthalmic images from 560,457 individuals, covering a wide range of ophthalmic diseases, imaging modalities, imaging devices, and demographic data.
• Multimodal multitasking learningVisionFM can handle multiple ophthalmic imaging modalities, including fundus photography, OCT, FFA, etc., and can be applied to various ophthalmic AI tasks such as disease screening, diagnosis, disease prognosis, and disease phenotype segmentation.
• Expert-level intelligence and accuracyThe pre-trained VisionFM demonstrated expert-level intelligence and accuracy in multiple ophthalmology AI applications. Its comprehensive intelligence surpassed that of junior and intermediate ophthalmologists in the joint diagnosis of 12 common eye diseases.

VisionFM's project address

• Github repository:https://github.com/ABILab-CUHK/VisionFM
• arXiv technical paper:https://arxiv.org/pdf/2310.04992

Application scenarios of VisionFM

• Ophthalmology clinical tasksVisionFM can help solve ophthalmological clinical tasks, especially in disease screening and diagnosis.
• Diagnosis of various eye diseasesThe model has demonstrated excellent performance in diagnosing and predicting a variety of eye diseases, including diabetic retinopathy, glaucoma, and age-related macular degeneration.
• Primary healthcare environmentThis model can play an important role in primary healthcare environments with limited imaging resources, reducing the workload of doctors.
• Areas with low ophthalmologist densityVisionFM is especially useful in regions and countries with low ophthalmologist density.
• Education and trainingVisionFM can serve as a training platform for experienced ophthalmologists to train junior ophthalmology practitioners, possessing extensive knowledge in ophthalmic imaging and disease diagnosis.
• Assist in generating diagnostic reportsThis model can be integrated with large language models (LLM) to generate diagnostic reports, completing the closed loop of ophthalmic disease diagnosis.