MindLLM - A medical AI model developed by Yale, Cambridge, and other institutions.

What is MindLLM?

MindLLM, an AI model jointly developed by Yale University, Dartmouth College, and the University of Cambridge, decodes functional magnetic resonance imaging (fMRI) signals from the brain into natural language text. MindLLM achieves high-performance decoding based on a subject-agnostic fMRI encoder and a large language model (LLM), incorporating Brain Instruction Tuning (BIT) technology to capture diverse semantic information from fMRI signals. MindLLM demonstrates superior performance across multiple benchmarks, showing a 12.0% improvement in downstream task performance, a 16.4% improvement in cross-individual generalization ability, and a 25.0% improvement in adaptability to new tasks. MindLLM offers new possibilities for brain-computer interface and neuroscience research.

MindLLM's main functions

• Decoding Brain ActivityTransforming the neural activity of the brain during perception, thinking, or recall into intuitive textual descriptions helps scientists and doctors better understand how the brain works.
• Cross-individual universalityIt can process brain signals from different individuals without requiring individual training for each individual, greatly improving the model's generalization ability.
• Multi-functional decoderMindLLM is adaptable to a variety of tasks, such as visual scene understanding, memory retrieval, language processing, and complex reasoning, demonstrating powerful versatility.
• Assisted healthcare and human-computer interactionTo help aphasic patients regain their communication abilities, or to promote the development of brain-computer interface technology by controlling prosthetics, virtual assistants, and other devices based on neural signals.

The technical principles of MindLLM

• fMRI encoderUsing neuroscience-inspired attention mechanisms, fMRI signals are encoded into a series of "brain feature tokens." The encoder learns functional and spatial information from different brain regions, dynamically extracting features and avoiding information loss due to individual differences.
• Large Language Models (LLM)This approach combines encoded brain feature tokens with a language model, leveraging the powerful generative capabilities of Language Modeling (LLM) to transform brain signals into natural language text. A pre-trained LLM (such as Vicuna-7b) is used as the decoder to ensure the generated text possesses semantic coherence and accuracy.
• Brain command optimizationThe model is trained on diverse datasets (such as visual question answering, image captioning, and memory retrieval tasks) to capture diverse semantic information in fMRI signals. The BIT dataset uses images as an intermediary to pair fMRI data with corresponding text annotations, training the model to perform multiple tasks and improving its versatility and adaptability.
• Subject-independent designBased on functional information of separated brain regions (consistent across individuals) and fMRI signal values, MindLLM shares prior knowledge among different individuals, achieving universal decoding capabilities across individuals.

MindLLM's project address

• arXiv technical paper:https://arxiv.org/pdf/2502.15786

Application Scenarios of MindLLM

• Medical RehabilitationIt helps patients with aphasia, paralysis, and other conditions to regain their communication abilities by decoding brain signals to help users express their thoughts or control external devices.
• Brain-computer interfaceTo develop more efficient and intuitive brain-computer interface systems, such as those for controlling prostheses, wheelchairs, or virtual reality devices, to improve the quality of life for people with disabilities.
• Neuroscience researchTo help scientists better understand the brain's cognitive mechanisms, conscious activity, and the relationship between neural signals and behavior, thus advancing the development of neuroscience.
• Human-computer interactionTo achieve a more natural and direct way of human-computer interaction, using brain signals to control electronic devices, smart home systems or autonomous driving systems, and improve the interactive experience.
• Mental health supportMonitoring and analyzing brain activity can aid in the diagnosis of mental illnesses or the evaluation of treatment effectiveness, providing new tools and methods for the field of mental health.