EmoLLM - A large language model focused on mental health support
EmoLLM is a large-scale language model focused on mental health support, providing users with emotional guidance and psychological support through multimodal emotion understanding. It combines various data formats such as text, images, and video, based on advanced multi-view perspectives...
What is EmoLLM?
EmoLLM is a large-scale language model focused on mental health support, providing users with emotional guidance and psychological support through multimodal emotion understanding. It combines various data formats such as text, images, and videos, and utilizes advanced multi-view visual projection technology to capture emotional cues from different angles, gaining a more comprehensive understanding of the user's emotional state. EmoLLM is fine-tuned based on multiple open-source large language models, supporting emotional tasks such as emotion recognition, intent understanding, humor detection, and hate detection.
EmoLLM's main functions
- Understanding users: Identify users' emotional state and psychological needs through dialogue interaction.
- Emotional supportIt provides emotional support to help users relieve stress and anxiety.
- Psychological counselingCombining cognitive behavioral therapy and other methods, we guide users to improve their emotion management and coping strategies.
- role playBased on the needs of different users, it provides a variety of dialogue experiences with different roles (such as a psychologist, a gentle older sister, a father figure boyfriend, etc.).
- Personalized tutoringBased on user feedback and progress, we provide customized psychological counseling solutions.
- Mental health assessmentUse scientific tools to assess users' mental state and diagnose potential psychological problems.
- Education and preventionProvides mental health knowledge to help users understand how to prevent mental health problems.
- Multi-turn dialogue support: Provide ongoing psychological counseling and support through multi-turn dialogue datasets.
- Social support systemConsidering the impact of family, work, community, and cultural background on mental health, provide guidance on social support systems.
EmoLLM's technical principles
- Multi-perspective visual projectionEmoLLM uses multi-view visual projection technology to capture emotional cues in visual data from multiple angles. It analyzes emotional information from a single viewpoint and constructs graph-based representations to capture relationships between object features. By jointly mining content and relational information, the model can extract features more suitable for emotional tasks.
- Emotional guidance prompt (EmoPrompt)EmoPrompt is a technique for guiding multimodal large language models (MLLMs) to correctly infer sentiment. By introducing task-specific examples and combining them with the capabilities of GPT-4V to generate accurate chain-of-thought (CoT), it ensures the model's accuracy in sentiment understanding.
- Multimodal codingEmoLLM integrates multiple modal encoders to handle various inputs such as text, images, and audio. For example, it uses the CLIP-VIT-L/14 model to process visual information, the WHISPER-BASE model to process audio signals, and an LLaMA2-7B-based text encoder to process text data.
- Command fine-tuningEmoLLM, based on advanced instruction fine-tuning techniques such as QLORA and full fine-tuning, refines the original language model to better adapt to the complex emotional context in the field of mental health.
EmoLLM's project address
- GitHub repository:https://github.com/yan9qu/EmoLLM
- arXiv technical paper:https://arxiv.org/pdf/2406.16442
Application Scenarios of EmoLLM
- Mental health counselingProvide users with emotional support and advice.
- Sentiment AnalysisUsed for social media emotion monitoring, mental health monitoring, etc.
- Multimodal emotion taskSuch as emotion recognition in images and videos.