Open-LLM-VTuber - An AI digital human voice interaction project that supports real-time voice dialogue and visual perception.

What is Open-LLM-VTuber?

Open-LLM-VTuber is an open-source, cross-platform voice-interactive AI companion project. It supports real-time voice dialogue and visual perception, features a vivid Live2D animated avatar, and can operate completely offline to protect privacy. Users can use it as a virtual girlfriend, boyfriend, or pet to enjoy a personalized interactive experience. The project integrates multiple Large Language Model (LLM), Automatic Speech Recognition (ASR), and Text-to-Speech (TTS) solutions, allowing users to customize the avatar's appearance, voice, and interactive functions according to their needs.

Main functions of Open-LLM-VTuber

• Voice interactionIt supports real-time voice dialogue, allowing users to communicate with AI using voice without manual input.
• Visual perceptionIt supports camera input, screen recording, and screenshots, and the AI can "see" the user and the screen content.
• Live2D Animated CharactersIt features vivid Live2D animated characters that support facial expressions and motion changes.
• Run offlineAll features support running completely offline locally, protecting user privacy.
• Cross-platform supportCompatible with Windows, macOS, and Linux, supporting both GPU acceleration and CPU operation.
• Personalized customizationUsers can customize their character's appearance, voice, and interactive features, including cloning specific voices.
• Rich interactive featuresIt supports features such as voice interruption, touch feedback, chat history saving, and multilingual TTS.
• Desktop Pet ModeSupports transparent backgrounds, global top placement, and mouse pass-through; the AI can move anywhere on the desktop.

The technical principles of Open-LLM-VTuber

• Large Language ModelAs the core interaction engine, the LLM is responsible for understanding user input (voice or text) and generating responses. The project supports various LLMs, such as Ollama, OpenAI, and Gemini, allowing users to choose different models based on their needs.
• Speech recognitionIt converts user voice input into text for LLM processing. It supports various ASR solutions, such as Whisper and FunASR, ensuring the accuracy and efficiency of speech recognition.
• Speech SynthesisIt converts text generated by LLM into speech output, supports multiple TTS engines such as MeloTTS and Bark, and supports multilingual synthesis.
• Live2D Animated CharactersLive2D technology is used to generate dynamic character images. Based on facial expression mapping and motion control, the characters can dynamically display facial expressions and actions according to the dialogue content or emotional changes.
• Visual perceptionBased on camera or screen recording capabilities, AI acquires visual information to enable richer interactive experiences, such as recognizing user expressions or screen content.
• Modular designThe project uses a modular architecture, allowing users to switch between different functional modules by simply modifying configuration files without having to delve into the code.

Open-LLM-VTuber project address

• GitHub repository:https://github.com/t41372/Open-LLM-VTuber

Application scenarios of Open-LLM-VTuber

• Virtual companionUsers can set up virtual girlfriends, boyfriends, or pets to enjoy emotional companionship and personalized interactions, thus fulfilling their emotional needs.
• Office AssistantIn desktop pet mode, it provides real-time information query, voice reminders, document reading and other auxiliary functions to improve office efficiency.
• Learning tutoringIt helps users learn languages and answer questions, and assists learning based on screen sharing.
• Entertainment and InteractionUsers can engage in entertainment activities such as voice games and role-playing with AI, increasing the fun.
• Technology Demonstration and DevelopmentDevelopers will develop and demonstrate AI interaction technologies to explore more application scenarios.