VideoChat - an open-source, real-time digital human dialogue system with a first-packet latency as low as 3 seconds.
VideoChat is an open-source, real-time digital human dialogue system that supports voice input and real-time conversation. Users can customize the digital human's appearance and voice, and voice cloning is possible without training. First-packet latency can be as low as 3 seconds, making it suitable for...
What is VideoChat?
VideoChat is an open-source, real-time digital human dialogue system that supports voice input and real-time conversation. Users can customize the digital human's appearance and voice, achieving voice cloning without training. First-packet latency can be as low as 3 seconds, making it suitable for various real-time voice interaction scenarios such as live streaming, news broadcasting, and chat assistants. The system supports GLM-4 Voice and provides two generation methods: ASR-LLM-TTS-THG and MLLM-THG. VideoChat uses the Grado framework to build interactive applications, supports streaming video output, and facilitates rapid deployment and development.
VideoChat's main functions
- Real-time voice interactionIt supports users to have real-time conversations with digital humans using voice.
- Customize appearance and toneUsers can choose or design the appearance and voice of their digital human to achieve personalized interaction.
- Voice input and text outputThe system converts the user's voice input into text and then generates a response text based on a large language model.
- Lip synchronizationWhen a digital human speaks, the movement of its lips is synchronized with the sound it makes, enhancing the realism.
- Streaming video outputBased on the Grado framework, it supports streaming video output, making the interaction smoother.
VideoChat's technical principles
- Automatic Speech Recognition (ASR)Use tools like FunASR to convert user voice input into text.
- Large Language Model (LLM)Based on models such as Tongyi Qianwen, corresponding response text is generated according to the input text.
- Text-to-speech (TTS): Convert text replies to speech using tools such as GPT-SoVITS.
- Speaker Generation (THG)Use solutions such as MuseTalk to generate lip-synced digital human videos based on speech.
- Streaming output parallel pipelineBased on parallel processing technology, it performs inference and playback simultaneously, improving response speed.
- Gradio frameworkUse Grado 5 to achieve streaming video output, making it easy to deploy and build interactive applications.
VideoChat's project address
- GitHub repository:https://github.com/Henry-23/VideoChat
- Experience the demo online:https://www.modelscope.cn/studios/AI-ModelScope/video_chat
Application scenarios of VideoChat
- Customer ServiceAs a virtual customer service provider, it offers 24/7 consultation services, answers user questions, and reduces corporate labor costs.
- Online EducationAs a virtual teacher, it provides services such as language learning and course explanation, increasing the interactivity and fun of learning.
- News BroadcastIn the news industry, it can be used to broadcast news in the form of digital humans, improving the efficiency and appeal of news broadcasts.
- Live streaming industryAs a virtual anchor, you can introduce products, sell goods via live streaming, and increase the interactivity of the live stream and the viewing experience for the audience.
- Entertainment and InteractionIn the entertainment field, such as games and virtual concerts, it provides richer interactive experiences.