GLM-4-Voice - An end-to-end emotion-based speech model launched by Zhipu AI.
GLM-4-Voice is an end-to-end emotional speech model developed by Zhipu AI. It can directly understand and generate Chinese and English speech, support real-time voice dialogue, and flexibly adjust the emotion, tone, speed, and dialect features of the speech according to user commands.
What is GLM-4-Voice?
GLM-4-Voice is an end-to-end emotional speech model launched by Zhipu AI. It can directly understand and generate Chinese and English speech, support real-time voice dialogue, and flexibly adjust the emotion, tone, speed, and dialect features of the speech according to user commands. The model consists of three parts: GLM-4-Voice-Tokenizer is responsible for converting continuous speech into discrete tokens; GLM-4-Voice-Decoder converts the tokens back into continuous speech output; and GLM-4-Voice-9B is pre-trained and aligned based on the GLM-4-9B model to understand and generate speech. The design of GLM-4-Voice uses end-to-end modeling to reduce information loss, improve the naturalness and fluency of voice interaction, and support low-latency real-time dialogue, providing users with a richer and more natural voice interaction experience.
Main functions of GLM-4-Voice
- Understanding and generating speechIt can directly understand and generate Chinese and English speech, enabling smooth human-computer interaction.
- Emotional expressionIt simulates different emotions and tones, such as happiness, sadness, anger, and fear, to make voice responses more natural.
- Adjust speech rateIt adjusts the speech rate according to user instructions, making it suitable for different conversation scenarios.
- Real-time interruption and command inputIt allows users to interrupt voice output at any time and input new commands to adjust the dialogue content.
- Multilingual and dialect supportSupports Chinese, English, and various Chinese dialects, such as Cantonese, Chongqing dialect, and Beijing dialect.
- Low-latency interactionDesign a streaming thinking architecture to achieve high-quality voice dialogue with low latency.
The technical principle of GLM-4-Voice
- End-to-end modelingUnlike traditional cascaded solutions (ASR + LLM + TTS), GLM-4-Voice completes speech understanding and generation in a unified model, avoiding information loss.
- Audio TokenizerA supervised training audio tokenizer is used to convert continuous speech input into discrete tokens, preserving semantic information and paralinguistic features at a low bit rate of 12.5Hz.
- Voice decoderThe speech decoder based on the Flow Matching model structure transforms discrete speech tokens into continuous speech output, requiring only a minimum of 10 tokens to start generating, thus reducing dialogue latency.
- Pre-training and AlignmentGLM-4-Voice-9B is pre-trained and aligned based on GLM-4-9B to understand and generate discretized speech tokens. Pre-training uses a large amount of audio and text data, giving the model powerful audio understanding and modeling capabilities.
- Streaming reasoningIt supports streaming inference, and the model can output text and speech alternately. The text is used as a reference to ensure the high quality of the response content, and the speech output is adjusted in real time according to the user's voice commands.
GLM-4-Voice project address
- Product Experience: https://ai-bot.cn/sites/2005.html
- Project official website:zhipuai.cn/news
- GitHub repository:https://github.com/THUDM/GLM-4-Voice
- arXiv technical paper:https://arxiv.org/pdf/2412.02612
Application scenarios of GLM-4-Voice
- Smart AssistantIn smartphones and smart home devices, it acts as a smart assistant, using voice interaction to help users complete various tasks, such as setting reminders, checking the weather, and controlling home devices.
- Customer ServiceIn the customer service center, as a virtual customer service representative, based on natural language understanding and speech synthesis technology, we provide users with consultation and problem-solving services.
- Education and LearningIn the field of education, it serves as a language learning assistant, helping students practice pronunciation, listening, and speaking, and providing personalized learning suggestions.
- Entertainment and MediaIn the entertainment industry, it is used in speech synthesis to provide natural and expressive voice output for animation, games, audiobooks, and more.
- News and broadcastsUsed in news broadcasting, it quickly converts text news into speech and provides it to users who need voice information.