GLM-TTS - Zhipu Open Source AI Text-to-Speech Model, supporting voice cloning

What is GLM-TTS?

GLM-TTS is an open-source AI text-to-speech model from Zhipu, featuring voice cloning, multi-emotion expression, high-precision text understanding, and high-quality speech output. It employs a two-stage generation paradigm, achieving efficient speech synthesis through semantic and acoustic modeling. GLM-TTS supports dialect cloning, fine-grained pronunciation control, and various emotional expressions, making it suitable for scenarios such as intelligent voice assistants, audiobooks, education, entertainment, customer service, accessibility assistance, advertising, news, smart homes, and virtual anchors. With its low word error rate and high naturalness, GLM-TTS provides users with a flexible, efficient, and personalized voice interaction experience, promoting the widespread application of speech synthesis technology across multiple fields. GLM-TTS is open-source on platforms such as GitHub, Hugging Face, and ModelScope.

Main functions of GLM-TTS

• Sound reproductionGLM-TTS can quickly replicate the timbre of a specific speaker, achieving highly similar speech synthesis with only a small amount of audio data, and supports timbre cloning of multiple languages and dialects.
• Multiple Emotion ClonesIt automatically matches the corresponding voice emotion based on the mood of the text content, supports a variety of emotional expressions such as happiness, sadness, and anger, and improves the naturalness and expressiveness of speech synthesis.
• High-precision text understandingIt possesses superior text understanding capabilities, accurately processes text content, reduces word error rates, and ensures the accuracy and fluency of synthesized speech.
• Dialect and Special Speech SynthesisIt supports the synthesis of various dialects and special voices, such as Sichuan dialect and Northeastern dialect, to meet the needs of different language and cultural backgrounds.
• Refined pronunciation control: By using phoneme-in technology, it solves the pronunciation problems of polyphonic characters and rare characters, and improves the accuracy and controllability of pronunciation.
• High-fidelity voice outputBased on a self-developed 2D-Vocos vocoder, it generates high-quality, high-fidelity speech waveforms, supports high sampling rate output, and improves sound quality performance.

GLM-TTS Technical Principles

• Two-stage generative paradigmGLM-TTS employs a two-stage generation paradigm, consisting of semantic modeling (Text-to-Token) and acoustic modeling and waveform reconstruction (Token-to-Wav). The first stage uses an autoregressive model to convert text into a sequence of semantic tokens, ensuring the accuracy and coherence of the content. The second stage uses a Conditional Flow-matching model to predict the Mel spectrogram, which is then converted into a high-quality speech waveform using a 2D-Vocos vocoder.
• Multi-reward reinforcement learningGLM-TTS introduces a multi-reward reinforcement learning mechanism based on the GRPO algorithm framework, which integrates multi-dimensional rewards such as character error rate (CER), similarity (Sim), emotion (Emotion), and paralinguistics (such as laughter). Through dynamic sampling and gradient pruning strategies, the training process is optimized, which significantly improves the emotional expression and anthropomorphism of speech.
• Fine-grained pronunciation control (Phoneme-in)GLM-TTS utilizes Phoneme-in technology, providing target phonemes for polyphonic and rare characters through a dynamically controllable dictionary and hybrid input methods, achieving precise pronunciation control. During the inference phase, the system combines the phoneme sequence with the text input model, preserving the text's rhythm while ensuring accurate pronunciation.
• LoRA (Premium Sound Customization)GLM-TTS adopts an optimized LoRA fine-tuning paradigm, requiring only about 15% of the model parameters to be fine-tuned. Combined with a small amount of high-quality audio data, it achieves timbre fidelity and naturalness comparable to full parameter fine-tuning, significantly reducing the development cost and implementation threshold of timbre customization, and improving the generalization ability and cross-scene stability of timbre.
• Data processing and feature extractionGLM-TTS constructs a comprehensive data processing pipeline, including steps such as speech standardization, background noise separation and denoising, speaker separation and splicing, WER filtering, punctuation optimization, and feature extraction. It extracts clean speech and high-quality features from heterogeneous audio, providing reliable data support for model training.
• Model structure optimizationGLM-TTS optimizes the Speech Tokenizer, increasing the token rate and vocabulary size, introduces a pitch estimation module (PE), removes the causal convolution constraint, and optimizes pitch modeling accuracy. At the same time, the 2D-Vocos vocoder improves the resolution accuracy and sound quality of spectral features through 2D convolution and DiT-like residual connections, enhancing the model's adaptability to complex vocal cues.

GLM-TTS project address

• GitHub repositoryhttps://github.com/zai-org/GLM-TTS
• HuggingFace model libraryhttps://huggingface.co/zai-org/GLM-TTS

How to use GLM-TTS

• Online experienceVisit https://audio.z.ai/, upload your text or voice prompt, and you can quickly generate audio.
• API callsObtain the API key through the Zhipu AI Open Platform, send a request according to the documentation, and send the text or tone requirements to the GLM-TTS server to obtain synthesized speech.
• Local deploymentDownload model resources from GitHub, Hugging Face, or the Moda community, deploy them in a local GPU environment, and use them for secondary development or customization.

Application scenarios of GLM-TTS

• Intelligent voice assistantGLM-TTS provides natural and fluent voice feedback for intelligent voice assistants, supports multiple languages and emotional expression, and generates context-appropriate voice interactions based on user commands to enhance user experience.
• Audiobooks and Audio Content CreationGLM-TTS can quickly generate voice content with different styles and emotions, supports multiple timbre switching and dialect reading, and meets the diverse audio creation needs of audiobooks, podcasts and other content.
• Education and TrainingGLM-TTS helps learners correct the pronunciation of polyphonic and uncommon characters through refined pronunciation control, supports multilingual and dialect teaching, and improves the quality and efficiency of education.
• Entertainment and GamesGLM-TTS generates dialects and emotions for game characters and entertainment content, enhancing the immersion and fun of games and entertainment.
• Customer service and intelligent interactionGLM-TTS adjusts the voice style based on the user's emotions, generating gentle or patient voice responses to improve the interactive experience and customer satisfaction of intelligent customer service.