IndexTTS2 - The latest open-source text-to-speech model from Bilibili

What is IndexTTS2?

IndexTTS2, a new text-to-speech (TTS) model developed by Bilibili's voice team, has been officially open-sourced. The model achieves significant breakthroughs in emotion expression and duration control, and is the first autoregressive TTS model to support precise duration control. It supports zero-sample voice cloning, accurately replicating timbre, rhythm, and speaking style with just one audio file, and supports multiple languages. IndexTTS2 implements emotion-timbre separation control, allowing users to independently specify the source of timbre and emotion. The model features multimodal emotion input capabilities, supporting emotion control through emotion reference audio, emotion description text, or emotion vectors.

Main functions of IndexTTS2

• Zero-sample speech cloningIt requires only one reference audio file to accurately mimic the voice, intonation, and rhythm of that file, supports multiple languages, and enables highly personalized voice synthesis.
• Emotion and Duration ControlIt supports zero-sample emotion cloning, simulating emotions based on reference audio, and allows control of voice emotions through text descriptions. It also features a world-first precise duration control function, allowing users to set the length of generated speech to meet the needs of scenarios such as movie dubbing and timeline synchronization.
• High-fidelity sound qualityWith an audio sampling rate of up to 48kHz, it supports lossless audio output. Combined with an optimized vocoder, it generates natural, fluent, and emotional speech, reducing mechanical sounds and improving sound quality.
• Multimodal input supportIt supports multiple input methods such as text and audio. Users can control the style and emotion of the generated speech through text descriptions, reference audio, or emotion vectors, providing a more flexible speech synthesis experience.
• Localized deployment and open sourceIt supports fully localized deployment, plans to open up model weights, provides developers with powerful tools, empowers more application scenarios, and promotes the widespread application of TTS technology.

The technical principles of IndexTTS2

• Modular architectureIt consists of three core modules: text-to-speech (T2S), semantic-to-melody (S2M), and vocoder. These modules work together to achieve the conversion from text to high-quality speech.
• Decoupling emotion and timbreBy using techniques such as gradient inversion layers, emotional and timbre features are decoupled from cues, allowing users to independently control emotions and timbre, thus achieving more flexible speech synthesis.
• Multi-stage training strategyA multi-stage training strategy is adopted to overcome the lack of high-quality emotional data, enhance the model's emotional expression ability, and improve the naturalness and emotional richness of speech synthesis.
• High sampling rate and optimized vocoderWith an audio sampling rate of up to 48kHz, combined with optimized vocoders such as BigVGAN2, it generates high-fidelity, natural and fluent speech, reducing mechanical sounds and improving sound quality.
• Zero-sample cloning technologyThrough advanced zero-sample cloning technology, it can accurately imitate voice, intonation and rhythm with only one reference audio, supports multiple languages, and achieves highly personalized voice synthesis.

IndexTTS2 project address

• Project official websitehttps://index-tts.github.io/index-tts2.github.io/
• Github repositoryhttps://github.com/index-tts/index-tts
• HuggingFace model libraryhttps://huggingface.co/IndexTeam/IndexTTS-2
• arXiv technical paper: https://arxiv.org/pdf/2506.21619

Upgrades of IndexTTS2 and IndexTTS1.5

• Precise duration control functionIndexTTS2 is the first autoregressive TTS model to support precise duration control, allowing the length of generated audio to be specified down to the millisecond level. IndexTTS1.5 does not have this feature.
• Emotional timbre separation modelingIndexTTS 2 implements separate modeling of emotion and timbre, allowing users to independently control both. In IndexTTS 1.5, the control over emotion and timbre was not as precise.
• Multimodal Emotional Input SupportIndexTTS2 supports controlling the emotion of generated speech through various methods, such as audio emotion reference, text emotion description, or emotion vectors. While IndexTTS1.5 also supports emotion control, its methods are relatively limited.
• Stronger emotional expression abilityIndexTTS 2 has been optimized in terms of emotional expression, and can better simulate various emotional states. IndexTTS 1.5 has also improved in terms of emotional expression, but it still lags behind the second generation.
• Better voice stabilityIndexTTS2 enhances the stability of speech generation through technologies such as GPT latent representations and soft instruction mechanisms. IndexTTS 1.5 also improved in terms of stability, but the second generation further optimizes upon that.

Application scenarios of IndexTTS2

• Film and television dubbingIt can provide high-quality dubbing for film and television works, accurately control the duration and emotion of voice, and meet the needs of audio-visual synchronization.
• virtual charactersTo give virtual characters natural and emotional voices, thereby enhancing the interactive experience and user immersion.
• audiobooksIt generates natural and fluent speech, providing high-quality speech synthesis for audiobook production and enhancing the listener's auditory experience.
• Smart AssistantIn scenarios such as intelligent assistants and voice broadcasts, it provides natural and smooth voice interaction to enhance the user experience.
• Advertising productionIt provides personalized voice synthesis for advertising production, supports multiple languages and emotional styles, and enhances the appeal of advertisements.
• EducationIn educational software and online courses, vivid audio explanations are provided to help students better understand and learn.