AB
AB
AiBoss

All Categories

project

SpeechGPT 2.0-preview - An end-to-end real-time speech interaction model developed by Fudan University

SpeechGPT 2.0-preview is a human-like real-time interactive system developed by the OpenMOSS team at Fudan University. Trained on millions of hours of Chinese speech data, it employs an end-to-end architecture and achieves a high degree of integration between speech and text modalities...

What is SpeechGPT 2.0-preview?

SpeechGPT 2.0-preview is a human-like real-time interactive system developed by the OpenMOSS team at Fudan University. Trained on millions of hours of Chinese speech data, it employs an end-to-end architecture to achieve a high degree of integration between speech and text modalities. The model features human-like spoken expression, millisecond-level low-latency response, and supports natural and fluent real-time interrupted interaction. It can precisely control speech rate, emotion, style, and timbre, achieving intelligent switching. SpeechGPT 2.0-preview possesses various speech talents, such as poetry recitation, storytelling, and speaking in dialects.

Main functions of SpeechGPT 2.0-preview

  • Emotion and Style ControlIt supports precise control of multiple emotions (such as weakness and joy), multiple timbres (switching between male and female voices), and multiple styles (poetry recitation, dialect imitation), with outstanding role-playing ability.
  • Interrupting interaction in real timeMillisecond-level response speed supports instant interruption and resumption in natural conversations.
  • Text capability integrationBuilding upon the expressive power of speech, it retains the intelligence of the text model and supports functions such as tool access, online search, and integration with external knowledge bases.
  • Multitasking compatibilityIt can handle scenarios such as long document parsing and multi-turn dialogues, and its performance for short text tasks is not reduced due to its ability to handle long contexts.

Technical Principles of SpeechGPT 2.0-preview

  • End-to-end speech modeling
    • Ultra-low bit rate streaming voice codecThe self-developed ultra-low bit rate streaming voice codec can process 24kHz voice input, compress the voice to 75 tokens per second, support streaming input and output, and achieve real-time interaction with a latency of less than 200ms.
    • Semantic-acoustic joint modelingIt directly processes speech input and generates speech or text output through semantic-acoustic joint modeling, without the need for traditional cascaded ASR (speech recognition) and TTS (speech synthesis) modules.
  • Speech-text hybrid modeling
    • Codec PatchifyBy aggregating speech tokens from adjacent time steps into a unified vector using Codec Patchify technology, the modal differences between speech and text sequences are effectively reduced, alleviating conflict issues in cross-modal modeling.
    • Multi-stage training processIt includes modality adaptation pre-training, cross-modal instruction fine-tuning, and chained modality fine-tuning, taking into account both text and speech capabilities, and avoiding a decrease in the model's intelligence when learning speech capabilities.
  • Speech-text alignment pre-trainingThrough sufficient pre-training with speech-text alignment, the model can "emerge" with generalization of speech styles, such as controlling speech rate without needing speech rate adjustment data, or mimicking the tone and style of unfamiliar characters.
  • Model architecture design
    • Joint speech and text modelingThe speech-text LLM takes speech and text representations as input and outputs simultaneously. The hidden state of the LLM is used for both speech and text decoding tasks.
    • Autoregressive Language Model with Multiple DecodersAn autoregressive language model with multiple decoding heads was designed as a patch decoder, which can decode step by step in an autoregressive manner, generating multiple RVQ codec tokens for one time step at a time, and outputting speech.

Project address for SpeechGPT 2.0-preview

Application scenarios of SpeechGPT 2.0-preview

  • Smart AssistantIt can be used as an intelligent assistant in fields such as customer service, education, or healthcare, providing services such as real-time spoken language practice and emotional companionship.
  • Content creationIt can automatically generate audiobooks, poetry recitations, or dialect content, enriching multimedia creation formats.
  • Accessible communicationProvides real-time speech-to-text and text synthesis services for people with hearing or speech impairments.
Back