VITA-Audio - An open-source, end-to-end multimodal speech model with low latency and fast inference.

What is VITA-Audio?

VITA-Audio is an open-source, end-to-end multimodal speech model characterized by low latency and fast inference speed. Through a lightweight Multimodal Cross-Label Prediction (MCTP) module, it can generate audio output in the first forward propagation, significantly reducing the time to generate the first audio tag block and substantially lowering latency in streaming scenarios. Employing a four-stage progressive training strategy, it achieves inference speeds 3 to 5 times faster than similarly sized open-source models with 7 billion parameters. It also demonstrates excellent performance in benchmark tests for tasks such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Spoken Question Answering (SQA).

Main functions of VITA-Audio

• Real-time chat functionalityVITA-Audio is a multimodal large language model capable of generating audio output in the first forward pass, achieving real-time dialogue capabilities with extremely low latency. Through a lightweight multimodal cross-tag prediction (MCTP) module, the model can efficiently generate multiple audio tags in a single forward pass, significantly reducing the latency of generating the first audio in streaming scenarios.
• High-efficiency reasoning accelerationWith 7 billion parameters, VITA-Audio achieves 3 to 5 times faster inference. Thanks to its unique four-stage progressive training strategy, it achieves efficient inference while minimizing speech quality loss.
• Multimodal interaction capabilitiesVITA-Audio can understand and process multimodal data, including audio and text. It has broad application potential in multimodal interaction scenarios, such as voice assistants and intelligent customer service, providing a richer and more natural interactive experience.
• Speech generation and recognitionVITA-Audio significantly outperforms open-source models of similar size in multiple benchmark tests for Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Spoken Question Answering (SQA) tasks. It demonstrates high accuracy and efficiency in speech generation and recognition, meeting the voice interaction needs of various scenarios.

VITA-Audio Technical Principles

• Multimodal Cross-Label Prediction (MCTP) ModuleThis is one of the key innovations of VITA-Audio. The module can efficiently generate multiple audio tags in a single model forward propagation, significantly reducing the latency of generating the first audio in streaming scenarios and achieving extremely low-latency real-time dialogue capabilities.
• TiCodec moduleThis is used to encode continuous speech waveforms into discrete tokens and decode discrete tokens back into waveforms. It enables unified modeling of speech and text in the same sequence space.
• Non-autoregressive (NAR) and autoregressive (AR) decodersNAR decoders predict the distribution of all speech tokens in a single sentence, which is fast but relatively inaccurate; AR decoders rely on previously generated tokens, producing higher quality tokens but are slower. VITA-Audio combines the two, first using NAR to generate an initial speech token distribution, and then using AR for refinement, balancing speed and quality.
• Four-stage progressive training strategy
  • Phase 1: Visual-Language Alignment:Build powerful visual capabilities by training a visual adapter and fine-tuning the model using descriptive captions and visual question-answering data.
  • Phase Two: Fine-tuning of Audio InputThe audio encoder was trained using speech transcription paired data and then fine-tuned using speech question-and-answer data, enabling the model to understand and respond to audio input.
  • Phase 3: Fine-tuning of audio outputTrain the audio decoder to achieve end-to-end voice output without the need for an external TTS module.
  • Phase 4: Multimodal Command Fine-tuningFurther optimize the model's performance in multimodal tasks and improve its ability to process inputs from different modalities.

VITA-Audio project address

• Github repository:https://github.com/VITA-MLLM/VITA-Audio
• HuggingFace model library:https://huggingface.co/collections/VITA-MLLM/vita-audio
• arXiv technical paper:https://arxiv.org/pdf/2505.03739

Application scenarios of VITA-Audio

• Smart Home ControlVITA-Audio can understand voice commands and control smart devices in your home, such as lights, temperature, and security systems.
• Intelligent Customer ServiceIn online customer service scenarios, VITA-Audio can simultaneously understand users' text descriptions, image uploads, or voice messages, and answer questions quickly and accurately.
• Educational SupportIn online education platforms or learning applications, VITA-Audio can assist students in their learning. For example, by analyzing the learning materials uploaded by students (such as textbook images, teaching videos, etc.) and combining them with students' questions, it can provide services such as explanations of knowledge points and solutions to exercises.
• Medical assistanceVITA-Audio can help doctors interpret medical images, analyze medical record data, and combine the latest medical research literature to provide auxiliary diagnostic and treatment suggestions.
• Content creationIn media companies or self-media platforms, VITA-Audio can assist content creators in generating articles, video scripts, etc., by understanding the creative theme and style requirements, and providing creative inspiration and writing suggestions.