AB
AB
AiBoss

All Categories

project

LLaMA-Omni - A low-latency, high-quality voice interaction model launched by the Chinese Academy of Sciences

LLaMA-Omni is a novel model architecture developed by researchers at the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, for achieving low-latency, high-quality speech interaction with large language models (LLMs). It integrates pre-trained...

What is LLaMA-Omni?

LLaMA-Omni is a novel model architecture developed by researchers from the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, designed for low-latency, high-quality speech interaction with Large Language Models (LLMs). By integrating a pre-trained speech encoder, speech adapter, LLM, and a real-time speech decoder, it directly and rapidly generates text and speech responses from speech commands, eliminating the traditional step of transcribing speech into text and thus improving response speed. The model is built on the latest LLaMA-3.1-8B-Instruct model and trained using a self-built InstructS2S-200K dataset, generating responses quickly with a latency as low as 226 milliseconds. Furthermore, LLaMA-Omni boasts high training efficiency, completing training on four GPUs in less than three days, laying the foundation for the efficient development of future speech interaction models based on the latest LLM.

Main functions of LLaMA-Omni

  • Low-latency speech recognition: Quickly generate responses from voice commands, reducing waiting time.
  • Direct speech to text responseIt generates a text response directly without first transcribing the speech into text.
  • High-quality speech synthesisIt can generate corresponding speech output while generating text responses.
  • Efficient training processTraining can be completed with fewer computing resources (such as 4 GPUs) and in a shorter time (less than 3 days).
  • Streaming audio decodingReal-time speech synthesis is achieved based on a non-autoregressive streaming Transformer model.
  • Multimodal interactionIt combines text and voice modes to provide a more natural and human-like interactive experience.

LLaMA-Omni's technical principles

  • Speech EncoderThe pre-trained Whisper-large-v3 model is used as the speech encoder to extract feature representations from the user's voice commands.
  • Speech AdapterThis maps the output of the speech encoder to the embedding space of a large language model (LLM). Downsampling reduces the sequence length, enabling the model to process the speech input.
  • Large Language Model (LLM)Based on Llama-3.1-8B-Instruct as an LLM, it boasts powerful text generation capabilities. It generates text responses directly from speech commands, eliminating the need for intermediate speech-to-text transcription steps.
  • Streaming Speech DecoderA non-autoregressive (NAR) streaming Transformer architecture is employed. Connectivity-Temporal Classification (CTC) is used to predict the discrete unit sequences corresponding to the speech response.
  • Two-stage training strategyPhase 1: Training the model to generate text responses directly from voice commands. Phase 2: Training the model to generate voice responses.
  • Dataset Construction (InstructS2S-200K):It contains 200K voice commands and their corresponding text and voice responses. Based onThe model is trained to adapt to voice interaction scenarios.

LLaMA-Omni project address

Application scenarios of LLaMA-Omni

  • Smart assistants and virtual assistantsProvide voice interaction services on smartphones, smart home devices, and personal computers.
  • Customer ServiceIn call centers and customer support systems, speech recognition and responses are used to handle customer inquiries and issues.
  • Education and trainingIt provides an interactive learning experience, including language learning, course explanations, and interactive teaching.
  • Medical consultationIn telemedicine and health consultations, voice interaction is used to provide medical information and advice.
  • automotive industryIt can be integrated into the vehicle system to provide voice-controlled navigation, entertainment and communication functions.
  • Accessibility and assistive technologies: To help visually impaired or mobility-impaired users operate devices and services using voice interaction.
Back