AB
AB
AiBoss

All Categories

project

OSUM - An open-source speech understanding model from Northwestern Polytechnical University

OSUM (Open Speech Understanding Model) is an open-source speech understanding model developed by the Audio, Speech and Language Processing Research Group of the School of Computer Science at Northwestern Polytechnical University. OSUM combines the Whisper encoder and Qwen2 LLM...

What is OSUM?

OSUM (Open Speech Understanding Model) is an open-source speech understanding model developed by the Audio, Speech and Language Processing Research Group of the School of Computer Science at Northwestern Polytechnical University. Combining the Whisper encoder and Qwen2 LLM, OSUM supports various speech tasks, including Automatic Speech Recognition (ASR), Speech Emotion Recognition (SER), and Speaker Gender Classification (SGC). OSUM is based on an "ASR+X" multi-task training strategy, using modality alignment and target task optimization to achieve efficient and stable training. Trained on approximately 50,000 hours of diverse speech data, OSUM demonstrates excellent performance across multiple tasks, with outstanding performance in Chinese ASR and multi-task generalization.

Main functions of OSUM

  • Speech recognitionIt converts speech to text and supports multiple languages and dialects.
  • Timestamped speech recognitionWhile recognizing the speech content, output the start and end times of each word or phrase.
  • Voice event detection: Recognize specific events in speech (such as laughter, coughing, background noise, etc.).
  • Voice emotion recognition: Analyze the emotional state (such as happiness, sadness, anger, etc.) in speech.
  • Speech style recognition: Identify the speaker's style (such as news broadcasting, customer service dialogue, everyday speech, etc.).
  • Speaker gender classification: Determine the speaker's gender (male or female).
  • Speaker age predictionPredict the speaker's age range (e.g., child, adult, elderly).
  • Voice-to-text chatIt converts voice input into natural language responses for use in dialogue systems.

OSUM's technical principles

  • Speech EncoderThe Whisper-Medium model (769M parameters) is used to encode the speech signal into a feature vector.
  • AdaptorIt contains 3 convolutional layers and 4 Transformer layers, used to adapt speech features to the input of the language model.
  • LLM (Language Model)Based on Qwen2-7B-Instruct, it is finely tuned using LoRA (Low-Rank Adaptation) to adapt to multi-tasking requirements.
  • Multi-task training strategy:
    • ASR+X Training ParadigmSimultaneously train the speech recognition (ASR) task and an additional task (such as SER, SGC, etc.). Based on shared features and optimization objectives, improve the model's generalization ability and stability.
    • Natural Language PromptBased on providing different natural language prompts to the LLM, the model is guided to perform different tasks.
  • Data processing and trainingThe Whisper model was trained on approximately 50,000 hours of diverse speech data for multi-task training. The dataset included both open-source and internally processed data. Training was divided into two phases: first, the Whisper model was fine-tuned for multi-task training, and then it was combined with an LLM for further supervised training.

OSUM project address

OSUM Application Scenarios

  • Intelligent Customer ServiceBased on speech recognition and sentiment analysis, it automatically understands customer needs and provides personalized services.
  • Smart Home: Recognize voice commands and background events to optimize the voice interaction experience.
  • Educational toolsAnalyze student speech to provide personalized learning feedback.
  • Mental health monitoring: Detect emotional changes in speech to assist in mental health assessment.
  • Multimedia content creationAutomatically generates subtitles and tags to assist in video editing.
Back