AB
AB
AiBoss

All Categories

project

OmniAudio-2.6B - An edge-side multimodal audio language model launched by Nexa AI

OmniAudio-2.6B is an audio language model from Nexa AI, designed for edge deployment, enabling fast and efficient audio-to-text processing. OmniAudio-2.6B is a multimodal model with 260 million parameters, fused with Gemma-2-2...

What is OmniAudio-2.6B?

OmniAudio-2.6B is an audio language model from Nexa AI, designed for edge deployment, enabling fast and efficient audio-text processing. OmniAudio-2.6B is a multimodal model with 260 million parameters, integrating Gemma-2-2b, Whisper Turbo, and a custom projection module to optimize the integration of automatic speech recognition and language models, reducing latency and resource consumption. On the 2024 Mac Mini M4 Pro, OmniAudio-2.6B demonstrated decoding speeds 5.5 to 10.3 times faster than Qwen2-Audio-7B-Instruct. OmniAudio-2.6B is suitable for various scenarios, including voice question answering, dialogue, and creative content generation. Running on local devices based on the Nexa SDK, it provides users with a powerful edge AI solution.

Main functions of OmniAudio-2.6B

  • Speech recognition and transcriptionIt converts voice input into text, suitable for scenarios such as meeting minutes and voice notes.
  • Voice Q&AUsers ask questions to the model via voice, and the model can understand the questions and provide text answers.
  • voice dialogueThe model can participate in voice dialogues, understand voice input, and generate corresponding text responses.
  • Creative content generationUsers can request the model to generate creative content, such as poems and stories, based on voice input.
  • Audio SummaryThe model can understand and summarize long-term speech recordings, providing a summary of key information.

Technical Principles of OmniAudio-2.6B

  • Integrated ArchitectureIt integrates Gemma-2-2b, Whisper Turbo, and a custom projection module to reduce latency and resource consumption caused by concatenating traditional ASR and LLM models.
  • sparsity utilizationBased on the sparsity of the language model embedding space, the projection module maps Whisper's audio tokens to sequences aligned with Gemma text embeddings, achieving effective fusion of audio and text.
  • Three-stage training process:
    • Pre-trainingUsing the MLS English 10k transcription dataset, a special token is introduced to distinguish between transcription and completion tasks.
    • Supervised fine-tuning (SFT): Create synthetic datasets based on transcription datasets for instruction tuning, enabling models to understand and process dialogue audio input.
    • Direct Preference Optimization (DPO)The model output is evaluated based on the GPT-4o API, and error responses are identified and optimized to improve the accuracy of the model.
  • High-efficiency inference engineThe Nexa SDK is a C++ inference engine based on GGML, designed specifically for deploying audio language models on edge devices, enabling efficient audio language model inference.
  • Quantification and optimizationThe model supports FP16 and Q4_K_M quantization versions, reducing memory and storage requirements and adapting to resource-constrained edge devices.

Project address for OmniAudio-2.6B

Application scenarios of OmniAudio-2.6B

  • Smart assistants and virtual assistantsIt serves as the core of voice interaction in smartphones and smart home devices, providing fast-response voice recognition and natural language understanding capabilities.
  • In-vehicle systemIt can be integrated into the car to provide functions such as voice control, navigation, and entertainment system operation, thereby improving driving safety and convenience.
  • Meeting minutes and transcriptionAutomatically record and transcribe meeting content during business meetings, generate meeting summaries, and improve work efficiency.
  • Education and LearningIt assists in language learning by providing speech recognition and feedback to help learners improve their pronunciation and language skills.
  • HealthcareIn a medical environment, voice commands are used to control medical equipment or provide voice interaction services to patients.
Back