Mini-Omni - An open-source, end-to-end real-time voice dialogue model

What is Mini-Omni?

Mini-Omni is an open-source, end-to-end speech dialogue model capable of real-time speech input and output, enabling "thinking while speaking" in conversations. The model is designed to support direct speech-to-speech dialogue without requiring additional Automatic Speech Recognition (ASR) or Text-to-Speech (TTS) systems. Mini-Omni employs a text-guided speech generation method, using a batch parallel strategy to improve performance during inference while maintaining the original model's language capabilities.

Mini-Omni's main functions

• Real-time voice interactionIt can conduct end-to-end real-time voice conversations without relying on additional automatic speech recognition (ASR) or text-to-speech (TTS) systems.
• Parallel generation of text and speechDuring the inference process, the model can generate both text and speech output simultaneously, using text information to guide speech generation, thus improving the naturalness and fluency of speech interaction.
• Batch Parallel InferenceThe batch parallel strategy improves the model's inference ability during streaming audio output, making the speech response richer and more accurate.
• Audio language modelingIt converts continuous speech signals into discrete audio tokens, enabling large language models to perform audio modal reasoning and interaction.
• Cross-modal understandingThe model can understand and process inputs from multiple modalities, including text and audio, enabling cross-modal interaction.

Mini-Omni's technical principles

• End-to-end architectureMini-Omni employs an end-to-end design, directly handling the entire process from audio input to text and audio output without the need for traditional separate ASR and TTS systems.
• Text-guided speech generationWhen generating speech output, the model first generates corresponding text information, and then uses this text information to guide speech synthesis. Leveraging the powerful text processing capabilities of language models, the quality and naturalness of speech generation are improved.
• Parallel generation strategyMini-Omni employs a parallel generation strategy, generating text and audio tokens simultaneously during inference. This strategy allows the model to maintain understanding and reasoning about text content while generating speech, resulting in more coherent and consistent dialogue.
• Batch Parallel InferenceTo further enhance the model's inference capabilities, Mini-Omni employs a batch parallel inference strategy. In this strategy, the model processes multiple inputs simultaneously, using text generation to improve the quality of the generated audio.
• Audio encoding and decodingMini-Omni uses an audio encoder (such as Whisper) to convert continuous speech signals into discrete audio tokens, and then uses an audio decoder (such as SNAC) to convert these tokens back into audio signals.

Mini-Omni project address

• Github repository:https://github.com/gpt-omni/mini-omni
• HuggingFace model library:https://huggingface.co/gpt-omni/mini-omni
• arXiv technical paper:https://arxiv.org/pdf/2408.16725

Mini-Omni application scenarios

• Smart assistants and virtual assistantsOn smartphones, tablets, and computers, Mini-Omni can act as a smart assistant, helping users perform tasks through voice interaction, such as setting reminders, querying information, and controlling devices.
• Customer ServiceIn the field of customer service, Mini-Omni can serve as a chatbot or voice assistant, providing 24/7 automated customer support to handle inquiries, resolve issues, and execute transactions.
• Smart Home ControlIn a smart home system, Mini-Omni can control smart devices in the home, such as lights, temperature, and security systems, via voice commands.
• Education and trainingMini-Omni can be used as an educational tool to provide an interactive voice learning experience, helping students learn languages, history, or other subjects.
• In-vehicle systemIn cars, the Mini-Omni can be integrated into the in-vehicle infotainment system, providing voice-controlled navigation, music playback, communication and other functions.