Stream-Omni - A language vision and speech model launched by the Chinese Academy of Sciences and the University of Science and Technology of China

What is Stream-Omni?

Stream-Omni is a large-scale language, vision, and speech model similar to GPT-4o, jointly developed by the Key Laboratory of Intelligent Information Processing of the Institute of Computing Technology, Chinese Academy of Sciences, the Key Laboratory of Artificial Intelligence Security of the Chinese Academy of Sciences, and the University of Chinese Academy of Sciences. It supports multi-modal interactions simultaneously. The model uses a large language model as its backbone, achieving visual text alignment based on sequence-dimensional concatenation and speech-text alignment based on CTC layer-dimensional mapping, efficiently transferring text capabilities to the speech modality. Stream-Omni performs exceptionally well in visual understanding, speech interaction, and visually guided speech interaction tasks, trained on a small amount of full-modal data (such as 23,000 hours of speech data). The model can simultaneously provide intermediate text outputs during speech interaction, such as automatic speech recognition (ASR) transcription and model responses, offering users a richer multimodal interactive experience.

Stream-Omni's main functions

• Multimodal input and outputIt supports multiple modalities of input, including text, visual (image) and voice, and can generate text and voice responses simultaneously.
• Seamless "listen and watch" experienceDuring voice interaction, it can output intermediate text results in real time (such as automatic speech recognition ASR transcription and model response), providing users with a richer interactive experience.
• High-efficiency trainingTraining requires only a small amount of full-modal data (such as 23,000 hours of speech data), resulting in low data requirements and high training efficiency.
• Flexible interaction modesIt supports multi-modal interaction combinations, including text + visual → text, text + visual → voice, voice + visual → text, voice + visual → voice, etc., to meet the interaction needs in different scenarios.
• Visual understanding and voice interactionIt performs well in visual understanding and voice interaction tasks, and can accurately understand and generate text and voice information related to visual content.

The technical principles of Stream-Omni

• LLM-based backbone architectureBased on a large language model (LLM) at its core, it provides fundamental support for multimodal interaction by leveraging its powerful language understanding and generation capabilities.
• Visual text alignmentBased on the sequence dimension concatenation method, the visual features extracted by the visual encoder are concatenated with the text input and then input into the LLM to achieve alignment of visual and text modalities.
• Voice-to-text alignment: Introducing a layer-dimensional mapping based on CTC (Connectionist Temporal Classification), adding speech layers at the bottom and top of LLM to realize speech-to-text mapping and text-to-speech generation, aligning speech modalities with text modalities.
• Multi-task learningBased on a multi-task learning strategy, the model is trained simultaneously on visual text, speech text, and full-modal (visual + text + speech) tasks, enabling it to better understand and generate multimodal content.
• Real-time speech generationBased on a unique speech layer design and layer dimension mapping, Stream-Omni generates corresponding speech output in real time while generating text, enabling smooth voice interaction.
• Combining data-driven and supervised learningThe model relies on a small amount of multimodal data for training. Based on a carefully designed alignment mechanism and multi-task learning, it can achieve efficient modal alignment and interaction capabilities on limited data.

Stream-Omni project address

• GitHub repositoryhttps://github.com/ictnlp/Stream-Omni
• HuggingFace model libraryhttps://huggingface.co/ICTNLP/stream-omni-8b
• arXiv technical paper: https://arxiv.org/pdf/2506.13642

Application scenarios of Stream-Omni

• Intelligent vehicle systemDrivers can use voice commands to query routes and obtain traffic conditions. The system combines visual information (such as navigation maps and traffic camera images) to display text prompts and voice feedback in real time, improving driving safety and interaction efficiency.
• Educational aidsIn educational settings, students ask questions via voice, and the system provides detailed text explanations and voice answers based on the visual content of the textbook (such as charts and pictures), helping students better understand and learn the knowledge.
• Smart Home ControlAs a smart home assistant, users control home appliances based on voice commands. The system combines visual input (such as environmental information captured by a camera) to provide text or voice feedback, enabling smarter and more convenient home control.
• Medical auxiliary diagnosisWhen doctors review patient medical records, they can query key information based on voice commands. The system combines visual reports (such as X-rays and CT images) to provide detailed text analysis and voice explanations, assisting doctors in making more accurate diagnoses.
• Intelligent customer serviceIn the customer service field, customer service personnel communicate with customers using voice, and the system displays relevant text information and visual cues (such as product images and operation flowcharts) in real time, helping customer service personnel quickly understand customer needs and provide accurate answers, thereby improving service quality and efficiency.