project
URO-Bench - An AI benchmark tool designed for end-to-end voice dialogue models.
URO-Bench is a comprehensive benchmarking tool for end-to-end speech dialogue models (SDMs). It covers multi-dimensional tasks such as multilingualism, multi-turn dialogue, and paralinguistic information to comprehensively evaluate the performance of speech dialogue models.
What is URO-Bench?
URO-Bench is a comprehensive benchmarking tool for end-to-end speech dialogue models (SDMs). It covers multi-dimensional tasks including multilingual, multi-turn dialogue, and paralinguistic information, comprehensively evaluating the performance of speech dialogue models. The benchmark includes a basic track with 16 datasets covering open-ended question answering, factual question answering, etc.; the advanced track includes 20 datasets covering more complex tasks such as code-switching question answering, speech emotion generation, and multilingual question answering.
URO-Bench's main functions
- Multilingual supportIt covers multiple languages, including English and Chinese, and supports cross-language dialogue tasks.
- Multi-round dialogue assessmentIt includes multi-turn dialogue tasks to evaluate the model's performance in continuous dialogue.
- Sub-language information assessmentIt involves tasks related to paralinguistic information such as speech emotion understanding and speech style generation, and is closer to real-world speech interaction scenarios.
- Basic TrackIt contains 16 datasets, covering various task types such as open-ended question answering, moral summarization, factual question answering, and mathematical word problems.
- Pro TrackIt contains 20 datasets, covering more advanced tasks such as code switching question answering, speech emotion generation, multilingual question answering, and audio understanding.
- Four-step assessment processUsers can quickly obtain model results on all test sets by simply modifying the inference code, configuring scripts, and running the automated evaluation pipeline. Sample code and scripts are provided to facilitate quick user adoption and lower the barrier to entry.
- Multi-indicator evaluationThe model's capabilities in speech understanding, reasoning, and spoken dialogue are comprehensively evaluated using multiple metrics (such as UTMOS, ASR-WER, and emotion understanding accuracy).
- UniversalityIt supports multiple end-to-end voice dialogue models, and users can connect their own models to URO-Bench for evaluation.
- Reference ModelEvaluation results of some pre-trained models (such as Whisper + GPT-4o, GLM-4-Voice, etc.) are provided for reference.
URO-Bench's technical principles
- Text-to-speech (TTS): Use advanced TTS systems (such as F5-TTS, CosyVoice) to convert text data into voice data.
- Automatic Speech Recognition (ASR)Speech data was transcribed into text using ASR systems such as Whisper-large-v3 for evaluation.
- Emotion recognitionUse models such as emotion2vec to evaluate emotional information in speech.
- Multilingual processingIt supports input and output in multiple languages and evaluates the model's cross-language capabilities.
URO-Bench project address
- Github repository:https://github.com/Ruiqi-Yan/URO-Bench
- HuggingFace model library:https://huggingface.co/datasets/Honggao/URO-Bench
- arXiv technical paper:https://arxiv.org/pdf/2502.17810
URO-Bench application scenarios
- Smart Home ControlUsers can control smart home devices, such as lights, temperature settings, and appliance switches, using voice commands. URO-Bench evaluates the voice assistant's performance in understanding user commands and generating natural voice feedback.
- Personal AssistantVoice assistants can help users schedule appointments, remind them of important events, and retrieve information. URO-Bench can test the fluency and accuracy of voice assistants in multi-turn conversations.
- Language learningVoice dialogue models can serve as language learning tools, helping users practice speaking and listening. URO-Bench can evaluate the model's performance in multilingual dialogue and emotional expression, providing language learners with a more natural interactive experience.
- Medical consultationThe voice dialogue model can provide users with initial medical consultations and answer common health questions. URO-Bench can evaluate the model's performance in understanding and generating professional medical information.
- Voice gamesVoice dialogue models can be used to develop voice-interactive games, providing a more immersive experience. URO-Bench can evaluate the model's performance in multi-turn dialogues and emotion generation, providing a reference for game development.