Gummy - Tongyi's end-to-end speech translation model can generate results in real time in a streaming manner.
Gummy is an end-to-end speech translation model launched by Tongyi Lab at the 2024 Yunqi Conference. The model can generate speech recognition and translation results in real-time streaming, supporting languages including Chinese, English, Cantonese, Japanese, Korean, French, German, etc.
What is Gummy?
Gummy is an end-to-end speech translation model launched by Tongyi Lab at the 2024 Yunqi Conference. The model can generate speech recognition and translation results in real-time streaming, supporting speech input in more than ten languages, including Chinese, English, Cantonese, Japanese, Korean, French, German, Russian, Italian, and Spanish, translating them into the target language. Through an end-to-end approach, the Gummy model reduces translation latency and improves translation quality, achieving state-of-the-art (SOTA) results on multiple test sets. Gummy supports commercial applications such as multilingual mixed translation, terminology intervention, and domain-specific suggestions, enabling fluent translation from various languages to the target language in scenarios such as multinational conferences without specifying the source language.
Gummy's main functions
- Multilingual supportGummy can handle voice input in more than ten languages, including Chinese, English, Cantonese, Japanese, Korean, French, German, Russian, Italian, and Spanish, and translate them into the target language in real time.
- End-to-end translationUnlike traditional cascaded systems, Gummy adopts an end-to-end design, directly translating speech into the target language without relying on an intermediate text stage.
- Low-latency translationGummy's translation latency was reduced to less than 0.5 seconds, which is faster than the simultaneous interpretation latency of human experts.
- High-quality translationGummy achieved SOTA (State of the Art) translation quality results on several industry-recognized open-source test suites.
- Streaming translationGummy supports on-the-speak translation, meaning you can listen and translate simultaneously, making it suitable for real-time communication scenarios.
Gummy's technical principles
- end-to-end designThe Gummy model uses an end-to-end architecture to map the speech input of the source language to the text output of the target language, simplifying the development process and improving system performance.
- Deep Neural NetworksBased on deep learning technology, especially deep neural networks, it learns the complex mapping relationship between speech and text.
- Real-time streaming processingIt supports real-time speech recognition and translation, enabling simultaneous listening and translation.
- wait & predict mechanismThe model employs a special mechanism to automatically determine the timing of translation and optimize translation quality and latency.
Gummy's project address
- Project official websiteThe Gummy voice translation model, available at tongyi.aliyun.com, is partially available for download on the Tongyi APP.
Gummy's application scenarios
- Real-time voice translationThe Gummy model can translate speeches in meetings in real time, providing simultaneous interpretation services for international conferences, multilingual negotiations, and more.
- Education and trainingIn the field of education, Gummy assists language learning by providing real-time translation of multilingual teaching content, helping students and teachers overcome language barriers.
- Tourism and NavigationIt provides travelers with real-time voice translation to help them communicate with locals speaking different languages, or to offer multilingual guidance during navigation.
- Customer ServiceIn the field of customer service, Gummy, as a multilingual customer service assistant, provides fast and accurate language support, thereby improving customer satisfaction.
- Medical consultationIn the medical field, Gummy provides multilingual medical consultation translation services to facilitate communication between doctors and patients.