Omnilingual ASR - An automatic speech recognition system launched by Meta AI

What is Omnilingual ASR?

Omnilingual ASR, developed by Meta AI, is an automated speech recognition system supporting over 1600 languages, including 500 low-resource languages. Omnilingual ASR achieves superior performance by extending the wav2vec 2.0 encoder to 7 billion parameters and introducing two decoders, with a character error rate below 10% for 78% of languages. The Omnilingual ASR framework is community-driven, allowing users to extend it to new languages with only a small number of samples. Meta AI has also open-sourced the Omnilingual ASR Corpus dataset and the new self-supervised large-scale multilingual speech representation model, Omnilingual wav2vec 2.0, contributing to the development of global speech technology and promoting language equality and cultural exchange.

Main functions of Omnilingual ASR

• Multilingual speech transcriptionOmnilingual ASR can convert speech to text in more than 1,600 languages, including many low-resource languages and languages that have never been transcribed by AI before.
• Community Expansion CapabilitiesUsers can extend the model to new languages by providing a small number of audio and text samples, without the need for a large amount of training data or expertise.
• High performance and low error rateIn 78% of the languages, the character error rate (CER) is below 10%, reaching an industry-leading level.
• Multiple model optionsOffers models ranging from the lightweight 300M to the robust 7B, suitable for different devices and use cases.
• Open source and data sharing: Open source Omnilingual wav2vec 2.0 model and Omnilingual ASR Corpus dataset to support further development and research by developers and researchers worldwide.

The technical principle of Omnilingual ASR

• wav2vec 2.0 extensionIt extends the wav2vec 2.0 encoder to 7 billion parameters, enabling the extraction of rich multilingual semantic representations from raw speech data.
• Dual decoder architectureTwo decoders are used: a traditional connectionist temporal classification (CTC) decoder and a Transformer-based decoder, the latter of which borrows techniques from large language models (LLM) to significantly improve the performance of long-tail languages.
• Contextual learning abilityInspired by LLM, the model can quickly adapt to a new language with a small number of contextual samples, without the need for large-scale training data or complex adjustments.
• Large-scale multilingual datasetsThe training corpus integrates publicly available datasets and speech recordings provided by the community, covering a large number of low-resource languages and providing a broad linguistic foundation for the model.

Omnilingual ASR project address

• Project official website: https://ai.meta.com/blog/omnilingual-asr-advancing-automatic-speech-recognition/
• GitHub repositoryhttps://github.com/facebookresearch/omnilingual-asr
• HuggingFace model libraryhttps://huggingface.co/datasets/facebook/omnilingual-asr-corpus
• Technical Papers: https://ai.meta.com/research/publications/omnilingual-asr-open-source-multilingual-speech-recognition-for-1600-languages/

Application scenarios of Omnilingual ASR

• Cross-language communicationIt helps people from different language backgrounds to communicate in real time, breaking down language barriers and promoting international cooperation and cultural exchange.
• Low-resource language protection: Provides high-quality speech transcription tools for endangered or low-resource languages, contributing to language preservation and transmission.
• Education and LearningIt can assist teaching in multilingual education, help students practice pronunciation, or provide instant voice translation for language learners.
• Voice assistant extensionAdd more language support to intelligent voice assistants so that they can serve a wider range of users.
• Content creation and mediaAutomatically transcribes multilingual video and audio content, improving content creation efficiency and supporting multilingual subtitle generation.