MEXMA - Meta's pre-trained cross-language sentence encoder

What is MEXMA?

MEXMA, developed by Meta AI, is a novel pre-trained cross-lingual sentence encoder. It improves the quality of sentence representations by combining sentence-level and word-level objectives. During training, MEXMA predicts masked words in another language using sentence representations from one language, supporting direct updates to the sentence representation and all words in the encoder. MEXMA demonstrates superior performance on multiple tasks, outperforming existing pre-trained cross-lingual sentence encoders such as LaBSE and SONAR. MEXMA supports 80 languages and performs well in downstream tasks such as sentence classification.

Main functions of MEXMA

• Cross-language sentence encodingMEXMA encodes sentences in different languages into fixed-size vectors, which are then compared and manipulated in a shared multilingual space.
• Combining sentence and word-level objectivesMEXMA improves the quality and alignment of sentence representation by simultaneously considering the overall meaning of the sentence and the contribution of each word in the sentence.
• Multitasking performance improvementMEXMA performs well in a variety of downstream tasks, including sentence classification, text mining, and semantic text similarity tasks.
• Supports 80 languagesMEXMA supports up to 80 languages, making it suitable for a wide range of multilingual applications.

MEXMA's technical principles

• Combining sentence-level and word-level goalsMEXMA trains the model using sentence-level objectives and introduces word-level objectives. During training, the model learns the overall representation of the sentence and the representation of each word in the sentence.
• Cross-language masking taskMEXMA uses sentence representations in one language to predict masked words in another language. Cross-language masking tasks force models to learn sentence representations that capture the core information of sentences, ensuring that information is alignable across different languages.
• Directly update the encoderIn MexMA, sentence representations update the encoder, and the representation of each word directly updates the encoder. This bidirectional update mechanism allows the model to learn high-quality sentence and word representations more effectively.
• Symmetric architectureMEXMA is based on a symmetric architecture and simultaneously performs masking and prediction on sentences in two languages, ensuring that the model is balanced between the two languages and can generate two clean sentence vectors, which is crucial for the alignment between languages.
• Non-contrastive loss functionTo avoid model collapse and improve the quality and alignment of representations, MEXMA uses non-contrastive loss functions, such as mean squared error (MSE) loss, to enhance the spatial proximity of semantically equivalent sentences in different languages.

MEXMA's project address

• GitHub repository:https://github.com/facebookresearch/mexma
• HuggingFace model library:https://huggingface.co/facebook/MEXMA
• arXiv technical paper:https://arxiv.org/pdf/2409.12737

Application scenarios of MEXMA

• Cross-language information retrievalMEXMA can be used to convert documents in different languages into the same embedding space, enabling cross-language search and retrieval.
• Machine translationMEXMA improves machine translation systems by enhancing translation quality by providing more accurate cross-language sentence representations.
• Multilingual text classificationWhen processing multilingual text data, MEXMA can classify text content, such as sentiment analysis and topic classification.
• Semantic text similarity assessmentMEXMA assesses the similarity between sentences in different languages and is applicable to comparing and matching similar text content.
• Cross-language question answering systemIn multilingual question-and-answer systems, MEXMA helps understand questions in different languages and find the answers in the corresponding languages.