KaLM-Embedding - A series of text embedding models launched by Tencent

What is KaLM-Embedding?

KaLM-Embedding is a series of high-performance text embedding models launched by Tencent, which improve the performance of text embedding through advanced training techniques and high-quality data. The latest version, KaLM-Embedding-V2, has made several innovations in architecture and training methods, such as removing the causal attention mask to achieve bidirectional representation learning and adopting a multi-stage training process (including pre-training, fine-tuning, and contrastive distillation), which significantly improves the model's generalization ability and semantic understanding ability. The latest KaLM-Embedding-Gemma3-12B-2511 is an important version of the series, based on a larger parameter scale (12B parameters), further optimizing model performance and making it suitable for complex tasks requiring higher accuracy.

Main functions of KaLM-Embedding

• High-efficiency text embedding generationKaLM-Embedding can efficiently convert text into fixed-length embedding vectors, making it suitable for a variety of natural language processing tasks, such as retrieval, classification, and semantic matching.
• Multilingual and cross-linguistic abilitiesIt supports multilingual text embedding, enabling semantic alignment and cross-language retrieval between different languages, thus improving the performance of cross-language tasks.
• Flexible embedding dimensionsIt supports flexible dimensional embeddings and uses Matryoshka representation learning technology to maintain high performance across different dimensions, adapting to various application scenarios.
• Strong downstream task adaptationIt is suitable for a variety of downstream tasks, including text classification, semantic matching, information retrieval and cluster analysis, providing comprehensive support for natural language processing.

The technical principle of KaLM-Embedding

• Bidirectional attention mechanismThe traditional causal attention mask is removed, and a bidirectional attention mechanism is adopted, which enables the model to consider contextual information at the same time and improves the accuracy of semantic representation.
• Mean poolingThe text sequence is converted into a fixed-length embedding vector using a simple mean pooling method, ensuring that the model output embeddings are applicable to a variety of downstream tasks.
• Multi-stage training processThe model's embedding capability is gradually improved by combining three stages: pre-training, fine-tuning, and comparative distillation. The pre-training stage uses large-scale weakly supervised data, the fine-tuning stage utilizes high-quality supervised data, and the comparative distillation stage learns fine-grained knowledge from stronger teacher models.
• Focus reweighting mechanismFocus-style reweighting focuses on samples that are difficult to learn from, optimizes the training process, and improves the model's ability to learn from difficult samples.
• Online difficult sample poolingDynamically generate difficult negative samples to continuously provide training signals with a large amount of information, avoid outdated negative samples, and enhance the model's discriminative ability.
• Matryoshka means learningIt supports flexible embedding dimensions and maintains high performance across different dimensions through Matryoshka technology, adapting to various application scenarios.
• High-quality data supportWe use diverse and high-quality datasets for training, and combine techniques such as task-specific instructions, hard-burden sample mining, and multi-class labeling to ensure embedding quality.
• Comparative learning and distillation: Using the InfoNCE loss function for contrastive learning, fine-grained soft signals are learned from stronger teacher models through contrastive distillation, further optimizing model performance.
• Temperature coefficient adjustmentIntroducing a temperature coefficient into comparative distillation optimizes the distribution of the learning signal and improves the model's learning efficiency by adjusting the temperature coefficient.
• Flexible model architectureBased on a compact model architecture (such as a parameter scale of 0.5B), high-performance text embedding is achieved while maintaining efficiency.

KaLM-Embedding model series versions

• KaLM-Embedding-V1This is the initial version of the series, based on a compact model architecture, employing causal attention masks, and primarily designed for basic text embedding tasks.
• KaLM-Embedding-V2Based on V1, several improvements were made, including removing the causal attention mask to achieve bidirectional representation learning and introducing a multi-stage training process, including pre-training, fine-tuning, and contrastive distillation, which significantly improved model performance.
• KaLM-Embedding-V2.5It is a further optimized version of V2, which learns fine-grained soft signals from a stronger teacher model by comparative distillation, further improving the model's embedding ability and generalization performance.
• KaLM-Embedding-Gemma3-12B-2511This is the latest version, based on a larger parameter scale (12B parameters), which further optimizes model performance and is suitable for complex tasks requiring higher accuracy.

KaLM-Embedding project address

• Project official websitehttps://kalm-embedding.github.io/
• HuggingFace model libraryhttps://huggingface.co/tencent/KaLM-Embedding-Gemma3-12B-2511
• arXiv technical paper: https://arxiv.org/pdf/2506.20923

Application scenarios of KaLM-Embedding

• Text classificationIt can efficiently classify text, helping users quickly identify the topic and category of the text.
• Semantic matchingIt can accurately determine the semantic similarity between different texts and has wide applications in fields such as search engines and recommendation systems.
• Information clusteringIt automatically categorizes similar text information, making it easier for users to manage and analyze large amounts of text data.
• Search RecommendationsBy leveraging semantic understanding, we can improve the relevance of search results and the accuracy of recommendations, providing users with a more personalized experience.
• Multilingual understandingIt supports multilingual semantic alignment and performs well in tasks such as cross-language retrieval and translation, enabling more accurate cross-language semantic understanding.