Seed 1.5-Embedding - A vector model launched by ByteDance's Seed team.

What is Seed1.5-Embedding?

Seed1.5-Embedding is the latest vector model released by ByteDance's Seed team, further trained based on Seed1.5 (Doubao-1.5-pro). The model achieved state-of-the-art (SOTA) performance in both Chinese and English on the authoritative benchmark MTEB, and also achieved excellent results on the BRIGHT leaderboard for inference-intensive retrieval tasks. The model employs a Siamese dual-tower structure, relying on a pre-trained LLM from Seed1.5, and strengthens its general representation capabilities through two-stage training. The first stage uses unsupervised data for pre-fine-tuning, transforming the generative model into an encoding model through contrastive learning; the second stage uses supervised and synthetic data for fine-tuning, mixing data from various tasks for multi-task optimization. Iterative hard negative example mining, pseudo-negative example filtering, and synthetic data optimization strategies improve data composition and quality, enhancing the model's performance in retrieval tasks. Seed1.5-Embedding supports multiple vector dimensions, including 2048, 1024, 512, and 256.

The main functions of Seed1.5-Embedding

• Text semantic encodingThis method encodes the semantics of the input text into a high-dimensional representation vector, increasing the vector similarity between related texts. This encoding method can support downstream tasks such as retrieval, classification, and clustering, and is widely used in search, recommendation, and content understanding scenarios.
• Search taskThis model quickly locates the most relevant information to a user's query from massive amounts of documents by calculating vector similarity. It excels in reasoning-intensive retrieval tasks and can understand complex query and document matching relationships.
• Multi-tasking optimizationIt supports a variety of tasks, including classification, clustering, pairwise classification, reordering, retrieval, and semantic text similarity (STS) tasks, and is suitable for a variety of application scenarios.
• Flexible vector dimension supportIt supports multiple vector dimensions (2048, 1024, 512, 256), allowing users to choose the appropriate dimension according to their specific needs. Even at lower dimensions, the model performance degradation is minimal, providing flexible options for storage and runtime efficiency.
• Reasoning ability optimizationBy constructing reasoning-intensive retrieval data, the model's reasoning ability in complex queries and document matching is optimized, enabling it to handle more complex semantic relationships and logical reasoning tasks.

The technical principle of Seed1.5-Embedding

• Model ArchitectureSeed1.5-Embedding employs a Siamese dual-tower vector model structure, where query and document vectors are matched using cosine similarity. The model relies on a Seed1.5 pre-trained LLM, replacing unidirectional attention with bidirectional attention to construct a small-scale MoE (Mixture of Experts) model. Parameters are shared between the query and document sides, ensuring high operational efficiency.
• Two-stage training process
  • Phase 1: Use unsupervised data for pre-fine-tuning, and transform the unidirectional attention generative model into a bidirectional attention encoding model through contrastive learning to fully model various text matching patterns.
  • Phase TwoFine-tuning is performed using supervised and synthetic data, and multi-task optimization is performed by mixing data from various tasks, allowing the model to learn the best representation pattern for each task.
• Data Engineering Strategy
  • Negative example miningDesign an iterative hard negative example mining strategy to mine hard negative examples based on the model's own preferences, thereby improving the model's ability to distinguish fine-grained relevance.
  • False negative filteringAutomatically filters text that is too similar to positive examples to avoid spurious negative examples from affecting learning.
  • Synthetic data: Construct data for general scenarios and inference-intensive scenarios to improve the model's performance in complex retrieval tasks.

Seed1.5-Embedding project address

• HuggingFace model library:https://huggingface.co/ByteDance-Seed/Seed1.5-Embedding

Application scenarios of Seed1.5-Embedding

• Information retrieval and semantic searchSeed1.5-Embedding can vectorize documents or web pages, supporting semantic-level search and significantly improving recall and precision. For example, in scenarios such as question-answering systems (QA), internal enterprise document retrieval, and customer service, the model can more accurately understand the user's query intent and quickly locate relevant documents.
• Text clustering and topic recognitionUsing text vectors, Seed1.5-Embedding can cluster massive amounts of documents and automatically identify different topics and classification information.
• Recommendation systemIn recommender systems, models can vectorize textual information such as user reviews and product descriptions, and by calculating similarity, they can retrieve similar products and users.
• Text classification and sentiment analysisSeed1.5-Embedding can generate vectors from text, which can then be input into downstream classification models to improve the performance of tasks such as text classification, sentiment analysis, and stance analysis. Compared to traditional TF-IDF features, the generated vectors can more accurately express the meaning and context of the text.
• Complex Query Understanding and ReasoningThe model excels in reasoning-intensive retrieval tasks, demonstrating a deep understanding of the underlying matching relationships between complex queries and documents. For example, in complex search tasks in fields such as biology, earth sciences, and programming, Seed1.5-Embedding provides more accurate retrieval results.