Qwen2.5-1M - An open-source language model from Alibaba's Tongyi Qianwen platform, supporting 1 million token contexts.

What is Qwen2.5-1M?

Qwen2.5-1M is an open-source model developed by the Alibaba Tongyi Qianwen team, supporting context lengths of up to 1 million tokens. The model includes two versions: Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M. It performs exceptionally well on long context tasks, outperforming the previous 128K version, especially when handling tasks exceeding 64K in length. The Qwen2.5-14B-Instruct-1M model outperforms Qwen2.5-Turbo and consistently surpasses GPT-4o-mini on multiple datasets. The Qwen2.5-1M series models perform comparably to their 128K version on short text tasks, ensuring that their fundamental capabilities are not compromised by the increased ability to process long sequences.

Main functions of Qwen2.5-1M

• Long context task processingThe Qwen2.5-1M series models can handle contexts of up to 1 million tokens, outperforming the previous 128K version. In a Passkey Retrieval task with a context length of 1 million tokens, the Qwen2.5-1M series models can accurately retrieve hidden information from documents up to 1M in length.
• performance advantagesThe Qwen2.5-14B-Instruct-1M model outperformed Qwen2.5-Turbo and consistently surpassed GPT-4o-mini on multiple datasets.
• Short sequence task processingThe Qwen2.5-1M series models perform comparably to their 128K version on short text tasks, ensuring that their basic capabilities are not compromised by the addition of long sequence processing capabilities.

Technical Principles of Qwen2.5-1M

• Long context trainingQwen2.5-1M employs a progressive length expansion method, extending the context length from 4K to 256K. The training process is divided into multiple stages:
  • Pre-training phaseStarting with 4K, gradually increasing to 256K, while using the Adjusted Base Frequency scheme to increase the RoPE base frequency from 10,000 to 10,000,000.
  • Supervision and fine-tuning phaseThe training is conducted in two phases. The first phase involves fine-tuning only short instructions (up to 32K in length), while the second phase involves training with a mix of short and long instructions (up to 256K).
  • reinforcement learning phaseIt can be trained on short texts (up to 8K in length) and even when trained on short texts, it can generalize human preference alignment performance well to long context tasks.
• Sparse attention mechanismTo accelerate the pre-filling stage, Qwen2.5-1M introduces a sparse attention optimization based on Minference and proposes a series of improvements:
  • Block prefillingThe input sequence is divided into blocks of length 32768, and each block is pre-filled, which significantly reduces the demand for video memory.
  • Integrated length extrapolation schemeIntegrating a DCA-based length extrapolation scheme into sparse attention mechanisms improves inference efficiency and accuracy for long sequence tasks.
  • Sparsity optimizationThis paper proposes a method to optimize sparsity configuration on sequences of length 1 million, thereby reducing the accuracy loss caused by sparse attention.
  • Other optimizations: Optimize operator efficiency and dynamic block pipeline parallelism to enhance the potential of the entire framework.
• Length extrapolationTo extend the model's context length from 256K to 1M, Qwen2.5-1M employs a length extrapolation technique. The Dual Chunk Attention (DCA) method remaps excessively large relative positions to smaller values, addressing the performance degradation issue in long context tasks.

Qwen2.5-1M project address

• Project official website:https://qwenlm.github.io/zh/blog/qwen2.5-1m/
• HuggingFace model library:https://huggingface.co/spaces/Qwen/Qwen2.5-1M-Demo
• Technical Papers:https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-1M/Qwen2_5_1M_Technical_Report.pdf

Application scenarios of Qwen2.5-1M

• In-depth understanding of long novelsIt can process 10 novels at once and conduct in-depth analysis and understanding of them.
• Reading multiple papersIt can process multiple academic papers simultaneously, helping researchers quickly obtain key information.
• Literary creationIt assists writers in creating novels, providing them with writing inspiration and creative content.
• Advertising copywritingHelps advertisers quickly generate compelling advertising copy.
• Teaching aidsIn the field of education, it serves as a teaching aid to help students understand complex concepts.
• Data AnalysisIn the research field, it helps analyze large amounts of data and literature.