TokenSwift - A framework for accelerating the generation of ultra-long text, generating 100,000 token texts within 90 minutes.
TokenSwift is an ultra-long text generation acceleration framework developed by the Beijing General Artificial Intelligence Research Institute team. It can generate text containing 100,000 tokens within 90 minutes, a 3x speed improvement compared to the nearly 5 hours required by traditional autoregressive models.
What is TokenSwift?
TokenSwift is an ultra-long text generation acceleration framework developed by the Beijing General Artificial Intelligence Research Institute team. It can generate text containing 100,000 tokens within 90 minutes, a 3x speed improvement compared to the nearly 5 hours required by traditional autoregressive models, while maintaining lossless generation quality. TokenSwift reduces model loading latency, optimizes cache update time, and ensures generation diversity through technologies such as multi-token generation and token reuse, dynamic KV cache updates, and context penalty mechanisms. It supports models of various sizes and architectures, such as 1.5B, 7B, 8B, and 14B MHA and GQA models.
TokenSwift's main features
- Accelerating the generation of ultra-long textsTokenSwift can significantly reduce the generation time of extremely long texts. For example, a traditional autoregressive model may take nearly 5 hours to generate text containing 100,000 tokens, while TokenSwift only takes 90 minutes, a 3x speed improvement, greatly enhancing generation efficiency.
- Lossless accelerationWhile accelerating the process, TokenSwift maintains the output quality of the original model, ensuring the quality and diversity of the generated text.
- Supports multiple modelsTokenSwift supports a variety of models of different sizes and architectures, including 1.5B, 7B, 8B, and 14B models, as well as multi-head attention (MHA) and grouped query attention (GQA) architectures.
TokenSwift's technical principles
- Parallel generation of multiple tokens and token reuseTokenSwift borrows from methods such as Medusa, introducing an additional linear layer that allows the model to generate multiple draft tokens simultaneously in a single forward propagation. Based on the n-gram frequency information in the generated text, the system automatically retrieves and reuses high-frequency phrases, further reducing the number of model reloads and improving overall efficiency.
- Dynamic KV Cache Update StrategyIn key-value (KV) cache management, TokenSwift employs a dynamic update strategy. The system retains the initial KV cache during the generation process and then replaces subsequent caches in an orderly manner based on the importance of the tokens.
- Tree-based multi-candidate token verificationTo ensure consistency between the generated results and the target model's predictions, TokenSwift introduces a tree-structured attention mechanism. By constructing a tree structure containing multiple candidate token combinations and employing parallel verification, the longest and most effective n-gram is randomly selected as the final output, ensuring a lossless generation process while enhancing diversity.
- Contextual penalty strategyTo further mitigate the problem of duplicate generation, TokenSwift employs a context penalty method. During the generation process, a penalty is applied to recently generated tokens, causing the model to favor more diverse outputs when selecting the next token, effectively reducing duplication.
TokenSwift's project address
- Project official website:https://bigai-nlco.github.io/TokenSwift/
- Github repository:https://github.com/bigai-nlco/TokenSwift
- HuggingFace model library:https://huggingface.co/TokenSwift
- arXiv technical paper:https://arxiv.org/pdf/2502.18890
TokenSwift Application Scenarios
- Content creation and copywritingTokenSwift can generate high-quality text content in a short time. This can significantly improve the efficiency of content creators, advertising agencies, and media organizations.
- Intelligent customer service and chatbotsIn intelligent customer service and chatbots, TokenSwift can quickly generate detailed answers, providing a more efficient and coherent conversational experience when dealing with complex questions or lengthy explanations.
- Academic research and paper writingFor academic research and paper writing, TokenSwift can help researchers quickly generate literature reviews, experimental reports, or paper drafts, saving a lot of time and effort.
- Code generation and programming assistanceIn the programming field, TokenSwift can be used to generate complex code snippets or documentation, helping developers quickly build and optimize code.