AB
AB
AiBoss

All Categories

project

FastMTP - Tencent's open-source technology for accelerating large language model inference.

FastMTP is Tencent's self-developed Large Language Model (LLM) inference acceleration technology. It optimizes Multi-Label Prediction (MTP) technology, replacing traditional multiple independent modules with a single MTP head sharing weights, and combines language-aware lexical compression and self-regulation...

What is FastMTP?

FastMTP is Tencent's self-developed Large Language Model (LLM) inference acceleration technology. By optimizing Multi-Label Prediction (MTP) technology and replacing traditional multiple independent modules with a single MTP head sharing weights, combined with language-aware lexical compression and self-distillation training, it significantly improves the inference speed of LLM, with an average speedup of up to 2.03 times, while maintaining lossless output quality. FastMTP does not change the main model structure, is easy to integrate into existing frameworks, and is suitable for structured tasks such as mathematical reasoning and code generation, providing a practical solution for the efficient deployment of LLM.

Main functions of FastMTP

  • Significantly accelerates LLM inferenceBy optimizing multi-label prediction (MTP) technology, FastMTP can improve the inference speed of large language models (LLM) by an average of 2.03 times without sacrificing output quality, significantly shortening the time for model to generate content and improving application response speed.
  • Maintain lossless output qualityWhile accelerating inference, FastMTP ensures that the quality of the model output is completely consistent with that of traditional autoregressive generation methods, and the accuracy or logic of the generated content will not be reduced due to acceleration.
  • Easy to integrate and deployFastMTP requires no changes to the main model structure; only a small module needs to be fine-tuned. It can be seamlessly integrated with existing LLM inference frameworks (such as SGLang), greatly reducing deployment costs and difficulties, and facilitating rapid application to real-world scenarios.
  • Reduce hardware resource consumptionThe single MTP head based on shared weights replaces multiple independent modules, significantly reducing memory usage. At the same time, language-aware lexical compression further reduces computational load, enabling LLM to run more efficiently on consumer-grade GPUs and reducing hardware resource requirements.

FastMTP Technical Principles

  • Speculative DecodingBy adopting the "draft + verification" strategy, multiple candidate tags are generated from a fast draft model, and batch verification is performed using the main model to achieve parallel processing and improve inference efficiency.
  • Shared weighted single MTP headerInstead of the traditional multi-independent module design of MTP, it uses a shared weight MTP head to recursively generate multiple tags, reducing memory usage, forcing the model to learn longer-distance dependencies, and improving draft quality.
  • Self-distillation trainingThe MTP head is trained using data generated by the main model. By using an exponentially decaying weighted cross-entropy loss function, the MTP head is allowed to prioritize the generation of drafts that are consistent with the style and logic of the main model, thereby improving the acceptance rate of the drafts.
  • Language perception lexical compressionDuring the draft generation stage, the language is determined based on the input context, and only the logits of high-frequency words are calculated to reduce the amount of computation. During the verification stage, the full vocabulary is used to ensure that the output quality is not affected.

FastMTP project address

  • GitHub repositoryhttps://github.com/Tencent-BAC/FastMTP
  • HuggingFace model libraryhttps://huggingface.co/TencentBAC/FastMTP
  • Technical Papers: https://github.com/Tencent-BAC/FastMTP/blob/main/FastMTP_technical_report.pdf

Application scenarios of FastMTP

  • Mathematical reasoningIn mathematical problem-solving scenarios, it can quickly generate solution steps, significantly shorten the time from problem input to answer output, and improve the response speed of mathematical aids.
  • Code generationFor programming assistance applications, FastMTP can quickly generate code snippets, helping developers complete code writing faster and improving development efficiency.
  • Long text summaryWhen processing long text summarization tasks such as news and articles, it can quickly extract key information and generate high-quality summary content, saving users' reading time.
  • Multiple rounds of dialogueIn intelligent customer service or chatbots, speeding up dialogue generation and achieving millisecond-level response improves user experience and interaction fluency.
Back