AngelSpec - An end-to-end inference decoding training framework open-sourced by Tencent's Hunyuan team.

What is AngelSpec?

AngelSpec is an open-source end-to-end speculative decoding training framework developed by Tencent's Hunyuan team. Built on TorchSpec, it supports six draft architectures, including MTP autoregressive and DFly block parallelism. Inference and training are decoupled; the inference engine streams hidden states to the training workers via Mooncake/RDMA. On Hy3-A21B, DFly achieves 1.98–2.40x end-to-end speedup, with throughput improved by 10.5–11.8% compared to DFlash.

AngelSpec's main functions

• Unified training on six draft frameworksDFly, DFlash, DFLare, Eagle3, DSpark, and MTP share a single training pipeline; switching between them only requires configuration changes.
• MTP Training and TTT DeploymentIt supports training at multiple depths of autoregression with memory usage close to that of a single causal forward pass, and supports up to 128k context through Ulysses sequence parallelism.
• Acceptance rate aligned objective functionSupports CE, top-k KL, LK Loss, D-PACE weighted and end-to-end TV loss, which can be freely combined through configuration.
• Document-aware sequence packagingMegatron-style fixed-length packaging with strict cross-document isolation, while also covering DFlash and MTP paths.
• Online Real AssessmentDuring training, realistic speculative decoding is performed using engines such as vLLM, and the average acceptance length and bit-by-bit acceptance rate are reported in real time.
• Multi-inference backend supportIt is compatible with mainstream inference engines such as vLLM, SGLang, and HuggingFace.

AngelSpec's technical principles

• Speculative Decoding BasicsAngelSpec is built on a speculative decoding mechanism: a lightweight draft model proposes multiple future tokens in parallel, and the target model performs batch verification through a single forward propagation using rejection sampling, extending single-step decoding to multi-step progression without changing the target distribution. Its core challenge lies in the trade-off between draft acceptance rate and verification overhead, and AngelSpec systematically addresses this issue from three levels: training, architecture, and inference.
• Workload heterogeneity modelingAngelSpec prioritizes the heterogeneity of real-world workloads in its design. Online requests span open-ended dialogues, code generation, mathematical reasoning, and tool calls, exhibiting significant differences in conditional entropy and continuation structures: in high-entropy dialogue scenarios, acceptance rates decay rapidly with depth, making them suitable for short-sequence autoregressive drafts; while code and mathematical scenarios have long, predictable spans, making them suitable for block-parallel diffusion drafts. Therefore, AngelSpec trains complementary dedicated drafters—MTP for dialogue data and DFly for code and mathematical data—rather than pursuing a single, general-purpose model.
• Shared parameter multi-depth MTP trainingThe MTP drafter employs a multi-depth training scheme with shared parameters. All logical prediction depths reuse the same physical MTP module, undergoing autoregressive expansion during training: depth k+1 receives the argmax prediction of depth k, while maintaining a progressively growing draft key-value cache and a diagonal attention mask. The Training-Time Test (TTT) mechanism exposes the drafter to the self-generated trajectory distribution encountered during inference, eliminating exposure bias caused by teacher coercion and significantly improving the acceptance rate of the second and third draft positions.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use AngelSpec

• Environmental installation:implement pip install -e ".[vllm]" and pip install mooncake-transfer-engine Install the framework and vLLM backend.
• Single-node fast startup8 GPUs are allocated as 4 for inference and 4 for training. ./examples/qwen3-8b-dfly/run.sh The entire production line can then be started.
• Configuration Coverage: Directly override YAML configuration items via CLI, such as training.learning_rate=5e-5 training.num_train_steps=500。
• Conda environment setup:run ./tools/build_conda.sh 1 vllm Create an isolated environment containing Mooncake with one click, and it's ready to use after activation.
• Multi-node deploymentCross-node decoupled training is achieved by scheduling the Inference Controller and Training Controller via Ray and combining it with Mooncake hidden state storage.

AngelSpec's core advantages

• Workload specializationMTP is optimized for high-entropy dialogue scenarios, while DFly is enhanced for predictable scenarios with code and mathematical lengths, avoiding the mediocre performance of a single draft model across all domains.
• DFly Architecture InnovationThe hybrid target condition backbone, combined with the layer-by-layer target view and the precursor condition autoregressive head, improves intra-block dependency modeling while retaining parallel generation capabilities.
• D-Cut Dynamic ValidationTreating target verification as a shared batch-level resource, the verification depth is adaptively adjusted based on prefix confidence and runtime cost model, continuously converting additional load into throughput under high concurrency.
• Decoupled independent expansionThe inference and training GPU pools are decoupled through Mooncake storage, allowing each to scale up or down independently according to its own load, thus avoiding the suboptimal problem of a unified parallel strategy.
• Open source complete kitIt simultaneously releases the framework code, Hy3-A21B/Qwen3-8B MTP and DFly draft weights, training configuration, and technical reports, enabling it to be used out of the box.

AngelSpec's project address

• Project official website：https://angelspec.readthedocs.io/
• GitHub repository：https://github.com/Tencent/AngelSpec
• HuggingFace model library：https://huggingface.co/collections/AngelSlim/angelspec
• arXiv technical paper：https://arxiv.org/pdf/2607.25852

AngelSpec's Competitive Comparison

Application scenarios of AngelSpec

• Accelerating Online Inference for Large ModelsDeploy speculative decoding drafts for mixed Hy3, Qwen3 and other MoE/dense models to reduce autoregressive decoding latency.
• High-concurrency dialogue serviceUtilizing D-Cut dynamic resource allocation, it maintains the highest average throughput within a concurrency range of 4–64, making it suitable for customer service and assistant scenarios.
• Code generation and mathematical reasoning: Accelerate structured output tasks such as IDE completion and problem-solving engines by capturing long, predictable spans through DFly block parallel drafts.
• Long Context Training and ReasoningUlysses sequence parallelism supports 128k context, adapting to long sequence scenarios such as document analysis and code repository retrieval.