Qwen3.8-27B-DFlash2 - Inco AI's open-source speculative decoding draft model

What is Qwen3.8-27B-DFlash2?

Qwen3.8-27B-DFlash2 is an open-source speculative token decoding draft model from Inco AI, specifically designed for Qwen3.8-27B. With only 1.92B parameters, the model generates entire token drafts in parallel during a single forward propagation through block-level parallel prediction, a lightweight path selector, and dual-tap dynamic convolution. On a single H200 card and with single-concurrency SGLang, the model achieves 2.7–3.4 times the throughput of autoregressive decoding, with an average acceptance length of 4.80, outperforming native MTP and the community-developed DSpark, while maintaining lossless decoding results.

Main functions of Qwen3.8-27B-DFlash2

• Parallel draft generationAs a speculative decoding draft model for Qwen3.8-27B, it can predict the entire token in parallel with a single forward propagation, replacing the traditional token-by-token autoregressive draft generation.
• Intelligent route selectionThe lightweight path selector traces a coherent path among the top-16 candidates at each position, thus solving the problem of unnatural token connections caused by parallel prediction.
• Local dependency modelingUsing dual-tap dynamic convolution to specifically build local dependencies between adjacent positions within a module significantly alleviates the "suffix decay" phenomenon at the end of the block.
• Lossless reasoning accelerationAchieve 2.7–3.4 times the throughput of autoregressive decoding on mainstream engines such as SGLang, vLLM, and llama.cpp, while strictly guaranteeing lossless decoding output.

Technical Principles of Qwen3.8-27B-DFlash2

• Block-level parallel draft generationTraditional speculative decoding draft models still require autoregressive generation of tokens one by one. DFlash changes the drafting process to non-autoregressive, predicting tokens at all positions within the block in parallel through a single forward propagation.
• Lightweight Path SelectorSince the top-1 candidates predicted independently at each position may not be coherent with each other, the selector retains the top-16 candidates at each position and scores all adjacent candidate pairs in parallel (based on low-rank bilinear attention and context gating). By greedily or sampling, it traces the best successor path starting from the last verified token. The acceptance length can be increased by about 0.4 tokens by only adding 2M parameters.
• Dual-tap dynamic convolutionObserving that the input of deep attention to block dependencies continued to decay, the team inserted content-adaptive dual-tap dynamic deep convolutions before and after the Attention and MLP sub-layers. Each position mixed its own representation with the predecessor representation, and used extremely low cost (+3% parameters, +0.7% latency) to undertake local modeling within the block, so that attention returned to context reading, with the effect close to adding ten Transformer layers.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Qwen3.8-27B-DFlash2

• Model preparationDownload the Qwen3.8-27B target model and the Qwen3.8-27B-DFlash2 draft model to your local machine using ModelScope.
• SGLang DeploymentAfter installing sglang, execute... sglang.launch_server, specify --speculative-algorithm DFLASH Simply specify the path to the draft model to start the acceleration service.
• vLLM DeploymentUse after installing the corresponding version of vLLM vllm serve,exist --speculative-config The method is set as dflash Then enter the name of the draft model.
• Deploying llama.cppPull and compile the PR branch that supports DFlash, and load the GGUF files of the target model and the draft model respectively. --spec-type draft-dflash Start the server.
• Apple SiliconUse the oMLX pre-compiled package to enable DFlash for the target model in Model Manager, specify the draft model, and set the Verify mode to dflash.
• command line tools:Install dflash After packaging, you can directly access... dflash generate Commands can be used with transformers, MLX, or OpenAI compatible backends to run inference and evaluation.

Qwen3.8-27B-DFlash2's core advantages

• extremely small volumeA draft model with only 1.92B parameters (approximately 3.85GB) can provide a significant acceleration for the 27B target model, with extremely low deployment costs.
• Parallel DraftsBreaking through the limitations of traditional autoregressive drafts, it generates entire token blocks in parallel during a single forward propagation, significantly reducing delays in the draft stage.
• Lossless accelerationAchieve 2.7–3.4 times the throughput of autoregressive decoding on a single H200 card, and ensure completely lossless output results through rigorous verification.
• Precise selectionThe lightweight path selector, with only 2M parameters and 0.6% latency overhead, tracks coherent paths from the top-16 candidates, significantly improving draft acceptance rates.
• Local modelingThe dual-tap dynamic convolution uses 3% of the parameters to specifically build intra-module dependencies, effectively mitigating the "suffix decay" at deep locations, achieving results comparable to adding ten Transformer layers.
• Eco-compatibleIt natively integrates mainstream inference engines such as SGLang, vLLM, llama.cpp, and oMLX, and supports deployment on multiple platforms including NVIDIA and Apple Silicon.

Qwen3.8-27B-DFlash2 project address

• Project official website：https://inco.ai/blog/dflash2/
• GitHub repository：https://github.com/z-lab/dflash

Comparison of Qwen3.8-27B-DFlash2 with similar competing products

Application scenarios of Qwen3.8-27B-DFlash2

• AI Agent ServiceSignificantly reduces the cost of token-by-token decoding during long-running Agent operations, supporting high-throughput execution processes such as tool invocation and task planning.
• Real-time dialogue systemSingle-threaded concurrency is 2.7–3.4 times faster, significantly reducing response latency and improving the interactive experience of chatbots and customer service assistants.
• Code generation assistantIt boasts leading acceptance length on HumanEval and MBPP benchmarks, making it suitable for scenarios requiring rapid completion, such as IDE plugins and programming Copilot.
• Long text creationThe longer acceptance length allows more tokens to be generated with each verification, accelerating the generation of long documents such as articles, reports, and marketing copy.
• Local deployment on the device sideIts small 1.92B size, combined with oMLX and llama.cpp support, enables efficient local inference on devices such as Apple Silicon laptops.