LLaDA2.2-flash - InclusionAI open-source diffusion language model - AiBoss

What is LLaDA2.2-flash?

LLaDA2.2-flash is an open-source diffusion language model (dLLM) from InclusionAI, employing the MoE architecture with approximately 100 bytes of parameters and native support for 128K context. The model fits the answer through multi-round parallel denoising and incorporates Levenstein editing to enable the addition, deletion, and modification of sequences. The model achieves an average score of 53.83 on seven agent benchmarks, approaching the 55.74 score of the similarly sized autoregressive model Ling-2.6-flash, while maintaining a 1.64x advantage in decoding throughput.

Main functions of LLaDA2.2-flash

Diffusion-based parallel generationHigh-throughput text generation is achieved by processing multiple token positions simultaneously through block parallel decoding.
Levinstein EditorDuring the noise reduction process, it supports four editing operations: KEEP, SUBSTITUTE, DELETE, and INSERT, breaking through the limitation of fixed-length replacement.
Long context processingIt natively supports 128K context windows, adapting to complex scenarios such as long-term software engineering and tool call trajectories.
Agent task executionSupports environment feedback tasks that require continuous error correction, such as code repair, API interaction, and multi-round tool calls.
MoE High-Efficiency ReasoningIt adopts a hybrid expert architecture and controls the inference cost under long contexts through a block routing mechanism.

Technical Principles of LLaDA2.2-flash

diffusion language model paradigmUnlike autoregressive models that generate tokens one by one from left to right, LLaDA2.2 first identifies a batch of noise positions to be filled, and then gradually fits the complete sequence through multiple rounds of denoising. The collaborative advancement of multiple positions has the potential for repeated refinement.
Levinstein editing mechanismThe generated draft is aligned with the target sequence using the longest common subsequence (LCS), and four types of editing instructions are constructed: KEEP, SUBSTITUTE, DELETE, and INSERT. DELETE removes redundant tokens and shifts the subsequent sequence to the left, while INSERT creates a [MASK] before the current position to be filled later, enabling the model to correct non-equal length sequences.
L-EBPO reinforcement learningThis paper proposes a block-level policy optimization algorithm based on the lower bound of Levenstein's edit evidence. The outer layer optimizes the trajectory-level decision-making of multi-round agent interactions, while the inner layer is responsible for insertion and deletion within a block in a single generation. By restoring the edit trajectory, the gradient covers the structural decision-making. The environmental reward comes from the correctness of tool calls, the validity of output format, and the degree of task completion.
Block RoutingTo address the issue of expert union inflation in MoE during long context block diffusion, a token race is used to obtain block-level admission scores. The Top-48 experts are selected from 256 routing experts to form a candidate pool. Then, Top-k routing is performed on each token within the pool, so that the expert working set obtains a predictable O(C) upper bound.

How to use LLaDA2.2-flash

Hardware preparationLocal deployment of LLaDA2.2-flash requires at least four A100-80GB or equivalent GPUs supporting BF16 full-precision loading. If FP8 quantization is used or 128K long context scenes are processed, more video memory resources need to be reserved accordingly.
Model DownloadWhen downloading a model via the ModelScope SDK, execute...from modelscope import snapshot_download; model_dir = snapshot_download("inclusionAI/LLaDA2.2-flash")It can automatically download model files of approximately 206GB.
Git cloneIf the model is obtained using Git LFS, rungit lfs installInitialize large file support and execute.git clone https://modelscope.cn/models/inclusionAI/LLaDA2.2-flash.gitPull the complete model repository.
Model loadingWhen loading a model using Transformers, callAutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, device_map="auto")And be sure to settrust_remote_code=TrueEnable custom diffusion decoding logic.
Precision settingsAfter the model is loaded, executemodel = model.to(torch.bfloat16)Convert it to BF16 precision, then callmodel.eval()Switch the model to evaluation inference mode.
Tokenizer loadingSynchronously load the corresponding word segmenterAutoTokenizer.from_pretrained(model_path, trust_remote_code=True)This ensures that the word segmentation rules are fully matched with the model configuration.
Input ConstructionAfter constructing the user prompt, throughtokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True, tokenize=True, return_tensors="pt")Format the dialogue into the input tensors required by the model.
Parameter configuration: callmodel.generate()Configure key diffusion decoding parameters at the time:block_length=32Control the number of tokens for parallel denoising in each step.threshold=0.5Set the denoising confidence threshold.editing_threshold=0.0Turn off active editing.temperature=0.0Enable greedy decoding.gen_length=512Set the maximum generation length.
Result DecodingAfter generation, usetokenizer.decode(generated_tokens[0], skip_special_tokens=True)Decode the token sequence into readable text and output the final result.

The core advantages of LLaDA2.2-flash

High-throughput decodingThe average decoding throughput of BF16 is 1.64 times that of the Ling-2.6-flash autoregressive model of the same scale, and it can be further improved by 18.6% after FP8 quantization.
Structured self-correctionLevinstein's editing enabled the model to add, delete, and modify data, increasing the SWE-bench Verified solution rate from 35.8% to 44.4%, an absolute gain of 8.6 percentage points.
Original growth context: Scale from 8K to 128K through continuous pre-training, without extrapolation, to adapt to the long-term interaction needs of agents.
Open source and reproducibleIt is licensed under the Apache-2.0 license, and the model and training details are publicly available, supporting complete local reproduction.
MoE inference efficiencyThe block routing mechanism keeps the expert working set of each block within a fixed capacity, significantly reducing HBM traffic and communication costs.

LLaDA2.2-flash project address

GitHub repository：https://github.com/inclusionAI/LLaDA2.X
ModelScope address：https://modelscope.cn/models/inclusionAI/LLaDA2.2-flash

Comparison of LLaDA2.2-flash with similar competing products

Comparison Dimensions	LLaDA2.2-flash	Ling-2.6-flash
Architectural Paradigm	The diffusion language model (dLLM) combined with the MoE architecture is used to generate sequences through multiple rounds of parallel denoising.	The text is generated sequentially from left to right, token by token, using an autoregressive (AR) approach combined with the MoE architecture.
Sequence editing capabilities	It natively supports Levinstein editing and features four types of add, delete, and modify operations: KEEP, SUBSTITUTE, DELETE, and INSERT.	It lacks native sequence editing capabilities; post-generation corrections require external resampling or post-processing.
Reinforcement learning strategies	The L-EBPO algorithm is used to jointly optimize trajectory-level decisions and intra-block editing actions in the context of environmental feedback.	Alignment and optimization are performed using standard RLHF or traditional reinforcement learning frameworks.
Agent benchmark average	The average score across 7 Agent benchmark tests was 53.83.	The average score across the same 7 Agent benchmark tests was 55.74, 1.91 points higher.
Decoding throughput	The average decoding throughput of BF16 is 1.64 times that of Ling-2.6-flash, and it is further improved by 18.6% after FP8 quantization.	As an autoregressive baseline model, decoding throughput is limited by the token generation method.
Context window	It natively supports 128K context through continuous pre-training, without location extrapolation.	The specific training methods for contextual capabilities were not disclosed in detail in the publicly available technical reports.

Application scenarios of LLaDA2.2-flash

Software Engineering AgentThe model natively supports Levinstein's editing capabilities for adding, deleting, and modifying text, making it capable of handling complex programming tasks that require editing text of varying lengths, such as code repair, bug location, and function completion.
Multi-round tool callsIn API interaction and MCP protocol execution, the model can dynamically correct parameters and add or delete fields to adapt to complex Agent workflows that require continuous error correction.
Long document processingWith a native 128K context window, the model can efficiently perform long text analysis, report generation, and knowledge extraction without relying on positional extrapolation.
Scientific research and experimental reproductionAs an open-source MoE diffusion language model, LLaDA2.2-flash provides the academic community with a reproducible research foundation and alternative technical path beyond the autoregressive route.