Dream-7B - A diffusion inference model open-sourced by the University of Hong Kong and Huawei Noah's Ark.

What is Dream-7B?

Dream-7B is a diffusion-based inference model jointly developed by the University of Hong Kong and Huawei Noah's Ark Lab. It is currently the most powerful open-source diffusion-based large language model. Dream-7B's training data covers text, mathematics, and code, with pre-training using 580 billion tags and taking 256 hours. Dream-7B performs excellently on general, mathematical, and programming tasks, comparable to similarly sized autoregressive models (Qwen2.5 7B, LLaMA3 8B), and in some cases outperforms the latest Deepseek V3 671B. The model uses a masked diffusion paradigm, based on bidirectional contextual modeling and flexible, controllable generation capabilities, significantly improving the global coherence of generated text.

Main functions of Dream-7B

• Powerful text generation capabilitiesIt outperforms autoregressive models of the same size on general text, mathematical, and programming tasks.
• Flexible generation methodsIt supports text generation in any order, allowing users to specify the generation order according to their needs.
• Highly efficient planning capabilitiesIt performs exceptionally well in tasks requiring multi-step planning, such as Countdown and Sudoku.
• Adjustable production qualitySupports users to adjust the number of diffusion steps to balance generation speed and quality.

The technical principles of Dream-7B

• Diffusion Model ArchitectureBased on Discrete Diffusion Models (DMs), unlike traditional autoregressive models, diffusion models start from a completely noisy state and gradually denoise to generate text. The diffusion model architecture supports bidirectional contextual modeling, integrating forward and backward information, significantly improving the global coherence of the generated text.
• Mask diffusion paradigmThe model uses a masked diffusion paradigm to progressively denoise all masked labels. This supports better alignment of the autoregressive model's weights during training, accelerating the training process.
• Autoregressive model initializationUsing weights from an autoregressive model (such as Qwen2.5) as initialization is more effective than training a diffusion model from scratch, thus accelerating the training of the diffusion model.
• Context-adaptive noise reschedulingA context-adaptive noise rescheduling mechanism is introduced to dynamically adjust the noise level based on the context information of each label. This allows the model to more finely control the learning process for each label, improving training efficiency.
• Flexible decoding strategyDuring the inference phase, the diffusion model flexibly adjusts the generation order and the number of diffusion steps to achieve a dynamic balance between speed and quality.

Dream-7B's project address

• Project official website:https://hkunlp.github.io/blog/2025/dream/
• GitHub repository:https://github.com/HKUNLP/Dream
• HuggingFace model library:https://huggingface.co/Dream-org
• Experience the demo online:https://huggingface.co/spaces/multimodalart/Dream

Application scenarios of Dream-7B

• Text generation and creationCreate high-quality general texts, such as news reports, story writing, and copywriting, providing rich and coherent text content.
• Solving mathematical problemsIt efficiently solves complex mathematical problems, including the derivation of mathematical problems and the generation of formulas, providing auxiliary tools for education and scientific research.
• Programming aidsGenerates programming code to help developers quickly build code frameworks, solve programming problems, and improve programming efficiency.
• Complex task planningIt is used in scenarios that require multiple constraints and multi-step reasoning, such as task scheduling and path planning.
• Flexible text processingIt can adjust the generation speed and quality according to needs, making it suitable for various applications that require flexible text processing.