LLaDA-V - A multimodal large-scale model jointly launched by Renmin University Hillhouse AI and Ant Financial.
LLaDA-V is a multimodal large language model (MLLM) developed by the Hillhouse Institute of Artificial Intelligence at Renmin University of China and Ant Group. Based on a pure diffusion model architecture, it focuses on fine-tuning visual instructions. The model builds upon LLaDA by introducing visual...
What is LLaDA-V?
LLaDA-V is a multimodal large language model (MLLM) developed by the Gaoling School of Artificial Intelligence at Renmin University of China and Ant Group. Based on a pure diffusion model architecture, it focuses on fine-tuning visual instructions. Building upon LLaDA, the model introduces a visual encoder and an MLP connector to map visual features to the language embedding space, achieving effective multimodal alignment. LLaDA-V achieves state-of-the-art performance in multimodal understanding, surpassing existing hybrid autoregressive-diffusion and pure diffusion models.
Main functions of LLaDA-V
- Image description generationGenerate detailed descriptive text based on the input image.
- Visual Q&AAnswer questions related to the content of the image.
- Multi-turn multimodal dialogue: To conduct multi-turn dialogues within the context of a given image and understand how to generate responses that are relevant to the image and the dialogue history.
- Complex reasoning tasks:Reasoning in complex tasks involving images and text, such as solving image-related mathematical or logical problems.
LLaDA-V Technical Principles
- Diffusion ModelsThe diffusion model is based on generating data by progressively removing noise. In LLaDA-V, masked diffusion models are used to randomly mask (replace with special markers [M]) some words in a sentence, and the model is trained to predict the original content of the masked words.
- Visual Instruction TuningThe framework is based on visual instruction fine-tuning and consists of a Vision Tower and an MLP Connector. The Vision Tower uses a SigLIP 2 model to convert images into visual representations. The MLP Connector maps the visual representations to the word embedding space of a language model, allowing visual and linguistic features to be effectively aligned and fused.
- Multi-stage training strategyIn the first stage, the MLP connectors are trained to align visual representations and language embeddings. In the second stage, the entire model is fine-tuned to understand and follow visual instructions. In the third stage, the model's multimodal reasoning capabilities are further enhanced, training it to handle complex multimodal reasoning tasks.
- Bidirectional attention mechanismIn multi-turn dialogues, LLaDA-V uses a bidirectional attention mechanism to support the model in considering the entire dialogue context when predicting mask words, which helps the model better understand the overall logic and content of the dialogue.
LLaDA-V project address
- Project official website:https://ml-gsai.github.io/LLaDA-V
- GitHub repository:https://github.com/ML-GSAI/LLaDA-V
- arXiv technical paper:https://arxiv.org/pdf/2505.16933
Application scenarios of LLaDA-V
- Image description generationAutomatically generates detailed descriptions of the image to help understand its content.
- Visual Q&AAnswering image-related questions is applicable to fields such as education and tourism.
- Multiple rounds of dialogue: Conduct multi-round, multimodal dialogues in scenarios such as intelligent customer service and virtual assistants.
- Complex ReasoningSolve complex reasoning tasks involving images and text, such as solving mathematical problems.
- Multi-image and video understandingIt analyzes multiple images and video content, and is suitable for scenarios such as video analysis and monitoring.