DeepEyes - A multimodal deep thinking model launched by Xiaohongshu in collaboration with Xi'an Jiaotong University.

What is DeepEyes?

DeepEyes is a multimodal deep thinking model jointly developed by the Xiaohongshu team and Xi'an Jiaotong University. Based on end-to-end reinforcement learning, it achieves "thinking with images" capabilities similar to OpenAI o3, without relying on supervised fine-tuning (SFT). DeepEyes dynamically invokes image tools during inference, such as cropping and scaling, to enhance the perception and understanding of details. The model achieves an accuracy of up to 90.1% on the visual inference benchmark V* Bench, demonstrating powerful visual search and multimodal inference capabilities. DeepEyes possesses excellent image localization capabilities, effectively reducing illusions and improving the model's reliability and generalization ability.

DeepEyes' main functions

• Think with diagramsIt directly integrates images into the reasoning process, allowing users to not only "look at pictures" but also "think with pictures," dynamically calling upon image information during the reasoning process to enhance the perception and understanding of details.
• Visual searchIt can quickly locate small objects or blurry areas in high-resolution images, and perform detailed analysis based on cropping and scaling tools, significantly improving search accuracy.
• Hallucination reliefBy focusing on image details, the model reduces the illusion phenomenon that may occur when generating answers, thereby improving the accuracy and reliability of the answers.
• Multimodal reasoningIt achieves seamless integration between visual and textual reasoning, enhancing the model's reasoning capabilities in complex tasks.
• Dynamic tool invocationThe model can autonomously decide when to invoke image tools, such as cropping and scaling, without the need for external tools, thus achieving more efficient and accurate inference.

DeepEyes' technical principles

• End-to-end reinforcement learningDeepEyes trains its models using end-to-end reinforcement learning (RL) without requiring supervised cold-start fine-tuning (SFT). It directly optimizes the model's behavior based on reward signals, autonomously learning how to effectively utilize image information during inference. The reward function includes accuracy rewards, format rewards, and conditional tool rewards, ensuring the model efficiently uses image tools while correctly answering questions.
• Interleaved Multimodal Thinking Chain (iMCoT)DeepEyes introduces Interleaved Multimodal Chain-of-Thought (iMCoT), which allows the model to dynamically alternate between visual and textual information during inference. At each step of inference, the model decides whether further visual information is needed, cropping key regions from the image based on the generated bounding box coordinates and re-inputting these regions as new visual evidence.
• Tool-guided data selectionTo better incentivize the model's tool-using behavior, a tool-use-oriented data selection mechanism is employed. Training data is carefully selected to ensure that samples effectively promote the model's tool-calling capabilities. The dataset includes high-resolution images, graphs, and inference data, covering various task types to enhance the model's generalization ability.
• Dynamic tool invocation behaviorDuring training, the model's tool-calling behavior goes through three stages: initial exploration, active use, and efficient utilization. The model gradually evolves from initial random attempts to efficient and accurate tool-calling, ultimately achieving a visual reasoning process similar to that of humans.
• Multimodal fusionDeepEyes is based on the deep fusion of visual and textual information to build a unified inference framework. This fusion improves the model's performance in visual tasks and enhances its overall performance in multimodal tasks.

DeepEyes project address

• Project official website:https://visual-agent.github.io/
• GitHub repository:https://github.com/Visual-Agent/DeepEyes
• HuggingFace model library:https://huggingface.co/ChenShawn/DeepEyes
• arXiv technical paper:https://arxiv.org/pdf/2505.14362

Application scenarios of DeepEyes

• Educational guidanceThis tool analyzes the charts and geometric figures in the exam paper, providing students with detailed problem-solving steps to improve learning efficiency.
• Medical imagingIt analyzes medical images to assist doctors in making diagnoses, improving the accuracy and efficiency of diagnosis.
• Intelligent TransportationReal-time analysis of road condition images helps autonomous driving systems make more accurate decisions and improve traffic safety.
• Security monitoringAnalyze surveillance video to identify abnormal behavior and enhance public safety and crime prevention capabilities.
• Industrial manufacturingOn-line quality inspection and equipment failure prediction improve production efficiency and reduce maintenance costs.