HistAgent - An AI-powered historical research assistant jointly launched by Princeton University and Fudan University

What is HistAgent?

HistAgent is an AI assistant system jointly developed by the Princeton AI Lab and the Department of History at Fudan University, designed specifically for historical research. It addresses challenges in historical research such as multimodal information processing, cross-linguistic analysis, and complex reasoning. HistAgent can process various historical materials, including manuscripts, images, audio, video, inscriptions, and text, supporting 29 ancient and modern languages, covering multiple historical periods from antiquity to the present day and content from different regions of the world. On HistBench, a specially designed benchmark for historical reasoning, HistAgent significantly outperforms general-purpose large language models and other AI agents.

HistBench is the world's first AI benchmark focusing on historical research capabilities, jointly developed by the Princeton University AI Lab and the Department of History at Fudan University. It fills a gap in AI testing within the humanities and promotes systematic testing and breakthroughs in AI capabilities within the field of history.

Main functions of HistAgent

• Multimodal data processingHistAgent can process various historical materials, including manuscripts, images, maps, audio, and video. It uses an OCR module to recognize documents such as manuscripts and inscriptions, supports reverse image search and artifact recognition, and can process audio materials such as historical speeches and interview transcripts.
• Multilingual supportHistAgent supports translation and processing of 29 languages, both ancient and modern, including classical and less common languages. It can translate the surface meaning of text and optimize the translation by incorporating context.
• Literature retrieval and document parsingIt supports multi-step web page search and parsing, and can retrieve academic websites and historical data. HistAgent can parse files in various formats such as PDF, DOCX, XLSX, and PPTX.
• Historical reasoning and information integrationHistAgent combines historical knowledge with reasoning to help researchers sort out clues, integrate information, and form academic judgments. Through a central scheduling module (Manager Agent), it intelligently coordinates various sub-modules, calls upon appropriate tools according to task requirements, integrates multimodal results, and ultimately outputs a complete answer that conforms to the standards of the historical discipline.
• Multi-agent collaborationHistAgent is a multi-agent collaborative system containing multiple sub-modules. It can simulate the process of historical research, break down complex tasks into different sub-tasks, and call the most appropriate tools according to the needs of each sub-task.

HistAgent's technical principles

• Multi-agent architectureHistAgent employs a multi-agent system design pattern. Complex tasks are broken down into multiple sub-tasks and assigned to different agents for processing. Each agent focuses on a specific task, such as image recognition, language translation, or document retrieval. In this way, HistAgent can efficiently process various types of historical data and integrate results from different modalities.
  • Task planning and executionThe user-input query is first broken down into multiple subtasks, each executed by a corresponding agent. The execution results are observed and verified; if the results are unsatisfactory or erroneous, the system will replan and adjust the tasks.
  • Multi-perspective analysis and collaborationThe multi-agent architecture supports analyzing problems from different perspectives, and each agent can independently handle problems in a specific domain, reducing the requirements for memory and prompt length.
• Multimodal processing technologyHistAgent can process historical data in multiple modalities, including text, images, audio, and video. The core of multimodal processing technology lies in transforming information from different modalities into a unified semantic representation, facilitating further analysis and reasoning.
  • Visual processingImages and videos are processed using computer vision (CV) models (such as YOLOv8) to extract key information and transform it into structured descriptions, which are then injected into the context of a large language model.
  • speech processingAudio is converted into text using automatic speech recognition (ASR) technology (such as Whisper), then processed by a large language model, and finally output using text-to-speech (TTS) technology.
• Knowledge Enhancement and ReasoningTo improve the accuracy and reliability of reasoning, HistAgent employs knowledge augmentation techniques. By storing documents in a knowledge base in vector form (such as ChromaDB), relevant knowledge is dynamically retrieved and injected when processing user queries. This effectively suppresses the illusion problem of large language models and improves the credibility of the output results.
• Tool Invocation and ExtensionHistAgent supports dynamic invocation of external tools and plugins. Through tool invocation modules, HistAgent can call specific APIs or tools based on task requirements, such as document retrieval and document parsing. This improves system flexibility and allows developers to extend HistAgent's functionality by adding new plugins.
• Memory systemHistAgent's memory system employs a hybrid memory architecture, including short-term and long-term memory. Short-term memory stores contextual information for the current task, while long-term memory stores important historical information through a vector database (such as ChromaDB).

HistAgent project address

• Github repository:https://github.com/CharlesQ9/HistAgent
• arXiv technical paper:https://arxiv.org/pdf/2505.20246

Application scenarios of HistAgent

• Literature retrieval and analysisThrough multi-step web search and page parsing, it retrieves academic websites and historical materials, providing authoritative background information and supporting evidence.
• Image and cultural relic recognitionIt can perform reverse image search and cultural relic identification, and find the source and supplement the background of historical image materials.
• Historical reasoning and clue integrationIt combines historical knowledge to aid reasoning, helping researchers to sort out clues, integrate information, and form academic judgments.
• History teaching aidsIt provides teachers with abundant historical materials and case studies to assist in instructional design and improve teaching effectiveness.
• Cultural heritage protectionUsing image recognition and OCR technology, we help protect and study cultural heritage such as ancient books and inscriptions.

Features of HistBench

• High-quality question bankThe HistBench dataset contains 414 high-quality history questions written by historians, covering multiple levels from basic historical source reading to in-depth interdisciplinary analysis.
• Multilingual and multimodal coverageThe benchmark covers 29 ancient and modern languages and supports various types of historical materials such as manuscripts, images, audio and video, and historical artifacts, realistically simulating historical research scenarios.
• Difficulty levelsThe problems are divided into three difficulty levels, ranging from basic information retrieval to complex multimodal historical data processing and interdisciplinary analysis.
  • Level 1 (Basic):The 166 questions, designed by the history background assistant, focus on basic information retrieval and extraction.
  • Level 2 (Advanced):Question 172, written by graduate students, requires a certain level of difficulty in material processing or logical reasoning.
  • Level 3 (Challenge):The 76 questions, designed by senior scholars, cover topics such as reading of small/dead languages, processing of multimodal historical materials, and interdisciplinary analysis.
• Broad historical fields coveredIt covers more than 20 historical regions and 36 sub-fields, including classical studies, global history, new cultural history, art history, environmental history, and the history of science, technology and medicine.