project
AgentReview - A framework for simulating the peer review process based on LLM agents.
AgentReview is a framework based on Large Language Models (LLM) that simulates the academic peer review process. Using LLM proxies, AgentReview simulates the roles of reviewers, authors, and area chairs, supporting researchers in respecting privacy...
What is AgentReview?
AgentReview is a framework based on Large Language Models (LLM) that simulates the academic peer review process. Using LLM agents, AgentReview simulates the roles of reviewers, authors, and area chairs, supporting researchers in exploring the impact of review biases, roles, and decision-making mechanisms on review outcomes while respecting privacy. AgentReview provides insights into improving peer review mechanisms and supports future research.
Main functions of AgentReview
- Simulated peer review processAgentReview simulates the real academic peer review process, including stages such as reviewer evaluation, author response, reviewer discussion, and area chair decision-making.
- Role SimulationThe framework integrates three roles: reviewer, author, and area chair (AC). Each role is driven by an LLM agent and exhibits different behavioral characteristics.
- Multivariate analysisBased on simulation, AgentReview explores and separates various variables that influence review outcomes, such as reviewers’ commitment, intent, and knowledge and competence, as well as the decision-making style of the AC.
- Privacy protectionDuring the simulation, AgentReview respects the privacy of the review data and does not require the use of real, sensitive review data.
- Validation of sociological theoriesAgentReview validates the application of sociological theories such as social influence theory, altruism fatigue, groupthink, and authority bias in peer review.
The technical principles of AgentReview
- Large Language Models (LLM)AgentReview is built on an LLM architecture and uses language understanding and generation capabilities to simulate the behavior of reviewers and authors.
- Proxy modelingEach role in the framework (reviewer, author, AC) is modeled as an agent with specific attributes and behaviors, and the agent interacts according to preset characteristics and rules.
- Structured review processAgentReview follows a structured five-stage review process, simulating the entire process from initial review to final decision.
- Customization and extensibilityThe framework is designed to be scalable, allowing researchers to customize role attributes and review processes as needed.
- Data-driven insightsBased on data generated through large-scale simulations, AgentReview provides statistically significant insights, supporting both content and numerical analysis.
AgentReview's project address
- Project official website:agentreview.github.io
- GitHub repository:https://github.com/Ahren09/AgentReview
- HuggingFace model library:https://huggingface.co/spaces/Ahren09/AgentReview
- arXiv technical paper:https://arxiv.org/pdf/2406.12708
- Experience the demo online:https://huggingface.co/spaces/Ahren09/AgentReview
Application scenarios of AgentReview
- Academic Journals and Conference ManagementIt is used to optimize and manage the peer review process for academic papers, thereby improving the quality and efficiency of the review.
- Reviewer Training and DevelopmentAs an educational tool, it helps new reviewers learn review standards and best practices.
- Sociological and psychological research: Based on simulation verification, the application of theories such as social influence and groupthink in review behavior.
- Interdisciplinary review researchThis study compares the review standards and processes across different disciplines to provide a basis for the design of interdisciplinary journals.
- Policy formulation and evaluation: To assist policymakers in evaluating and developing policies and guidelines related to peer review.