Collaborative Gym - An evaluation framework that supports real-time interactive collaboration between humans and AI agents.

What is Collaborative Gym?

Collaborative Gym (Co-Gym) is a framework focused on human-agent collaboration, supporting real-time interaction and collaboration between humans and AI agents. Through both simulated and real-world experimental conditions, it allows developers to iterate in controlled environments while simultaneously deploying and evaluating collaboration effectiveness in real-world scenarios. Co-Gym supports asynchronous interaction and can handle three-way interactions between agents, humans, and the task environment.

Main functions of Collaborative Gym

• Support asynchronous interactionCo-Gym breaks through the limitations of synchronous action in traditional multi-agent frameworks, enabling humans and AI agents to flexibly initiate actions in collaboration without strictly adhering to a sequential interaction pattern. This more closely resembles a realistic human collaboration model.
• Task Environment DesignCo-Gym defines each task as a partially observable Markov decision process (POMDP), supporting both public and private observation spaces. The public part is visible to all participants, while the private part is visible only to the owner, similar to the difference between a shared whiteboard and individual notes in a human team.
• Evaluation FrameworkCo-Gym focuses on the outcome of task completion while emphasizing the collaboration process itself. It provides a comprehensive evaluation framework that assesses agents from two dimensions: collaboration quality and collaboration process.
• Simulated vs. Real ConditionsCo-Gym supports both simulated and real-world experimental conditions. Simulated conditions use pre-collected task instances and simulated human behavior, facilitating rapid iterative development; real-world conditions support collaboration with real humans in actual task environments, supporting various tasks such as travel planning, literature reviews, and tabular analysis.

The technical principles of Collaborative Gym

• Collaboration-driven environment designCo-Gym draws inspiration from the design of OpenAI Gym, optimizing it for human-computer collaboration. It supports defining public and private sections in the observation space: public sections are visible to all participants, while private sections are only visible to the owner. It simulates the difference between shared information and personal notes in real-world collaborative scenarios.
• Asynchronous interactionCo-Gym supports asynchronous interaction, allowing humans and agents to act independently when necessary without waiting for a response from the other. To this end, Co-Gym introduces two collaborative behaviors: SendTeammateMessage and WaitTeammateContinue, using a notification protocol to inform participants of changes in the environment in real time.
• Notification AgreementCo-Gym implements a notification protocol through a Redis server, supporting four event types: shared observation updates, private observation changes, new message notifications, and environment inactivity timeout notifications. This allows agents to monitor environment changes in real time and collaborate better with humans.
• Task Environment Interface (CoEnv)Co-Gym provides a flexible task environment interface, allowing developers to easily add new task environments. Developers only need to define the task description, action space, and observation space. Co-Gym supports various task environments, such as travel planning, literature review, and tabular data analysis.

Collaborative Gym project address

• Github repository:https://github.com/SALT-NLP/collaborative-gym
• arXiv technical paper:https://arxiv.org/pdf/2412.15701

Application scenarios of Collaborative Gym

• Travel PlanningIn travel planning tasks, Co-Gym enables collaboration between humans and AI agents to create detailed travel itineraries. The agent can leverage its search and planning capabilities, while humans can provide preferences and expertise to jointly complete the travel plan.
• Tabular AnalysisCo-Gym provides agents and humans with a shared workspace and real-time communication capabilities, supporting effective collaborative analytics.
• Literature Review (Related Work)In literature review tasks, Co-Gym supports collaboration between humans and AI agents to organize and analyze academic literature. The agent can quickly retrieve and filter relevant literature, while humans can provide domain expertise to jointly produce a high-quality literature review.