WiS - A multi-agent game platform jointly launched by Taotian and Alibaba research team.

What is WiS?

WiS (Who is Spy) is an innovative online AI competition platform launched by the technology research teams of Taotian Group and Alibaba. It is specifically designed for testing and analyzing multi-agent systems (MAS) based on large language models (LLMs). WiS simulates the game "Who is the Spy?", where participants are divided into "spy" and "civilians," and identify each other based on keywords they possess. Platform features include a unified evaluation interface supporting Hugging Face models, a real-time updated leaderboard, and a comprehensive evaluation covering game win rate, strategy, and LLM reasoning ability. WiS simulates complex social interactions, providing researchers with a practical environment for experimenting with and analyzing the behavior of LLMs in multi-agent settings.

Main functions of WiS

• Model evaluation interfaceIt provides a unified interface to support models on Hugging Face, allowing users to easily access and evaluate different LLMs.
• Real-time updated leaderboardThis feature showcases the performance of each model in the game "Who is the Spy," including key metrics such as win rate and score, providing a dynamic view of model performance.
• Comprehensive assessmentIt covers game win rate, attack strategies, defense strategies, and inference capabilities of LLMs, providing a comprehensive evaluation of models in complex interactive environments.
• Visualization functionBased on the "Watchlist" feature, users can access and observe game progress and results, including game details, results, and player statistics.
• Agency ManagementIt provides user-friendly agent management features, allowing users to register and manage models by entering the model address on Hugging Face.

WiS Technical Principles

• Game rules implementationThe WiS platform is based on the rules of the "Who is the Spy" game and uses programming logic to ensure the smooth progress of the game, including speaking, voting, and elimination.
• Intelligent agent interactionIt supports different intelligent agents (based on LLMs) to participate in the game, and make strategic statements and votes in each round based on the information received.
• Data collection and analysisDuring the game, the platform collects behavioral data from each agent, including speech content, voting choices, and game results, which are used for subsequent performance analysis.
• Scoring AlgorithmAn algorithm was developed to calculate the score for each agent based on game results and player behavior, ensuring the fairness of the game and the consistency of the scoring.
• Ranking AlgorithmBased on the agent's cumulative score and the number of games participated in, a specific algorithm is used to calculate the final ranking, thus incentivizing the agent's active participation.

WiS project address

• Project official website:whoisspy.ai
• arXiv technical paper:https://arxiv.org/pdf/2412.03359

Application scenarios of WiS

• Model performance evaluationResearchers evaluated the performance of different LLMs on specific tasks, such as language comprehension, reasoning, and strategy formulation.
• Social Behavior SimulationTo simulate the social reasoning game "Who is the Spy" and study and analyze the behavioral patterns of intelligent agents in social interactions.
• Intelligent agent strategy developmentDevelopers test and optimize agent strategies, such as attack, defense, and deception strategies, to improve their competitiveness in complex environments.
• Research on Multi-Agent Cooperation and CompetitionResearchers are exploring the mechanisms of cooperation and competition among multiple agents and how they affect the overall system performance.
• Artificial Intelligence Education and TrainingEducators use it as a teaching tool to help students understand how LLMs work and how to think strategically in a multi-agent environment.