GTA - Shanghai AI Lab, in collaboration with Shanghai Jiao Tong University, launched a benchmark test for evaluating general-purpose intelligent agents.

What is GTA?

GTA (a benchmark for General Tool Agents) is a benchmark jointly launched by Shanghai Jiao Tong University and the Shanghai AI Lab to evaluate the ability of large language models (LLMs) to invoke tools in real-world scenarios. GTA establishes a comprehensive and fine-grained evaluation framework based on providing real user questions, real-world deployed tools, and multimodal inputs and outputs, effectively measuring the tool usage capabilities of LLMs in complex scenarios. GTA includes 229 human-designed questions covering multiple categories such as perception, action, logic, and creativity, requiring models to reason about appropriate tools, plan operational steps, and solve complex real-world tasks.

GTA's main functions

• Real user queryIt contains 229 problems written by humans. The problems have simple real-world goals, but the solution steps and required tools are implicit. It requires LLMs to select appropriate tools and plan operation steps based on reasoning.
• Real-world deployment toolsGTA provides an assessment platform that deploys 14 tools across four categories—perception, operation, logic, and creativity—to evaluate the agent's actual task performance.
• Multimodal input/outputGTA introduces multimodal inputs such as spatial scenes, webpage screenshots, tables, code snippets, and handwritten/printed materials, requiring the model to process rich contextual information and provide text or image outputs.
• Fine-grained assessmentGTA designs fine-grained evaluation metrics, including instruction compliance accuracy (InstAcc), tool selection accuracy (ToolAcc), parameter prediction accuracy (ArgAcc), answer summary accuracy (SummAcc), and final answer accuracy (AnsAcc).
• Model EvaluationGTA evaluates language models in both step-by-step and end-to-end modes, providing a comprehensive assessment of the ability to use model tools.

GTA's technical principles

• Dataset ConstructionThe GTA dataset construction involves two steps: problem construction and toolchain construction. Problems are designed by experts using sample examples and annotation documents, and annotation personnel then design more problems based on these examples. The toolchain is built by annotation personnel manually calling pre-deployed tools to ensure that each problem can be solved using the provided tools.
• Tool callGTA uses ReAct-style prompt templates to allow LLMs to invoke tools in a specific format and process the results returned by the tools. The templates support LLMs in reasoning and planning, determining when and how to invoke tools.
• Multimodal processingGTA requires LLMs to process and understand multimodal inputs, including images and text, and requires models to have cross-modal understanding and reasoning capabilities.
• Fine-grained evaluation metricsGTA's evaluation metrics cover the entire tool invocation process, from the LLM tool invocation process to the execution results, providing a comprehensive evaluation of model performance.
• Model ComparisonBased on a comparison of the performance of different models on GTA, this study reveals the bottlenecks in the use of tools by existing models when dealing with real-world problems, and provides directions for improvement for future general-purpose intelligent agents.

GTA project address

• Project official website:open-compass.github.io/GTA
• GitHub repository:https://github.com/open-compass/GTA
• HuggingFace model library:https://huggingface.co/datasets/Jize1/GTA
• arXiv technical paper:https://arxiv.org/pdf/2407.08713

GTA Application Scenarios

• Smart Assistant DevelopmentGTA assesses and trains its intelligent assistant to better understand and execute complex user requests involving multiple steps and tool calls.
• Multimodal interactionIn scenarios that require processing images, text, and other multimedia content, GTA helps models learn how to combine multiple input types to solve problems.
• Automated customer serviceGTA is used to develop systems that automatically solve customer problems. These systems need to call upon different tools and resources to provide accurate answers and solutions.
• Education and trainingGTA serves as an educational tool to help students understand how to design and implement complex tasks that require multi-step reasoning and tool usage.
• Research and developmentResearchers used GTA to test and compare different LLMs, exploring new ways to leverage tool usage capabilities and advance AI technology.