Optima - Tsinghua University and Beijing University of Posts and Telecommunications jointly launch a training framework to optimize communication efficiency and task effectiveness.
Optima is a framework developed by Tsinghua University for optimizing multi-agent systems (MAS) based on large language models (LLMs). Based on an iterative generation, ranking, selection, and training paradigm, it significantly improves communication efficiency and task performance...
What is Optima?
Optima, developed by Tsinghua University, is a framework for optimizing multi-agent systems (MAS) based on large language models (LLMs). Based on an iterative generation, ranking, selection, and training paradigm, it significantly improves communication efficiency and task performance. Optima balances task performance, token efficiency, and communication readability, explores various reinforcement learning algorithms, and integrates Monte Carlo tree search to generate high-quality training data. In multi-agent tasks, Optima demonstrates performance exceeding single-agent baselines and traditional MAS, achieving up to a 2.8x performance improvement while reducing token usage. Optima's efficiency gains offer new possibilities for more efficient inference computation and improved inference time extension rules.
Optima's main functions
- Improved communication efficiency:O optimizes communication between agents in a multi-agent system (MAS), reduces the number of tokens required to complete a task, and improves communication efficiency.
- Task performance enhancementBased on the balance between iterative training and reward function, improve the performance of intelligent agents in complex tasks, including asymmetric question answering and complex reasoning tasks.
- ScalabilityIt enables MAS to remain effective when handling larger-scale and more complex tasks, improving system scalability.
- Improved Reasoning Time Extension RuleReducing token usage makes it possible to improve the inference time scaling rule, which helps to achieve better performance at a lower computational cost.
Optima's technical principles
- Iterative training paradigmBased on an iterative paradigm of generation, ranking, selection, and training, the behavior of the agent is gradually optimized.
- reward functionDesign a reward function that balances task performance, token efficiency, and communication readability, guiding the agent to complete the task while maintaining communication efficiency.
- Reinforcement learning algorithmsExplore various reinforcement learning algorithms, including supervised fine-tuning (SFT), direct preference optimization (DPO), and hybrid methods, to optimize agent behavior.
- Monte Carlo Tree Search (MCTS)Integrating MCTS heuristics, it treats dialogue turns as tree nodes, explores diverse interaction paths, and generates high-quality DPO training data.
- Multi-objective optimizationBased on the reward function, multiple objectives are considered simultaneously, which improves task performance while focusing on communication efficiency and the interpretability of the output.
Optima's project address
- Project official website:chenweize1998.github.io/optima-project-page
- GitHub repository:https://github.com/thunlp/Optima
- arXiv technical paper:https://arxiv.org/pdf/2410.08115
Application scenarios of Optima
- Information asymmetry question and answerIn question-answering systems, when the answer to a question requires integrating information from multiple sources, optimizing communication between agents can improve the accuracy of the answer and the response speed.
- Complex reasoning tasksFor problems requiring multi-step reasoning, such as legal case analysis and scientific problem solving, it helps intelligent agents collaborate more effectively and arrive at the correct conclusions.
- Software developmentIn software development, coordinating the development of different functional modules and optimizing the development process and improving code quality based on effective communication between intelligent agents are crucial.
- Decision support systemIn business decision-making or policy-making, it helps multiple decision-makers or agents share information, discuss solutions, and reach a consensus.
- Multi-agent gamesIn games that require collaboration among multiple players or agents, optimize communication strategies among players to improve teamwork efficiency.