AB
AB
AiBoss

All Categories

project

SPAR - A self-play training framework developed by the Zhipu team

SPAR is a self-playing framework developed by the Zhipu team, which enhances the ability of large language models to follow instructions. The framework is based on the interaction of two roles: an internal generator and a perfecter. The generator executes instructions to produce responses, and the perfecter...

What is SPAR?

SPAR, a self-playing framework developed by the Zhipu team, enhances the instruction-following capabilities of large language models. The framework involves an interaction between two internal roles: a generator and a refiner. The generator executes instructions and produces responses, while the refiner analyzes and improves these responses. SPAR uses tree search technology to refine and optimize responses, eliminating irrelevant interference and highlighting key differences crucial for instruction adherence. This process improves the model's accuracy in executing instructions and enhances its self-improvement capabilities. Experimental results show that the SPAR framework significantly improves model performance on benchmarks such as IFEval, demonstrating its effectiveness in enhancing the instruction-following capabilities of large language models.

Main functions of SPAR

  • Improve command compliance: Improve the ability of large language models (LLMs) to accurately understand and execute instructions.
  • Constructing efficient preference pairsBased on self-game and tree search strategies, it is possible to construct effective and comparable preference pairs that are free from confounding factors, which helps the model learn key differences.
  • Self-game iterative improvementThe model is based on playing the roles of both generator and improver, engaging in self-game, and continuously improving its ability to follow instructions through iteration.
  • Tree search strategyThe tree search algorithm is used to refine the model's response, ensuring that the generated response follows the instructions more accurately.
  • Model performance optimizationBased on two models, namely the optimizer and the improver, the overall instruction compliance performance is improved.
  • Scalability and portabilityIt demonstrates the scalability and portability of models of different sizes, and can improve the instruction compliance of models of various sizes.

SPAR's technical principles

  • Self-game frameworkIn the SPAR framework, LLMs play two roles: generators and improvers. Generators generate responses, while improvers evaluate and refine those responses.
  • Tree search algorithmThe algorithm explores possible response paths and finds the best response based on tree search algorithms (including breadth-first search (BFS) and depth-first search (DFS).
  • Remove interfering factorsBased on refined response pairs, interference factors unrelated to instruction compliance are eliminated, allowing the model to focus on learning the key requirements of the instructions.
  • Iterative trainingBased on iterative training, the generator and improver models are continuously optimized, with each iteration based on the results of the previous round.
  • Data building: Construct a high-quality dataset containing complex instruction-following prompts and corresponding supervised fine-tuning (SFT) data for use in initializing and training actor and refiner models.
  • Model optimizationSPAR uses tree search strategies to generate refined response pairs, and trains actor and refiner models based on direct preference optimization (DPO) and rejection resampling fine-tuning (RFT) to achieve continuous self-improvement.

SPAR's project address

Application scenarios of SPAR

  • Smart AssistantIn personal or business intelligent assistants, this helps models better understand user commands and provide more accurate services and responses.
  • Customer ServiceIn the field of customer service, training chatbots to more accurately follow customer requests can improve problem-solving efficiency and customer satisfaction.
  • Educational TechnologyIn educational applications, it assists in the development of intelligent teaching assistants that can understand and execute complex instructions from teachers or students, providing a customized learning experience.
  • Medical consultationIn medical consultation systems, improving the model's ability to understand patient problems ensures the provision of safe and accurate medical advice and information.
  • Smart Home ControlIn the field of smart homes, this helps voice control assistants to more accurately understand and execute users' home control commands.
Back