AB
AB
AiBoss

All Categories

project

Aligner - A residual correction model alignment technology developed by Peking University

Aligner is a large language model alignment technique proposed by a team from Peking University. It improves model performance by learning the corrected residuals between aligned and unaligned answers. It employs an autoregressive seq2seq model, working in the question-answer-corrected residual relationship...

What is Aligner?

Aligner, a language model alignment technique proposed by a team from Peking University, improves model performance by learning the corrected residuals between aligned and unaligned answers. It employs an autoregressive seq2seq model, trained on a Query-Answer-Correction (Q-A-C) dataset, eliminating the need for complex reinforcement learning from human feedback (RLHF) processes. Aligner's core advantages lie in its efficiency and flexibility. As a plug-and-play module, it can be directly applied to various open-source and API-based models without requiring access to model parameters.

Aligner's main functions

  • Corrected residual learningAligner, as an autoregressive seq2seq model, is trained on the Query-Answer-Correction (Q-A-C) dataset to learn the differences between aligned and unaligned answers, achieving more accurate model alignment.
  • Weak to strong generalizationFine-tuning LLMs with a large number of parameters using an Aligner model with a small number of parameters can significantly improve the performance of strong models.
  • Plug and playAligner can align models such as GPT3.5, GPT4, and Claude2, for which parameters are not available.
  • Training process:
    • Data collectionIt retrieves queries from various open-source datasets and generates original answers.
    • Answer correctionThe original answer was corrected using GPT-4, Llama2-70B-Chat, and human annotation to align with human values.
    • Model trainingBased on the corrected dataset, the Aligner model is trained to correct the original answers to aligned answers.

Aligner's performance

  • Enhance helpfulness and safetyExperiments show that using Aligner-7B can improve the helpfulness and safety of GPT-4, by 17.5% and 26.9%, respectively.
  • Weak to strong generalizationBased on the fine-tuning of Llama2-70B under the supervision of Aligner-13B, the assistiveness and safety were improved by 8.2% and 61.6%, respectively.
  • Multi-model compatibilityAligner-7B alignment improves the help and security of 11 models, including closed-source, open-source, and secure/unsecure aligned models.

Aligner's project address

Aligner application scenarios

  • Multi-turn dialogue scenariosIn multi-turn dialogues, Aligner can improve the alignment of dialogues, especially under the challenge of sparse rewards.
  • Alignment of human values with reward modelsAligner can be trained on specific corpora and correct the output of the preceding model to reflect specific values.
  • Streaming and Parallel Processing of MoE-AlignerBy specializing and integrating Aligner, a more powerful and comprehensive Hybrid Expert (MoE) Aligner can be created.
Back