LongReward - Tsinghua University, Chinese Academy of Sciences, and Zhipu AI jointly launched a method to improve the performance of large language models for long texts.

What is LongReward?

LongReward, jointly developed by Tsinghua University, the Chinese Academy of Sciences, and Zhipu AI, is a method for improving the performance of large language models (LLMs) for long texts based on AI feedback. LongReward scores the model's response across four dimensions: usefulness, logicality, fidelity, and completeness, providing reward signals and using reinforcement learning to optimize the model. This makes the model more accurate and consistent when processing long texts, and better able to follow instructions. Ultimately, it enhances the model's ability to process long texts and improves its efficiency in following short instructions.

The main functions of LongReward

• Multi-dimensional ratingThe response of the long text model is scored from four dimensions: helpfulness, logicality, faithfulness, and completeness.
• Reward signal providedUsing readily available large language models (LLMs) as scoring tools, reward signals are provided for the responses of long text models, and these signals are used in reinforcement learning (RL).
• Strengthen learning integrationBy combining the offline reinforcement learning algorithm DPO (Direct Preference Optimization), the model's performance is improved by optimizing the model output to meet preference requirements.
• Performance improvementSignificantly improves the model's performance on long text tasks, including better understanding and utilizing contextual information and reducing hallucinations.
• Enhanced ability to follow instructionsEnhance the model's ability to follow short instructions, thereby improving the model's usability and flexibility.

The technical principles of LongReward

• Multi-dimensional assessment:
  • HelpfulnessThe evaluation criteria include whether the model's response is relevant to the user's query, provides useful information, and meets the user's needs.
  • LogicalityCheck the logical consistency of the model's response, including the consistency of viewpoints and the correctness of reasoning.
  • Faithfulness: To measure whether the factual information in the model's response is consistent with the context, ensuring the authenticity of the information.
  • CompletenessThe assessment task is to determine whether the model's response covers all the key points in the context relevant to the question and provides sufficient information and detail.
• Utilize existing large-scale language models (LLMs)Use readily available high-performance LLMs as scoring tools to score the content generated by the model.
• Few-shot learning and Chain-of-Thought (CoT)For usefulness and logicality scoring, LLM uses few-shot learning and CoT to score directly based on query and response content.
• Factual statement decomposition and retrievalFor fidelity scoring, the model response is broken down into factual statements, and the most relevant parts of the context are retrieved to determine whether each statement is supported.
• coarse-grained context decompositionFor completeness scoring, the context is broken down into coarse-grained blocks, information relevant to the question is extracted, and then the model response is evaluated to see if it contains all the important information.

LongReward's project address

• GitHub repository:https://github.com/THUDM/LongReward
• HuggingFace model library:https://huggingface.co/datasets/THUDM/LongReward-10k
• arXiv technical paper:https://arxiv.org/pdf/2410.21252

Application scenarios of LongReward

• Long document comprehension and Q&A (Questions and Answers)Used to evaluate and improve the model's question-answering performance when processing long documents, reports, or books.
• Text Summary: To help the model better understand and summarize the main content of long articles or collections of multiple documents.
• Education and academic researchIn academic research, LongReward is used to evaluate and improve the ability of models to handle large amounts of literature and data, supporting research and learning.
• Legal and financial analysisIn the legal and financial fields, LongReward is used to analyze and understand large volumes of legal documents, contracts, or financial reports.
• Medical record analysisIn the medical field, LongReward's auxiliary models understand and analyze patients' detailed medical records to support diagnostic and treatment decisions.