LongCite - An open-source model launched by Tsinghua University that improves the accuracy of LLM references and reduces illusions.
ongCite is a project launched by Tsinghua University aimed at improving the credibility and verifiability of large language models (LLMs) in long-text question answering tasks. The project enables users to verify the model's performance by generating fine-grained sentence-level citations...
What is LongCite?
LongCite is a project launched by Tsinghua University aimed at improving the credibility and verifiability of Large Language Models (LLMs) in long text question answering tasks. The project enables users to verify the accuracy of the model's answers by generating fine-grained sentence-level citations. Its core components include the LongBench-Cite evaluation benchmark, the CoF automated data construction process, the LongCite-45k dataset, and the LongCite-8B and LongCite-9B models trained on this dataset. The models can understand long text content and provide accurate question-answering services, while also providing directly referential text citations to enhance the transparency and reliability of the information.
LongCite's main functions
- Generate fine-grained referencesLongCite enables language models to generate sentence-level references when answering long text questions, allowing users to directly access specific information in the original text.
- Increase the fidelity of the answersLongCite helps ensure that the model's answers are more faithful to the original text, reducing the "illusion" (i.e., generating information that does not match the original text) that the model may produce.
- Enhance verifiabilityUsers can verify the authenticity and accuracy of answers based on the fine-grained references provided by the model, thereby improving the credibility of the model's output.
- Automated data constructionLongCite employs the CoF (Coarse to Fine) process to automatically generate high-quality long-text question-answering data with fine-grained references, providing rich annotation resources for model training.
- Evaluation CriteriaLongCite introduces the LongBench-Cite benchmark to measure a model’s ability to generate citations in long text question answering, including correctness and citation quality.
LongCite's technical principles
- Long text processing capabilitiesLongCite supports large language models with very long context windows (such as GLM-4-9B-1M, Gemini 1.5, etc.), and can process and understand texts of tens of thousands of words.
- Fine-grained reference generationLongCite trains models to generate sentence-level citations, enabling each answer to be traced back to a specific sentence in the original text, thus improving the verifiability of the answers.
- Automated Data Construction Process (CoF)This method automatically generates question-and-answer pairs from long texts using a self-instructed approach. It retrieves sentence blocks related to the answers from the long text and generates block-level references. Based on these block-level references, it extracts the specific sentences supporting each statement and generates sentence-level references.
- Supervised Fine-Tuning (SFT)The high-quality dataset with fine-grained references generated by the CoF process is used to fine-tune large language models, improving their performance in long text question answering tasks.
LongCite's project address
- GitHub repository:https://github.com/THUDM/LongCite
- HuggingFace model library:https://huggingface.co/THUDM
- arXiv technical paper:https://arxiv.org/pdf/2409.02897
Application scenarios of LongCite
- academic researchResearchers and scholars use LongCite to search through a vast amount of literature and obtain detailed answers with citations to support their research.
- Legal ConsultationLegal professionals use LongCite to analyze legal documents, obtain specific legal provisions or case citations, and support legal analysis and case research.
- Financial AnalysisFinancial analysts and investors use LongCite to understand complex financial reports and market research, and to obtain accurate references to key data and trends.
- Medical consultationHealthcare professionals rely on LongCite to search medical literature and obtain citations for diagnostic and treatment recommendations based on the latest research findings.
- News reportJournalists and news organizations use LongCite to verify information in their reports, ensuring the accuracy of published news content and providing reliable source citations.