OpenScholar - An open-source academic search tool developed by the University of Washington in conjunction with the Allen Institute for Advanced Study.

What is OpenScholar?

OpenScholar, a retrieval-enhanced language model (LM) jointly developed by the University of Washington and the Allen Institute for AI, helps scientists answer questions based on relevant papers retrieved and synthesized from scientific literature. The system uses a large-scale database of scientific papers, a custom-designed searcher and reorderer, and an optimized 8B-parameter language model to generate accurate answers based on real-world literature. OpenScholar surpasses existing proprietary and open-source models in providing factual answers and accurate citations. On ScholarQA Bench, OpenScholar-8B outperforms GPT-4o by 5% in accuracy and PaperQA2 by 7%. All related code and data are open-source, supporting and accelerating scientific research.

Main functions of OpenScholar

• Literature retrieval and synthesisIt retrieves a large amount of scientific literature and synthesizes relevant information to answer user queries.
• Generate reference-based answersThe generated answers contain accurate citations, improving the reliability and transparency of the responses.
• Interdisciplinary applicationsIt is applicable to multiple scientific fields, including computer science, biomedicine, physics, and neuroscience.
• Improve search efficiencyBased on specialized search and rearranger tools, it improves the efficiency and accuracy of retrieving relevant scientific literature.
• Self-feedback iterationUse a self-feedback mechanism to iteratively improve answers, thereby enhancing answer quality and the completeness of citations.

OpenScholar's technical principles

• Data storage (OpenScholar Datastore)It contains over 45 million scientific papers and their corresponding 237 million paragraph embeddings, providing the foundational data for retrieval.
• Specialized search engines and reorderersRetrieval and reordering machines trained for storing scientific literature data are used to identify and sort relevant literature paragraphs.
• 8B Parametric Language ModelA large 8B-parameter language model optimized for scientific literature synthesis tasks, balancing performance and computational efficiency.
• Self-feedback generationDuring inference, the model output is iteratively refined based on natural language feedback. Each iteration may involve additional literature searches to improve the quality of the response and fill citation gaps.
• Iterative retrieval enhancementAfter generating an initial answer, the model generates feedback to guide further retrieval, iteratively improving the answer until all feedback has been processed.

OpenScholar project address

• Project official website:allenai.org/blog/openscholar
• GitHub repository:https://github.com/AkariAsai/OpenScholar
• HuggingFace model library:https://huggingface.co/collections/OpenScholar/openscholar-v1-67376a89f6a80f448da411a6
• arXiv technical paper:https://arxiv.org/pdf/2411.14199

Application scenarios of OpenScholar

• Research supportResearchers can quickly access the latest research findings, helping them stay up-to-date with their knowledge in their field.
• literature reviewWhen writing academic papers or reports, authors integrate and summarize a large amount of literature to improve writing efficiency.
• Interdisciplinary researchBecause OpenScholar covers multiple scientific fields, it helps researchers explore the connections and intersections between different disciplines.
• Education and LearningIt assists students and teachers in learning and teaching, enabling them to obtain in-depth literature analysis and summaries.
• Technical monitoringCorporate R&D departments monitor technological development trends, especially in rapidly changing technological fields.