PIKE-RAG - A search-enhanced generative framework developed by Microsoft Research Asia

What is PIKE-RAG?

PIKE-RAG (Pecicalized Knowledge and Rational Augmented Generation) is a retrieval-enhanced generative framework developed by Microsoft Research Asia, addressing the limitations of traditional RAG systems in complex industrial applications. PIKE-RAG is based on extracting, understanding, and applying professional knowledge to construct coherent reasoning logic, guiding large language models (LLMs) to generate accurate answers. PIKE-RAG introduces knowledge atomizing, decomposing knowledge into fine-grained atomic units, storing them in question format for efficient retrieval and organization. PIKE-RAG also proposes a multi-agent planning module for handling creative problems, enabling reasoning and planning from multiple perspectives.

Main functions of PIKE-RAG

• Professional knowledge extraction and understandingExtracting domain-specific knowledge from diverse data sources and transforming it into structured knowledge units provides precise knowledge support for complex problems.
• Reasoning Logic ConstructionBased on dynamic task decomposition and knowledge perception-based reasoning path planning, a coherent reasoning logic is gradually constructed to guide the language model to generate accurate answers.
• Multi-hop problem handlingBased on knowledge atomization and task decomposition, complex problems are broken down into multiple atomic problems, and multi-hop reasoning tasks are solved step by step.
• Creative Problem SolvingIntroducing multi-agent systems allows for reasoning and planning from multiple perspectives, inspiring innovative solutions.
• Phased System DevelopmentBased on the complexity of the task, it supports phased development from basic factual problems to advanced creative problems, gradually improving the system's capabilities.

PIKE-RAG Technical Principles

• Knowledge atomizationThe knowledge in a document is broken down into fine-grained "atomic knowledge" and stored in the form of questions. This atomic knowledge serves as an index for retrieval, more efficiently matching user questions and improving the accuracy of knowledge retrieval.
• Knowledge perception task decompositionThe system dynamically decomposes complex problems into multiple atomic problems and selects the optimal reasoning path based on the knowledge base. It iteratively retrieves and selects relevant information and constructs a complete reasoning logic step by step.
• Multi-agent planningWhen tackling creative problems, multiple agents are introduced, each reasoning and planning from a different perspective. Based on the collaborative work of these multiple agents, more comprehensive and innovative solutions are generated.
• Multi-granularity searchThis approach enables multi-granularity retrieval within a multi-layered heterogeneous knowledge graph, progressively refining the search scope from overall documents to fine-grained knowledge units. By leveraging the structure of the multi-layered knowledge graph, the efficiency and accuracy of knowledge retrieval are improved.
• Phased System DevelopmentBased on task complexity, the RAG system is divided into different levels (L1-L4) to progressively improve system capabilities. Each level targets a specific type of problem, from simple factual problems to complex creative problems, gradually enhancing the system's reasoning and generation capabilities.

PIKE-RAG project address

• GitHub repository:https://github.com/microsoft/PIKE-RAG
• arXiv technical paper:https://arxiv.org/pdf/2501.11551

Application scenarios of PIKE-RAG

• legal fieldIt assists legal professionals in interpreting regulations and analyzing cases, providing precise legal advice and suggestions.
• medical fieldIt helps doctors diagnose diseases and plan treatment strategies, providing medical advice based on professional knowledge.
• Semiconductor designIt helps engineers understand complex physical principles and optimize semiconductor design and development processes.
• Financial sectorUsed in risk assessment and market forecasting to provide data support and analysis reports for investment decisions.
• Industrial manufacturingOptimize production processes and supply chain management to improve industrial efficiency and quality control.