OmniSearch - A multimodal retrieval enhancement and generation framework launched by Alibaba Tongyi

What is OmniSearch?

OmniSearch, developed by Alibaba's Tongyi Lab, is a multimodal search enhancement and generation framework with adaptive planning capabilities. OmniSearch dynamically decomposes complex problems and adjusts its search strategy based on search results and problem context, simulating human behavior in solving complex problems to improve search efficiency and accuracy. OmniSearch introduces a dynamic search planning framework, based on recursive search and reasoning processes, gradually approaching the solution to the problem, significantly improving the flexibility and effectiveness of multimodal search.

OmniSearch's main functions

• Adaptive retrieval planningBased on the stage of problem-solving and the current search content, each search action is planned in real time, simulating the behavior of humans when solving complex problems.
• Dynamic problem decompositionBreak down complex problems into multiple sub-problems and develop corresponding retrieval steps and strategies for each sub-problem.
• Multimodal information processingIt supports image retrieval, text retrieval, and cross-modal retrieval, and processes multimodal information including text and images.
• Iterative Reasoning and RetrievalBased on a recursive retrieval and reasoning process, it determines whether to continue searching or raise new sub-questions based on the current solution status.
• Multimodal feature interactionEffective interaction with the retrieved multimodal features allows for flexible adjustment of retrieval strategies based on different modal information.

OmniSearch's technical principles

• Planning AgentOmniSearch's planning agent is responsible for breaking down the original question step by step, and determining the next sub-question to be solved and the search tools based on the feedback from each search step.
• Retriever: Perform actual retrieval operations, conduct image retrieval, text retrieval or cross-modal retrieval according to the instructions of the planning agent, and return relevant information for the planning agent to analyze.
• Sub-question SolverSummarize and answer the retrieved content, including any multimodal large language model, or the planning agent itself.
• Dynamic retrieval planningThe core innovation lies in dynamic retrieval planning, which is based on simulating how humans think about problems and gradually approaches the final solution to the problem.
• Recursive retrieval and reasoningBased on a recursive retrieval and reasoning process, after each sub-question is posed and a preliminary answer is obtained, it is determined whether to continue the retrieval based on the current solution status.

OmniSearch's project address

• GitHub repository:https://github.com/Alibaba-NLP/OmniSearch
• arXiv technical paper:https://arxiv.org/pdf/2411.02937
• Experience the demo online:https://modelscope.cn/studios/iic/OmniSearch/summary

Application scenarios of OmniSearch

• Intelligent Customer Service SystemIt can be integrated into the intelligent customer service system to understand users' complex queries and dynamically retrieve relevant information to provide more accurate and personalized services.
• Education and learning supportIn the field of education, it helps students and researchers solve complex problems by providing in-depth academic materials and explanations based on dynamic retrieval and reasoning.
• Medical consultation and diagnosisIn the medical field, it assists doctors and patients in symptom analysis, disease diagnosis, and treatment plan retrieval, providing the latest medical knowledge and research.
• News and information aggregationUsed on news aggregation platforms, it enhances generation based on multimodal retrieval, providing users with richer and more accurate news content and background information.
• Enterprise knowledge managementIn enterprises, it helps employees quickly retrieve internal documents, reports, and data, improving work efficiency and decision-making quality.