project
Vision Search Assistant - An open-source framework that combines visual language models and web proxy search technology
Vision Search Assistant (VSA) is a framework that combines Visual Language Models (VLMs) and web agents to enhance the model's ability to understand unknown visual content. Based on internet retrieval, it enables VLMs to process and answer questions about unknown...
What is Vision Search Assistant?
Vision Search Assistant (VSA) is a framework that combines Visual Language Models (VLMs) and a web agent to enhance a model's ability to understand unknown visual content. Based on internet retrieval, it enables VLMs to process and answer questions about unseen images. VSA performs exceptionally well in both open and closed set question-answering tests, significantly outperforming other models including LLaVA-1.6-34B, Qwen2-VL-72B, and InternVL2-76B. Vision Search Assistant can be widely applied to existing VLMs to enhance their ability to handle new images and events.
Main functions of Vision Search Assistant
- Visual content expressionThe process of identifying key objects in an image, generating descriptions, and considering the correlations between objects is called Correlated Formulation.
- Online knowledge searchBased on an iterative algorithm called "Chain of Search", multiple sub-problems are generated, and network proxies are used to search for relevant information to obtain network knowledge related to user questions and image content.
- Collaborative generationThe VLM algorithm combines the original image, the user's question, relevant statements, and knowledge obtained through web searches to generate the final answer.
- Multimodal search engineTransform any VLM into a multimodal automated search engine that can understand and respond to visual content.
- Real-time information accessBy leveraging the real-time information access capabilities of network proxies, VLM can obtain the latest network data, thereby improving the accuracy of responses.
- Open World Search Enhanced GenerationBased on internet retrieval, this expands the ability of VLMs to process new visual content, enabling them to handle and answer questions about unseen images or new concepts.
The technical principles of Vision Search Assistant
- Visual content recognition and descriptionThe VLM tool analyzes the input image, identifies key objects in the image, and generates text describing the objects.
- Correlation analysisIt generates a description of a single object, analyzes the correlations between objects, and generates a textual representation that comprehensively considers these relationships, i.e., a related statement.
- Subproblem generationBased on the user's question and related statements, VSA uses a large language model (LLM) to generate a series of sub-questions. These sub-questions guide the search process to find more specific information.
- Web Search and Knowledge IntegrationBased on network proxy execution of sub-question searches, analyze the web pages returned by the search engine, extract and summarize relevant information, and form network knowledge.
- Iterative search processThe "Chain of Search" algorithm is used to refine the search step by step based on an iterative process, thereby obtaining richer and more accurate network knowledge.
Vision Search Assistant project address
- Project official website:cnzzx.github.io/VSA
- GitHub repository:https://github.com/cnzzx/VSA
- arXiv technical paper:https://arxiv.org/pdf/2410.21220
Application scenarios of Vision Search Assistant
- Image recognition and searchUsers upload an image, and the system identifies the content within the image and provides relevant information, such as identifying historical figures, landmarks, and species of flora and fauna.
- News Event AnalysisIt analyzes news images and provides detailed reports on event background, participant information, and event impact, helping users quickly understand the full picture of news events.
- Education and LearningIn the field of education, it assists learning, such as explaining scientific concepts, historical events, or providing visual aids in language learning.
- e-commerceOn e-commerce platforms, image search helps users find the products they want to buy, or provides detailed product information and reviews.
- Tourism PlanningUsers upload photos of their travel destinations and obtain information such as attraction descriptions, travel guides, and cultural background to help them plan their trips.