MMSearch - A benchmark tool for evaluating the capabilities of large-scale multimodal AI search engines.
MMSearch is a benchmark for evaluating the capabilities of large multimodal models (LMMs) as AI search engines. It includes an MMSearch-Engine framework and the MMSearch test suite, which contains 300 questions covering 14...
What is MMSearch?
MMSearch is a benchmark for evaluating the capabilities of large multimodal models (LMMs) as AI search engines. It includes an MMSearch-Engine framework and an MMSearch test set containing 300 questions covering 14 subdomains. The MMSearch-Engine framework enhances the search capabilities of LMMs through three stages: question reconstruction, webpage ranking, and answer summarization. The test set is divided into two categories: news and knowledge, ensuring that the test content does not overlap with the LMMs' training data and providing a fair evaluation. Experiments show that the GPT-4o model performs exceptionally well on the MMSearch benchmark, outperforming the commercial product Perplexity Pro. Research has found that increasing the computational cost during testing is more efficient than increasing the model size.
MMSearch's main functions
- Multimodal search capability assessmentMMSearch aims to evaluate the search capabilities of large multimodal models (LMMs) when handling complex queries that include both images and text.
- Problem RefactoringTransform the user's original query into a format suitable for search engine processing.
- Page sortingSelect the most relevant web pages from the search engine results.
- Answer SummaryExtract and summarize answers from selected web page content.
MMSearch's technical principles
- MMSearch-Engine framework:
- Problem RequeryIt uses LMMs to understand the user's query intent and transforms it into a format that the search engine can understand. If the query contains images, it also uses tools such as Google Lens to identify key information in the images.
- Page ranking (Rerank)Use LMMs to reorder the results returned by the search engine to determine which web pages are most likely to contain the correct answer.
- Answer summaryExtract relevant information from selected web pages and summarize it into an answer.
- Dataset DesignMMSearch contains 300 manually collected query instances covering the news and knowledge domains, ensuring no overlap with the training data of LMMs.
- Evaluation strategyThe performance of LMMs was evaluated through end-to-end tasks and individual tasks (question restructuring, webpage sorting, answer summarization).
MMSearch project address
- Project official websitemmsearch.github.io
- GitHub repository:https://github.com/CaraJ7/MMSearch
- HuggingFace model library:https://huggingface.co/datasets/CaraJ/MMSearch
- arXiv technical paper:https://arxiv.org/pdf/2409.12959
Application scenarios of MMSearch
- academic researchResearchers used MMSearch to evaluate and compare the performance of different large multimodal models in processing complex queries that include images and text.
- Search Engine OptimizationSearch engine developers use MMSearch's framework and evaluation methods to improve search engine algorithms, enabling them to better understand and respond to users' multimodal queries.
- Smart AssistantWhen developing intelligent assistants or chatbots, MMSearch enhances the assistant's ability to understand and respond to user-inputted images and text.
- Content recommendation systemContent recommendation platforms use MMSearch's technical principles to more accurately analyze user query intent, thereby providing more relevant recommended content.
- Education and trainingThe educational tools integrate MMSearch's assessment methods to design courses and training materials, helping students better understand the concept of multimodal information retrieval.