AutoConsis - An AI-powered UI content consistency detection tool launched by Meituan in partnership with Fudan University.

What is AutoConsis?

AutoConsis is an intelligent UI consistency detection tool jointly developed by Meituan's technology team and Fudan University. Based on deep learning and large-scale language models, the tool automatically identifies and extracts key data from the interface, detecting and recognizing inconsistencies between data. AutoConsis improves user experience, reduces user frustration caused by data display errors, and lightens the workload of testers. In practical applications, AutoConsis demonstrates advantages such as low cost, high generalization, and high confidence, making it significant for ensuring the quality of front-end UI design.

Main functions of AutoConsis

• Target region identificationFirst, identify the key areas in the UI that are relevant to detection, and then locate the UI parts containing important information based on image processing and pattern recognition technologies.
• Target information extractionAfter the target area is identified, the tool uses OCR technology and UI components to analyze and extract the text and elements of the target area, and uses Large Language Model (LLM) reasoning to extract the key information required for consistency verification.
• Consistency checkPerform consistency checks on the extracted information to ensure the accuracy and consistency of UI information, including checks on numerical logical types and semantic types.
• Automated intelligent inspection processImplement an automated intelligent detection process that can detect UI content consistency with low cost, high generalization, and high confidence.
• Multi-business scenario adaptationThe tool can adapt to different page templates, technology stacks, and apps, automatically adapting to cover multiple business types and diverse layouts.

The technical principles of AutoConsis

• Multimodal deep learning models: Use multimodal deep learning models to analyze GUI pages and combine image and text information to improve recognition accuracy.
• Large Language Models (LLM)LLM extracts deep semantic information from text and identifies and analyzes key data.
• Object detection and content understandingThis transforms UI page analysis tasks into a combination of object detection and content understanding, leveraging the capabilities of large models to adapt to pages with different technology stacks.
• OCR and UI component analysisThe text information is extracted using OCR technology, combined with UI component analysis tools to extract elements, and then filled into a preset CoT Prompt (Chain of Thought Prompt). Key information is extracted based on large-scale model reasoning.
• Consistency verification rulesBased on predefined numerical logic and semantic rules, the consistency of the extracted UI content is judged. The consistency of numerical logic types is directly checked using rules, and for complex semantic rules, the understanding ability of LLM is used to perform the verification.
• Information Extraction Prompt DesignTo address the common "illusion" problem in large language models, we designed Prompt (CoT), which incorporates context learning, to improve the accuracy of information extraction.
• Multimodal UI area recognitionCLIP, a visual semantic recognition model, is used to identify target regions. It uses semantic mapping between images and text to perform matching in a high-dimensional vector space, thereby improving the generalization and accuracy of recognition.

AutoConsis project address

• Technical Papers:https://dl.acm.org/doi/10.1145/3639477.3639748

Application scenarios of AutoConsis

• e-commerce platformIn e-commerce platforms, it is necessary to check whether the price and inventory information is consistent across the product details page, shopping cart page, and checkout page to ensure that the information seen by users on different pages is accurate and synchronized.
• Marketing campaign validationDuring promotional and marketing campaigns, verify that descriptions of discounts, offers, and event rules are consistent across different pages to avoid user confusion caused by inconsistencies in information.
• Financial ApplicationsIn financial applications, this involves checking whether key financial data such as account balances, transaction records, and investment returns are displayed consistently across different pages.
• Travel and hotel bookingIn travel and hotel booking apps, check that prices, availability, and booking terms are consistent across different pages to improve user experience.
• Social media and content platformsIn social media and content platforms, this involves checking whether the data and information in user profiles, posts, and comments are consistent.