LLMDet - An open vocabulary object detection model launched by Alitongyi in collaboration with Sun Yat-sen University and other institutions.

What is LLMDet?

LLMDet is an open-source lexical object detector developed by Alibaba Group's Tongyi Lab, the School of Computer Science and Engineering at Sun Yat-sen University, and Pengcheng Lab, among other institutions. It enhances object detection performance through co-training with Large Language Models (LLMs). LLMDet collects datasets containing images, localization labels, and detailed image-level descriptions (GroundingCap-1M), enriching visual features with long descriptions generated by LLMs. It is trained based on standard localization and description generation losses. LLMDet has achieved excellent zero-shot detection performance on multiple benchmarks. As a powerful foundational visual model, it can be further used to build more robust multimodal models, achieving a mutually beneficial relationship with LLMs.

LLMDet's main functions

• Open vocabulary detectionLLMDet can detect any category of target that has not been seen during the training phase, and achieves the recognition of new categories based on the alignment of text labels and visual features.
• Zero-sample transfer capability: It demonstrates strong generalization ability when directly transferred to a new dataset for detection without target category labeling.
• Image understanding and description generationLLMDet can generate image-level captions, containing rich details such as object type, texture, color, and action, helping the model better understand image content.
• Improve the performance of multimodal modelsAs a foundational visual model, it can be combined with large language models (LLM) to build more powerful multimodal models and improve the model's performance in tasks such as visual question answering and image description.

LLMDet's technical principles

• Dataset ConstructionThe GroundingCap-1M dataset was used, where each image is accompanied by a location label and a detailed image-level description. The description contains rich details, helping the model better understand the objects in the image and their relationships.
• Model ArchitectureIt consists of a standard open-vocabulary object detector and an LLM (Locally Module). The detector is responsible for extracting image features and locating objects, while the LLM uses these features to generate detailed image-level descriptions and region-level phrases.
• Collaborative trainingLLMDet achieves co-optimization with LLM based on a two-stage training process. First, the projector is trained to map the detector's features to the LLM's input space. Then, the detector, projector, and LLM are fine-tuned as a whole, with training objectives including standard localization loss and description generation loss.
• Multi-task learningLLMDet introduces image-level and region-level description generation tasks, enriching visual features with detailed descriptions and improving the model's overall image understanding. The multi-task learning approach enhances detection performance and strengthens the model's open-vocabulary capabilities.

LLMDet project address

• GitHub repository:https://github.com/iSEE-Laboratory/LLMDet
• arXiv technical paper:https://arxiv.org/pdf/2501.18954

Application scenarios of LLMDet

• Smart securityIt can detect abnormal targets or behaviors in the camera in real time, is highly adaptable, and does not require retraining.
• autonomous drivingIt helps vehicles identify various obstacles and unfamiliar scenes on the road, improving safety and reliability.
• Image content reviewAutomatically review image content, identify illegal or inappropriate content, and improve review efficiency.
• Intelligent photo album managementAutomatically categorizes and labels photos for easy searching and management, and supports multiple unseen categories.
• Medical image analysisIt can analyze medical images and quickly identify abnormal areas without requiring a large amount of labeled data.