project
Finedefics - A fine-grained, multimodal large-scale model developed by a Peking University team.
Finedefics, developed by Professor Peng Yuxin's team at Peking University, is a fine-grained multimodal large-scale model that improves the performance of multimodal large language models (MLLMs) in fine-grained visual recognition (FGVR) tasks. The model enhances the performance of these models by incorporating object-specific...
What is Finedefics?
Finedefics is a fine-grained multimodal large model developed by Professor Peng Yuxin's team at Peking University. It improves the performance of multimodal large language models (MLLMs) in fine-grained visual recognition (FGVR) tasks. The model solves the problem of misalignment between visual objects and fine-grained subcategories in traditional models by introducing fine-grained attribute descriptions of objects and aligning the representations of visual objects and category names based on contrastive learning.
The main functions of Finedefics
- Improved fine-grained visual recognition capabilitiesFinedefics addresses the misalignment between visual objects and fine-grained subcategories in traditional models by introducing fine-grained attribute descriptions of objects and using contrastive learning to align the representations of visual objects with category names.
- Data and knowledge collaborative trainingThe model constructs fine-grained attribute knowledge of visual objects by prompting a large language model, aligns this knowledge with images and text, and achieves collaborative training of data and knowledge.
- High performanceOn several authoritative fine-grained image classification datasets (such as Stanford Dog-120, Bird-200, FGVC-Aircraft, etc.), Finedefics achieved an average accuracy of 76.84%, which is a significant improvement compared to other similar models.
- Attribute description construction and alignmentBy mining key features that distinguish fine-grained subcategories (such as coat color and coat type), Finedefics transforms them into natural language descriptions. Using these descriptions as intermediate points, Finedefics aligns visual objects with category names in the representation space of a large language model.
Finedefics' technical principles
- Attribute description constructionFinedefics first constructs a feature set by attribute descriptions and mines key features that distinguish fine-grained subcategories, such as fur color, fur type, and fur texture. These features are extracted as attribute pairs of image objects (e.g., "fur color: brown"), and then transformed into object attribute descriptions in natural language (e.g., "the kitten in the picture has brown fur with stripes and a soft texture").
- Attribute Enhancement AlignmentFinedefics uses attribute-enhanced alignment, employing constructed object attribute descriptions as common alignment targets between visual objects and fine-grained subcategories. The model leverages object-attribute, attribute-category, and category-category contrastive learning to fully establish the correspondence between visual objects and fine-grained subcategories.
- Comparative learning and instruction fine-tuningDuring the training phase, Finedefics employs contrastive learning, inputting global representations of visual objects, attribute descriptions, and category names into a large language model. It optimizes alignment by introducing difficult negative samples. Fine-tuning through instructions further enhances its performance in fine-grained visual recognition tasks.
Finedefics' project address
- Github repository:https://github.com/PKU-ICST-MIPL/Finedefics
- HuggingFace model library:https://huggingface.co/StevenHH2000/Finedefics
- arXiv technical paper:https://arxiv.org/pdf/2501.15140
Application scenarios of Finedefics
- Biodiversity monitoringFinedefics can be used to automatically identify and classify biological species, such as distinguishing different kinds of birds, plants, or animals.
- Intelligent TransportationIn the transportation sector, Finedefics can be used for vehicle identification and classification, such as distinguishing between different brands or models of cars (e.g., BMW, Mercedes-Benz, Audi), and can further identify different models within the same brand (e.g., Audi A4, A6, A8).
- Smart RetailFinedefics can be applied in retail scenarios to help identify and classify goods, such as different types of fruits, flowers, or other retail products. It can be used for inventory management, product recommendations, and automated checkout systems to improve the operational efficiency of retail businesses.
- Industrial testing and quality controlIn industrial production, Finedefics can be used to detect and classify fine-grained differences in parts or products, such as identifying different models of mechanical components or detecting product quality problems.