AB
AB
AiBoss

All Categories

project

DINO-X - A general-purpose large-scale visual model launched by IDEA Research Institute

DINO-X is a general-purpose large-scale vision model developed by IDEA Research, possessing open-world object detection and understanding capabilities. It supports text, visual, and custom cues, and can recognize any object in an image without user prompts. Based on over 100 million...

What is DINO-X?

DINO-X is a general-purpose visual model developed by IDEA Research, possessing open-world object detection and understanding capabilities. It supports text, visual, and custom cues, and can recognize any object in an image without user prompts. Based on the Grounding-100M dataset with over 100 million samples, DINO-X sets new performance records in the COCO, LVIS-minival, and LVIS-val benchmarks. The model includes two versions: DINO-X Pro, which provides powerful perception capabilities, and DINO-X Edge, which optimizes inference speed and is suitable for edge deployment. DINO-X excels in long-tail object recognition and can be widely applied in fields such as autonomous driving and intelligent security, bringing new impetus to industrial upgrading.

Main functions of DINO-X

  • Open World Object Detection and SegmentationIt can detect and segment various objects in images, even those that are not common in long-tail categories.
  • Phrase locationThe model locates the corresponding object in an image based on a given text phrase.
  • Visual cue countingBased on visual cues, such as drawing bounding boxes or dots in an image, count the number of specific objects.
  • Attitude estimationPredict key points of people or specific categories in an image, such as human pose and hand pose.
  • Unannounced object detection and recognitionSupports the detection of any object in an image without any prompts.
  • Dense Area SubtitlesGenerate detailed descriptive captions for specific areas in an image.
  • Object-based question answering: Able to answer questions about specific objects in an image.

DINO-X Technical Principles

  • Transformer encoder-decoder architectureBased on the Transformer architecture, the encoder extracts image features, and the decoder performs object detection and understanding tasks.
  • Multimodal pre-trainingPre-trained on the large-scale dataset Grounding-100M, DINO-X learns rich visual and linguistic features, enhancing its ability to detect open words.
  • Tip extensionExpanded input options, supporting text prompts, visual prompts, and custom prompts, covering a wider range of detection scenarios.
  • Multi-sensor integrationIt integrates multiple perception heads, such as bounding box heads, segmentation heads, key point heads, and language heads, to support a variety of perception and understanding tasks.
  • Two-stage training strategy:
    • Phase 1Jointly train text cue detection, visual cue detection, and object segmentation.
    • Phase TwoFreeze the DINO-X backbone, add key nods and language heads for separate training, and expand the model's fine-grained perception and understanding capabilities.
  • Knowledge distillation and FP16 reasoning optimizationThe model extracts knowledge from the Pro model based on knowledge distillation technology and uses FP16 quantization technology to improve inference speed.
  • Language Header DesignThe DINO-X language header uses frozen DINO-X extracted object tags, combines them with task tags, and generates response output in an autoregressive manner.

DINO-X project address

Application scenarios of DINO-X

  • autonomous drivingThe ability to identify and understand the road environment in real time in autonomous vehicles, including pedestrians, vehicles, and traffic signs, improves safety and responsiveness.
  • Smart securityIn a monitoring system, it detects and identifies suspicious behavior, intruders, or other security threats, thereby improving the level of intelligence in monitoring.
  • Industrial testingIn manufacturing, it is used for quality control, detecting product defects, and ensuring the efficiency of production lines and product quality.
  • Robot VisionIt can be integrated into service robots and industrial robots to help them better understand their surroundings and improve their interaction and operational capabilities.
  • Assisting visually impaired peopleIt can be integrated into assistive devices for the blind to help visually impaired people better understand and navigate their surroundings.
Back