LocateAnything - NVIDIA's visual language localization model

What is LocateAnything?

LocateAnything is a visual language localization model from NVIDIA. Based on parallel bounding box decoding technology, it allows users to accurately select targets in images by inputting natural language. The model supports tasks such as multi-object detection, GUI localization, OCR text detection, and point-level pointing. Its inference speed reaches 12.7 BPS (H100), which is 10 times faster than Qwen3-VL and achieves state-of-the-art performance on benchmarks such as LVIS. It is suitable for robotics, document intelligence, and autonomous driving scenarios.

LocateAnything's main functions

• This indicates that the location is being reached.Locate and select specific targets in an image based on natural language descriptions.
• Multi-target dense detectionIt supports the simultaneous detection of multiple object categories in an image and outputs a dense set of bounding boxes.
• GUI element positioningIt can identify and locate interactive elements such as buttons and input boxes in the interface, and supports point-level output.
• Text detection and OCRIt can locate text regions in images and supports the recognition of text in documents, scenes, and table structures.
• Point-level positioningSupports outputting precise point coordinates for fine-grained pointing tasks.
• Hybrid reasoning modeThe default mode is fast parallel mode, which automatically reverts to stable serial mode when encountering complex scenarios.

The technical principles of LocateAnything

• Parallel Frame Decoding (PBD)The model treats each bounding box or point as an atomic unit, predicting the complete coordinate set in parallel during a single forward propagation. (x1,y1,x2,y2)It is not a traditional method of generating tokens sequentially.
• Model ArchitectureThe model uses the Moon-ViT visual encoder to extract native resolution visual features, which are then mapped by a dual-layer MLP projector and input into the Qwen2.5 language decoder for localization and inference.
• Structured outputThe model generates semantic blocks, bounding box blocks, negative sample blocks, and end blocks, through... <box>,<ref> Special tokens are used to organize and locate structural results.
• Hybrid reasoning mode: By default, it uses Fast Parallel Mode (MTP) to maximize throughput, and automatically and seamlessly falls back to Stable Serial Mode (NTP) when encountering format anomalies or spatial ambiguities.
• Corrective re-decodingWhen parallel decoding detects a distorted frame structure or a coordinate conflict, the problematic block is automatically discarded and the system reverts to the verified prefix, regenerating and correcting it via NTP.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use LocateAnything

• Environmental preparationClone the Eagle repository from GitHub and enter... Embodied Directory Execution pip install -e . Dependency installation complete.
• Model loadingImport LocateAnythingWorker Class and instantiate, load from Hugging Face nvidia/LocateAnything-3B Pre-trained weights.
• Target detection: call detect() The method takes an image and a list of category names (e.g., ...). ["person", "car"]The model returns the bounding box coordinates of all matching targets.
• Phrase location:use ground_multi() The method inputs an image and descriptive text (such as...) "people wearing red shirts"), to locate specific targets that conform to the semantic description.
• Text detection: call detect_text() The method involves the model automatically identifying text regions in an image and outputting the corresponding set of bounding boxes.
• GUI positioning:pass ground_gui() The method locates the interface element and sets... output_type="point" It can output point coordinates instead of bounding boxes.
• Point-level pointing:use point() The method takes an image and description as input and obtains the precise single-point coordinates of the specified target for fine-grained pointing.
• Batch inferenceConfiguration la_flash Run and will batch_utils/ join in PYTHONPATH, call detect_batch() Simultaneously process multiple image-query pairs.
• Results Analysis: Extracting output <box> Integer coordinates within the label (range 0-1000), divided by 1000, are mapped to the width and height pixel values of the original image.
• Fine-tuning adaptationPrepare JSONL data recipes and use them. torchrun Launch the full fine-tuning script, or run the LoRA script to freeze the backbone and train only the projector and low-rank adapter layers.

LocateAnything's core advantages

• Speed LeadingIt achieves 12.7 BPS on a single H100 image, which is about 10 times faster than Qwen3-VL and about 2.5 times faster than Rex-Omni.
• Accuracy SOTAAchieve best performance on multiple benchmarks including LVIS, M6Doc, ScreenSpot-Pro, and DocLayNet.
• General and unifiedA single model can cover a variety of tasks such as detection, localization, GUI, OCR, and layout, without the need to switch to a dedicated model for different scenarios.
• Geometric consistencyParallel decoding preserves the geometric coupling relationship of the coordinates within the frame, avoiding structural distortions caused by serial generation.
• Video memory optimization:pass la_flash At runtime, peak video memory can be reduced from 35GB to 11GB on consumer/data center GPUs such as the A100.

LocateAnything's project address

• Project official websitehttps://research.nvidia.com/labs/lpr/locate-anything/
• GitHub repositoryhttps://github.com/NVlabs/Eagle/tree/main/Embodied
• HuggingFace model libraryhttps://huggingface.co/nvidia/LocateAnything-3B
• Technical Papers: https://research.nvidia.com/labs/lpr/locate-anything/LocateAnything.pdf

Comparison of LocateAnything with similar competing products

Application scenarios of LocateAnything

• Embodied Intelligence and RoboticsIt provides robots with visual perception capabilities, enabling them to locate and manipulate targets through language commands.
• Intelligent document processingAutomatically identifies and selects structured elements such as titles, tables, paragraphs, and stamps in a document.
• GUI automated testingIt can locate buttons, input boxes, and icons in the interface and supports automated testing and RPA processes.
• Autonomous driving perceptionIt can densely detect vehicles, pedestrians, and traffic signs in street view images and output accurate bounding boxes.
• Retail and Warehousing: Describes how to quickly locate a specific item on a shelf or a target package in a warehouse.