SpatialClaw - A training-free spatial inference framework jointly developed by NVIDIA and KAIST

What is SpatialClaw?

SpatialClaw is a training-free spatial reasoning framework jointly developed by NVIDIA Research and KAIST. It iteratively executes complex 3D/4D spatial reasoning tasks in a persistent Python kernel through a code-as-action mechanism, achieving an average accuracy of 59.9% on 20 benchmarks, surpassing the previous best method by 11.2 percentage points.

Main functions of SpatialClaw

• Code as an Action InterfaceThe agent writes Python code cell by cell in the persistent Jupyter kernel and freely combines the output of perception tools such as depth reconstruction and segmentation masking.
• Five-stage agent cycleThe system sequentially executes planning, code generation, AST security verification, execution and feedback assembly, and answer submission or iterative correction.
• Multimodal sensing integrationIt natively supports vision tools such as Depth Anything 3 and SAM 3, and works with NumPy/SciPy for real-time geometric calculations.
• Completely training-free deploymentThe same set of system prompts and tools runs directly on six backbone networks, including Qwen3.5/3.6 and Gemma4.
• Dynamic 4D Inference SupportIt covers spatial relationship understanding and measurement calculation for single-image, multi-view, video, and 4D temporal scenes.

Technical Principles of SpatialClaw

• Code as an Action InterfaceSpatialClaw replaces traditional structured tool calls with Python code generation. VLM allows users to freely write, execute, and modify code within a persistent Jupyter kernel. The action space is no longer limited to a predefined JSON format, and can express arbitrarily complex spatial computation logic.
• Five-stage agent cycleThe system sequentially executes planning, code generation, AST security verification, execution and feedback assembly, and answer submission or iterative correction, forming a closed loop; the variable states and visualization results after execution are fed back to VLM as multimodal feedback, driving the next step of code correction.
• Persistent Jupyter kernelAfter the code is executed, variables and objects remain in memory, and subsequent cells can directly reuse intermediate results, supporting multi-step iterative reasoning; the output of matplotlib in the kernel is captured as image feedback, enabling VLM to "see" the calculation process and adjust accordingly.
• Perception tool integrationVisual foundational models such as Depth Anything 3 and SAM 3 are exposed as Python modules. VLM enables modular combination by freely arranging depth estimation, segmentation mask generation, and NumPy/SciPy geometric calculations through code.
• AST security checkIt intercepts dangerous calls through static analysis of abstract syntax trees, allowing only whitelisted detection tools and mathematical libraries to execute, thus preventing malicious code and controlling the upper limit of resource consumption.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use SpatialClaw

• Environmental preparationClone the GitHub repository and configure persistent Jupyter kernel and dependencies such as Depth Anything 3 and SAM 3.
• Model AccessChoose a VLM supported by Qwen3.5/3.6 or Gemma4 as the inference backbone.
• Task InputSubmit image, multi-view, or video data that contains spatial reasoning questions.
• Agent loopThe system automatically executes an iterative cycle of planning, code generation, AST verification, execution, and feedback assembly.
• Results AcquisitionExtract the final spatial reasoning answer from the kernel variable state and visualization output.

SpatialClaw's core advantages

• No training required, plug and playNo benchmarks or model-specific fine-tuning are required, reducing deployment barriers and computing costs.
• Code action flexibilityCompared to fixed JSON tool calls, Python code can express arbitrarily complex spatial computation logic.
• Significantly superior performanceIt improved performance by 17.6 and 15.3 percentage points on dynamic tasks such as DSI-Bench and MindCube, respectively.
• Cross-model generalityThe same architecture achieved consistent gains across six different VLMs with parameter scales ranging from 26B to 397B.
• Safe execution guaranteeBuilt-in AST security verification mechanism to prevent malicious code execution and ensure kernel stability.

SpatialClaw project address

• Project official websitehttps://spatialclaw.github.io/
• GitHub repositoryhttps://github.com/NVlabs/SpatialClaw
• arXiv technical paper: https://spatialclaw.github.io/static/pdfs/spatialclaw.pdf

SpatialClaw's Competitive Product Comparison

Application Scenarios of SpatialClaw

• Robot navigation and graspingIt provides accurate object metric distance measurement and spatial relationship judgment for mobile robots and robotic arms, and assists in path planning and grasping posture decision-making.
• AR/VR Indoor LayoutIt automatically identifies the relative positions and dimensions of elements such as furniture, doors, and windows, generates a reasonable interior layout plan, and overlays it onto the virtual space in real time.
• Autonomous driving perceptionIt integrates inputs from multiple cameras to reconstruct 3D scenes and understands the spatial topological relationships of road elements to support decision-making and planning.
• Building safety inspectionBased on multi-view image analysis, the spatial compliance of construction site structures is assessed, and safety indicators such as scaffold spacing and guardrail height are detected.
• Industrial Precision MeasurementNon-contact 3D dimensional measurement and geometric tolerance inspection of parts are performed through multi-view depth estimation.