AB
AB
AiBoss

All Categories

project

Find3D - A 3D part segmentation model developed by Caltech

Find3D is a 3D part segmentation model developed by Caltech, capable of segmenting any part of any object based on any text query. Find3D uses a powerful data engine to automatically generate training data from 3D assets on the internet...

What is Find3D?

Find3D is a 3D part segmentation model developed by Caltech, capable of segmenting any part of any object based on any text query. Find3D uses a powerful data engine to automatically generate training data from 3D assets on the internet and trains a scalable 3D model using a contrastive training method. Find3D demonstrates outstanding performance on multiple datasets, including Objaverse-General, ShapeNet-Part, and PartNet-E, achieving up to 3x the mean intersection-union (mIoU) improvement over the second-best methods, and can handle 3D field constructions from iPhone photos and AI-generated images.

Find3D's main functions

  • Open World 3D Partial SegmentationIt can identify and segment any part of any object using only a text query, without being limited by a predefined set of parts.
  • No manual annotation required: Use a data engine to automatically generate training data from 3D assets on the Internet without manual annotation.
  • High performance and generalization abilityIt performs well on multiple datasets, with a 3x improvement in mean intersection-union ratio (mIoU) compared to the second-best method.
  • Rapid reasoningIt is 6 to 300 times faster than the existing baseline, significantly improving inference speed.
  • robustnessMaintain stable segmentation results under different object poses and query conditions.
  • Query flexibilityIt supports different types of text queries, including partial queries with different granularities and descriptive styles.

The technical principles of Find3D

  • Data Engine:
    • Automatically annotate 3D objects using 2D base models such as SAM and Gemini.
    • The 3D asset is rendered into multiple views, and each view is passed to SAM for segmentation.
    • For each mask returned by SAM, query Gemini to get the corresponding part name, forming a (mask, text) pair.
    • Embed partial names into the latent embedding space of visual and language foundational models (such as SigLIP).
    • Based on projection geometry, the mask is back-projected onto the 3D point cloud to form (point, text embedding) pairs.
  • Model training:
    • A Transformer-based point cloud model that treats point clouds as sequences and performs block attention.
    • The point features returned by the model are used to perform queries on any free-form text by comparing their cosine similarity to the text embeddings.
    • The contrastive learning objective addresses the issues of label ambiguity and partial visibility, and supports scalable training on data generated by the data engine.
  • Comparative learning objectives:
    • This addresses the issue of each point having multiple labels, as well as the problem of unlabeled points caused by each mask only covering a portion of the view.
    • Based on the contrastive learning objective, the model can learn robust feature representations and achieve accurate partial segmentation in open worlds.

Find3D project address

Application scenarios of Find3D

  • Robot Vision and Operation:In the field of robotics, it helps robots identify and locate specific parts of objects for precise grasping, manipulation, or interaction.
  • Virtual Reality (VR) and Augmented Reality (AR):In VR/AR applications, it provides a deeper understanding of virtual objects and enhances the user's interactive experience with the virtual environment.
  • Computer-aided design (CAD):In CAD software, it helps designers quickly identify and edit specific parts of 3D models, improving design efficiency.
  • Game development:In game development, more complex 3D object interactions can be created, such as changing character equipment or object destruction effects.
  • Architecture and Engineering:In the fields of architecture and engineering, it helps in the analysis and understanding of complex 3D structures, such as architectural models or mechanical components.
Back