SAM 3 - Meta's open-source visual segmentation model

What is SAM 3?

SAM 3 (Segment Anything Model 3) is Meta AI's latest advanced computer vision model, capable of detecting, segmenting, and tracking objects in images and videos using text, examples, and visual cues. The model supports open-vocabulary phrase inputs, possesses powerful cross-modal interaction capabilities, and can correct segmentation results in real time. SAM 3 delivers superior performance in image and video segmentation tasks, outperforming existing systems by twice as much, and supports zero-shot learning. The model extends to 3D reconstruction, supporting applications in various scenarios such as home previews, creative video editing, and scientific research, providing a powerful impetus for the future development of computer vision.

SAM 3's main functions

• Multimodal prompt supportSAM 3 supports the detection and segmentation of objects in images and videos using text, examples, and visual cues such as clicks and selection boxes, catering to a variety of user needs.
• Image and video segmentationSAM 3 can detect and segment all matching objects in an image, supports object tracking in a video, and can interactively correct segmentation results in real time.
• Zero-shot learningSAM 3 can process unseen concepts through textual prompts with open vocabulary and can segment new object categories without additional training.
• Real-time interactivityIt allows users to correct model errors by adding additional prompts (such as clicks or selection boxes), further optimizing segmentation results and improving user experience.
• Cross-domain applicationsSAM 3 is widely used in creative media tools (such as Instagram Edits), home decor previews (such as Facebook Marketplace), and scientific fields (such as wildlife monitoring).

SAM 3 Technical Principles

• Unified model architectureSAM 3 is based on a unified model architecture that supports segmentation tasks in both images and videos. The model combines a powerful visual encoder (such as the Meta Perception Encoder) with a text encoder, capable of handling textual cues with open words. The model architecture includes an image-level detector and a memory-based video tracker, both sharing the same visual encoder.
• Multimodal input processing:
  • Text EncoderThe text prompts are encoded into feature vectors to guide the segmentation task.
  • Visual encoderEncodes image or video frames into feature vectors for object detection and segmentation.
  • Fusion encoderThis method fuses textual and visual features to generate conditional image features for subsequent segmentation tasks.
• Presence HeadTo improve the model's classification ability, SAM 3 introduces a Presence Head, specifically used to predict whether a target concept exists in an image or video. This helps decouple the recognition and localization tasks, improving the model's accuracy and efficiency.
• Large-scale data engineTo train SAM 3, Meta built a high-efficiency data engine, combining human and AI-assisted annotation to generate high-quality labeled data for over 4 million unique concepts. The data covers a variety of visual domains and tasks, ensuring the model has broad generalization capabilities.
• Zero-shot learningSAM 3 supports zero-shot learning and can handle unseen concepts. Using textual cues with open vocabulary, the model identifies and segments new object categories using pre-trained visual and language encoders.
• Real-time interactivitySAM 3 supports real-time interactivity, allowing users to correct model errors and further optimize segmentation results by adding additional prompts (such as clicks or selection boxes). Interactivity enables the model to better adapt to user intent.
• Video tracking and segmentationIn video tasks, SAM 3 uses a memory-based tracker to handle the spatiotemporal consistency of objects. The tracker uses the detector output and historical information from memory to generate a high-quality segmentation mask, which is then propagated between video frames.

SAM 3 project address

• Project official websitehttps://ai.meta.com/sam3/
• GitHub repositoryhttps://github.com/facebookresearch/sam3/
• Experience the demo onlinehttps://www.aidemos.meta.com/segment-anything

Application scenarios of SAM 3

• Creative Media ToolsCreators can quickly apply special effects to people or objects in videos, improving creative efficiency.
• Home Decor PreviewIn the Facebook Marketplace, SAM 3 supports the "Room Preview" feature, which allows users to preview how home furnishings will look in their own space, aiding in purchasing decisions.
• Scientific ApplicationSAM 3 is used in wildlife monitoring and marine exploration to help researchers better understand and protect the natural environment, for example, by analyzing wildlife behavior through video.
• 3D ReconstructionSAM 3D can reconstruct 3D objects and the human body from a single image, providing a new standard for 3D reconstruction in physical world scenes and facilitating virtual reality and augmented reality applications.
• Video creationSAM 3 provides AI visual creation tools that support the mixing and editing of existing AI-generated videos, enhancing creative flexibility.