Meta AI Releases Segment Anything Model 3 (SAM 3) for Promptable Concept Segmentation in Images and Videos
How do you reliably discover, phase and observe each occasion of any idea throughout giant picture and video collections utilizing easy prompts? Meta AI Team has simply launched Meta Segment Anything Model 3, or SAM 3, an open-sourced unified basis mannequin for promptable segmentation in pictures and movies that operates straight on visible ideas as an alternative of solely pixels. It detects, segments and tracks objects from each textual content prompts and visible prompts akin to factors, bins and masks. Compared with SAM 2, SAM 3 can exhaustively discover all situations of an open vocabulary idea, for instance each ‘crimson baseball cap’ in an extended video, utilizing a single mannequin.
From Visual Prompts to Promptable Concept Segmentation
Earlier SAM fashions targeted on interactive segmentation. A person clicked or drew a field and the mannequin produced a single masks. That workflow didn’t scale to duties the place a system should discover all situations of an idea throughout giant picture or video collections. SAM 3 formalizes Promptable Concept Segmentation (PCS), which takes idea prompts and returns occasion masks and secure identities for each matching object in pictures and movies.
Concept prompts mix brief noun phrases with visible exemplars. The mannequin helps detailed phrases akin to ‘yellow college bus’ or ‘participant in crimson’ and may use exemplar crops as optimistic or damaging examples. Text prompts describe the idea, whereas exemplar crops assist disambiguate nice grained visible variations. SAM 3 can be used as a imaginative and prescient software inside multimodal giant language fashions that generate longer referring expressions and then name SAM 3 with distilled idea prompts.
Architecture, Presence Token and Tracking Design
The SAM 3 mannequin has 848M parameters and consists of a detector and a tracker that share a single imaginative and prescient encoder. The detector is a DETR primarily based structure that’s conditioned on three inputs, textual content prompts, geometric prompts and picture exemplars. This separates the core picture illustration from the prompting interfaces and lets the identical spine serve many segmentation duties.
A key change in SAM 3 is the presence token. This element predicts whether or not every candidate field or masks truly corresponds to the requested idea. It is very essential when the textual content prompts describe associated entities, akin to ‘a participant in white’ and ‘a participant in crimson’. The presence token reduces confusion between such prompts and improves open vocabulary precision. Recognition, that means classifying a candidate because the idea, is decoupled from localization, that means predicting the field and masks form.
For video, SAM 3 reuses the transformer encoder decoder tracker from SAM 2, however connects it tightly to the brand new detector. The tracker propagates occasion identities throughout frames and helps interactive refinement. The decoupled detector and tracker design minimizes job interference, scales cleanly with extra information and ideas, and nonetheless exposes an interactive interface just like earlier Segment Anything fashions for level primarily based refinement.
SA-Co Dataset and Benchmark Suite
To prepare and consider Promptable Concept Segmentation (PCS), Meta introduces the SA-Co household of datasets and benchmarks. The SA-Co benchmark incorporates 270K distinctive ideas, which is greater than 50 occasions the variety of ideas in earlier open vocabulary segmentation benchmarks. Every picture or video is paired with noun phrases and dense occasion masks for all objects that match every phrase, together with damaging prompts the place no objects ought to match.
The related information engine has robotically annotated greater than 4M distinctive ideas, which makes SA-Co the most important prime quality open vocabulary segmentation corpus as talked about by Meta. The engine combines giant ontologies with automated checks and helps exhausting damaging mining, for instance phrases which can be visually comparable however semantically distinct. This scale is important for studying a mannequin that may reply robustly to various textual content prompts in actual world scenes.
Image and Video Performance
On the SA-Co picture benchmarks, SAM 3 reaches between 75 % and 80 % of human efficiency measured with the cgF1 metric. Competing techniques akin to OWLv2, DINO-X and Gemini 2.5 lag considerably behind. For instance, on SA-Co Gold field detection, SAM 3 reviews cgF1 of 55.7, whereas OWLv2 reaches 24.5, DINO-X reaches 22.5 and Gemini 2.5 reaches 14.4. This exhibits {that a} single unified mannequin can outperform specialised detectors on open vocabulary segmentation.
In movies, SAM 3 is evaluated on SA-V, YT-Temporal 1B, SmartGlasses, LVVIS and BURST. On SA-V check it reaches 30.3 cgF1 and 58.0 pHOTA. On YT-Temporal 1B check it reaches 50.8 cgF1 and 69.9 pHOTA. On SmartGlasses check it reaches 36.4 cgF1 and 63.6 pHOTA, whereas on LVVIS and BURST it reaches 36.3 mAP and 44.5 HOTA respectively. These outcomes verify {that a} single structure can deal with each picture PCS and lengthy horizon video monitoring.
SAM 3 as a Data-Centric Benchmarking Opportunity for Annotation Platforms
For data-centric platforms like Encord, SAM 3 is a pure subsequent step after their present integrations of SAM and SAM 2 for auto-labeling and video monitoring, which already let clients auto-annotate greater than 90 % of pictures with excessive masks accuracy utilizing basis fashions inside Encord’s QA pushed workflows. Similar platforms akin to CVAT, SuperAnnotate and Picsellia are standardizing on Segment Anything type fashions for zero shot labeling, mannequin in the loop annotation and MLOps pipelines. SAM 3’s promptable idea segmentation and unified picture video monitoring create clear editorial and benchmarking alternatives right here, for instance, quantifying label price reductions and high quality positive aspects when Encord like stacks transfer from SAM 2 to SAM 3 in dense video datasets or multimodal settings.
Key Takeaways
- SAM 3 unifies picture and video segmentation right into a single 848M parameter basis mannequin that helps textual content prompts, exemplars, factors and bins for Promptable Concept Segmentation.
- The SA-Co information engine and benchmark introduce about 270K evaluated ideas and over 4M robotically annotated ideas, making SAM 3’s coaching and analysis stack one of many largest open vocabulary segmentation sources out there.
- SAM 3 considerably outperforms prior open vocabulary techniques, reaching round 75 to 80 % of human cgF1 on SA Co and greater than doubling OWLv2 and DINO-X on key SA-Co Gold detection metrics.
- The structure decouples a DETR primarily based detector from a SAM 2 type video tracker with a presence head, enabling secure occasion monitoring throughout lengthy movies whereas conserving interactive SAM type refinement.
Editorial Comments
SAM 3 advances Segment Anything from Promptable Visual Segmentation to Promptable Concept Segmentation in a single 848M parameter mannequin that unifies picture and video. It leverages the SA-Co benchmark with about 270K evaluated ideas and over 4M robotically annotated ideas to approximate 75 to 80 % of human efficiency on cgF1. The decoupled DETR primarily based detector and SAM 2 type tracker with a presence head makes SAM 3 a sensible imaginative and prescient basis mannequin for brokers and merchandise. Overall, SAM 3 is now a reference level for open vocabulary segmentation at manufacturing scale.
Check out the Paper, Repo and Model Weights. Feel free to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Also, be happy to observe us on Twitter and don’t neglect to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The publish Meta AI Releases Segment Anything Model 3 (SAM 3) for Promptable Concept Segmentation in Images and Videos appeared first on MarkTechPost.
