|

Xiaomi’s MiLM Plus Releases PROVE: Perception-Aligned Object Removal Metrics RC-S and RC-T With a Real-World Video Benchmark

ByRicardo

Object elimination fashions have improved sooner than the metrics used to guage them. Diffusion erasers now reconstruct shadows, reflections and occluded construction convincingly, but PSNR, SSIM, LPIPS, ReMOVE and CFD incessantly rank their outputs the unsuitable manner. The root trigger is structural: erasure is an ill-posed, one-to-many activity, so no single floor fact exists to check in opposition to. A staff from MiLM Plus, Xiaomi Inc. has launched PROVE (Perceptual RemOVal cohErence), accepted at ACM MM 2026, to shut that hole. PROVE pairs two perception-aligned metrics — RC-S for spatial coherence and RC-T for temporal consistency — with PROVE-Bench, a two-tier real-world video benchmark. Both metrics rating the edited area regionally, utilizing sliding-window Maximum Mean Discrepancy over DINOv2 options, and neither requires a reference video.

Is it deployable?

Yes — as an analysis harness, not a product function. PROVE ships as an Apache 2.0 PyTorch repo with one CLI entry level (run_prove_metrics.py). It wants Python 3.10+, PyTorch 2.6+, Transformers 4.51+, and DINOv2-giant weights. Masks are obligatory; white pixels mark the eliminated object.

  • Company stage: Any staff with one GPU and a masks pipeline. RC-S runs at 134.6 ms/body on a single RTX 4090, so nightly CI gates are sensible for startups; enterprises with giant enhancing catalogs acquire most, since no paired floor fact is required.
  • Industries: Smartphone gallery and digicam apps, short-video enhancing, e-commerce catalog cleanup, promoting and inventory media, movie/VFX post-production, actual property imagery, and privateness redaction in mapping archives.
  • Applications: Model bake-offs, CI gates on RC-S/RC-T, tuning inference steps or quantization with out a reference video, filtering coaching knowledge, and shaping reward fashions.
  • Who buys in: CV researchers and utilized AI/ML engineers on enhancing groups first; MLOps engineers wiring the rating into CI second; product managers operating vendor comparisons third.
  • Where it doesn’t match: Real-time on-device scoring, and uncomfortable side effects resembling giant shadows or reflections extending past the cropped analysis area.

Three documented failure modes

Object elimination is ill-posed and one-to-many: many restorations are believable for a similar gap, so no distinctive floor fact exists.

  • Full-reference bias: PSNR, SSIM and LPIPS assume point-to-point correspondence, so that they reward copy-paste over real erasure. Residual shadows occupy few pixels and incur virtually no penalty. Worse, slicing diffusion inference steps improves PSNR and SSIM whereas visible high quality collapses — regression to the imply.
  • No-reference blind spots: On ROSE-Bench, the authors progressively blur the masked area. Neither ReMOVE nor CFD degrades; each ultimately surpass their unblurred baselines. CFD’s SAM-based hallucination time period additionally misfires: a legitimately restored bicycle seat is flagged as a hallucination, scoring worse than the unedited enter.
  • Temporal insensitivity: Temporal Consistency and Temporal Flickering function on full-frame options. Since the edited space is a small fraction of the body, injected Random Drop and Random Replace corruptions on DAVIS barely transfer them, and typically transfer them the unsuitable manner.

RC-S and RC-T

Both share one concept: native distribution matching in deep function house as an alternative of worldwide aggregation.

RC-S (spatial): Connected-component evaluation splits the masks into unbiased targets. Each bounding field is expanded by one third of its aspect size, the crop goes to DINOv2, and the masks is downsampled to function decision. A w×w window slides throughout the function map, computing squared Maximum Mean Discrepancy with a Gaussian RBF kernel between masked and native background options. Scores are averaged per goal, then throughout targets.

RC-T (temporal): Adjacent frames are cropped collectively below the union of their masks to keep away from misalignment, then MMD is computed solely contained in the intersection — the area restored in each frames. An ablation exhibits that eradicating the crop makes RC-T blind to injected corruption.

Results

Against human rankings from 20 contributors aggregated by Borda depend, RC-S reaches 0.59 common Kendall’s τ and 0.66 Spearman’s ρ, versus 0.26/0.29 for ReMOVE and 0.16/0.18 for CFD. It ranks first on 5 of six benchmarks. Background-only FR variants correlate negatively on most datasets. On RORD-Val, RC-S prefers the clear picture over blurred and region-swapped variants in 100% of instances; ReMOVE manages 60.06% and CFD 49.27% below blur. RC-T responds monotonically to rising corruption the place TC and TF don’t.

Ablations isolate every contribution: DINOv2 (0.59 avg τ) beats DINOv3 (0.51) and SAM (0.44); dropping the sliding window prices 0.11; swapping MMD for cosine prices 0.07. RC-S can also be the most cost effective spatial metric examined, 13.7× sooner than CFD.

PROVE-Bench

PROVE-M holds 80 paired movies: tripod-captured enter and target-free footage shot inside two minutes, SAM3 masks refined body by body, three-stage high quality management, then Ken Burns-style movement augmentation utilized synchronously to the triplet. Each clip is 81 frames at 1080p. PROVE-H provides 100 laborious movies with out floor fact — crowds, flowing water, flames, textured terrain, multi-puddle reflections, quick movement — utilizing unrefined SAM3 masks on function.

On the general public leaderboard, SVOR (1.3B) leads mixed RC-S at 0.5197; EffectErase leads RC-T at 0.2525.