VinciPhotoHow we find similar photos on device
By VinciPhoto engineering. Updated .
Summary: VinciPhoto embeds each photo once with Apple’s MobileCLIP image encoder, uses that vector for both search and grouping, links photos from the same day that are close together, and treats each connected set as a group. All of it runs on the device.
The problem
People rarely have many exact copies of a photo. They have many takes of it. To group those takes, you have to tell “same moment, slightly different frame” apart from “similar-looking but unrelated”. It also has to run over a whole library, on a phone, without overheating it.
One embedding for two features
Our first version used a MobileNetV3 model for grouping. When we added search by description, which needs MobileCLIP, each photo went through two fetches, two decodes and two model runs, and we stored two sets of vectors.
We measured where the time went. Decoding the image took about 76% of the per-photo cost, so doing it twice was the obvious waste. Moving grouping onto the vectors the search index already holds changed the per-photo work:
| Per photo | Two models | One model |
|---|---|---|
| Library fetches | 2 | 1 |
| Decodes | 2 | 1 |
| Model runs | 2 | 1 |
| Vector storage | 6,144 bytes | 2,048 bytes |
It also let us remove a dependency and a whole model from the app. The clustering code doesn’t generate vectors any more. It’s handed a source that returns a cached vector, or generates and stores one on a miss, so a scan where every vector is cached never has to read the original images.
From vectors to groups
- Bucket by day. Near-identical shots almost always come from the same day, and limiting comparisons to a day keeps the work small and stops look-alike scenes from different occasions from joining up.
- Build a sparse graph. Within a day, we compare vectors by cosine similarity and connect each photo to its closest neighbors above a similarity cutoff.
- Take connected components. Each connected set of photos becomes a group.
The risk with connected components is chaining: A resembles B, B resembles C, and you end up with one large group where A and C have little in common. Limiting neighbors and setting a strict cutoff keeps this in check.
How we checked the switch
Changing models changes every group, so we compared the two on the same test library before removing the old one. The new setup produced 452 groups against 475, or about 95% of the old volume. Its largest group held 24 photos against 26, so chaining didn’t get worse. We reviewed the 18 cases where groups merged by eye and accepted them. This is one library, so it doesn’t prove the result in general, but it was enough to act on. We ran the comparison before deleting the old store, because the old vectors were the only baseline we had.
Picking the best shot
Once a group exists, a quality pass scores each photo on signals such as sharpness, faces and exposure, and the app shows the reason for the pick. We originally decoded each full-resolution original a second time here. Now we ask the system for a smaller rendering at the size scoring needs, which removes that extra decode.
Not cooking the phone
A first scan over a large library can make a phone hot. Since heat depends on power over time, the foreground scan now reads the device’s thermal state. It slows down as the device warms up, stops if it reaches the critical state, and resumes after it cools. This changes when the work runs and not what it computes, so results stay the same.
What’s left out
We don’t publish our similarity cutoff or scoring details. They’re tuned to our data and change between releases. If you’re building something similar, measure your own: review the differences between settings by eye, and watch the size of your largest group as an early warning for chaining.