Trackshot

Trackshot is a concept for local-first semantic search on macOS: it would make a screenshot folder searchable by what the images show and what they say.

It's for software engineers, designers, and researchers who take dozens of screenshots a day and can't find them later.

Nothing is built or benchmarked yet, so every number below is a design target or an estimate.

The plan is a background daemon that watches screenshot folders, turns each new image into a SigLIP embedding on the Apple Neural Engine (ANE), and reads its text with Apple Vision OCR.

A hybrid query would merge vector similarity with BM25 text ranking over a USearch index, entirely offline.

I'm designing this because screenshots are the fastest way to capture context and the slowest to get back.

I think people remember a screenshot in one of two ways: how it looked, or one exact string on it.

I want one search box that handles both, without sending my screenshots to someone else's server.

Semantic search for screenshots: find them by what they show

Trackshot would run two indexes side by side, a vector index for what an image looks like and a text index for what it says.

Every query hits both and merges the results, so a description and an exact string work equally well.

Why bother? Because a screenshot vanishes into ~/Desktop or ~/Downloads under a name like Screenshot 2026-01-15 at 11.42.08.png, and the contents are the part worth finding: UI inspiration, stack traces, architecture diagrams, chat snippets.

Can't Spotlight do this? Partly. macOS Spotlight leans on filenames and recognized text, so a query like "that minimalist dark mode dashboard with a circular progress gauge" is a weak fit.

OCR alone has the same blind spot, since it only sees verbatim strings, never a layout, a color scheme, or a diagram.

The two streams exist because embeddings and OCR fail in opposite places. Embeddings handle broad themes but struggle with UUIDs, error numbers, and hex color codes. Plain OCR text is the reverse.

The cost is two stores, USearch for the vectors and SQLite FTS5 for the text.

That's more moving parts than a filename search, and I'd carry them rather than ship a search that only works when I remember the right words.

Can I download it? No. It's a concept, so there's nothing to install yet.

When a new screenshot is saved, Trackshot would process it through an asynchronous, two-stage pipeline:

┌─────────────────────────────────────────────────────────────┐
│                 macOS Filesystem (FSEvents)                 │
│         ~/Desktop/Screenshot-2026-01-15-at-14.22.png         │
└──────────────────────────────┬──────────────────────────────┘
                               │ Non-blocking Event
┌──────────────────────────────▼──────────────────────────────┐
│                  Trackshot Ingestion Daemon                 │
│   ┌─────────────────────────────────────────────────────┐   │
│   │   Perceptual Hash (pHash) Deduplication Check       │   │
│   └──────────────────────────┬──────────────────────────┘   │
│                              │                              │
│         ┌────────────────────┴────────────────────┐         │
│         │                                         │         │
│         ▼                                         ▼         │
│  [ Stream A: Dense Vision ]        [ Stream B: Sparse Text ]│
│  - Quantized UForm / SigLIP        - Apple Vision OCR Engine│
│  - Float16 Vector (dense)          - Full Text Tokens & BBox│
│  - Apple Neural Engine (ANE)       - Inverted Keyword Index │
│         │                                         │         │
│         └────────────────────┬────────────────────┘         │
│                              │                              │
│   ┌──────────────────────────▼──────────────────────────┐   │
│   │         Hybrid Index Storage & Memory Map           │   │
│   │   - USearch Native Vector Index (.usearch)          │   │
│   │   - SQLite FTS5 Metadata Store                      │   │
│   └─────────────────────────────────────────────────────┘   │
└──────────────────────────────┬──────────────────────────────┘
                               │
┌──────────────────────────────▼──────────────────────────────┐
│                    Search & Retrieval Flow                  │
│       User Query: "kubernetes pod out of memory trace"       │
│  → Dense Cosine Sim + Sparse BM25 → Reciprocal Rank Fusion  │
│  → Result displayed with bounding box (<10ms target)        │
└─────────────────────────────────────────────────────────────┘

Fuse vector and OCR rankings instead of picking a winner

I'd rank every screenshot twice, once by vector similarity and once by BM25 over the OCR text, then merge the two lists by rank.

Obvious version: Pure vector search, because embeddings already understand images. It falls over on the queries engineers actually type, like E0502 compiler error, where one exact token matters more than the overall look.

The merge works like two friends helping you find an old photo. One remembers what it looked like, the other remembers a word printed on it.

Each hands you a ranked shortlist, and you can't compare their confidence (one says "nine out of ten," the other says "probably").

So you only look at where each photo sits on each list.

A photo near the top of either list scores well. A photo near the top of both scores best. That's Reciprocal Rank Fusion (RRF), and it only looks at rank.

Trackshot would fuse the dense ranking (cosine similarity over vision vectors) with the sparse one (BM25 over Apple Vision OCR tokens).

The starting weights are 0.65 dense and 0.35 sparse, to be tuned once there's a real library to tune against.

// Sketch: hybrid ranking with Reciprocal Rank Fusion (RRF)
func computeHybridRank(denseRank: Int, sparseRank: Int, k: Double = 60.0) -> Double {
    let denseScore = 1.0 / (k + Double(denseRank))
    let sparseScore = 1.0 / (k + Double(sparseRank))
    return (0.65 * denseScore) + (0.35 * sparseScore)
}

The payoff should show at both ends.

E0502 compiler error would anchor on exact OCR tokens, and clean glassmorphism card layout should still return relevant designs when no matching text exists.

The cost is a second ranking to maintain and weights I haven't validated.

My test: "Does a query with no matching text still return something useful?"

Skip the vector database and map a file instead

I'd use USearch, a C++ vector search library with Swift bindings, and memory-map its index file instead of running a vector database inside the app.

Obvious version: Reach for Chroma or Pinecone. Chroma would mean bundling a Python runtime or running a local server next to a desktop app. Pinecone is hosted, so every query would leave the machine, which defeats the point.

USearch builds Hierarchical Navigable Small World (HNSW) graphs and ships SIMD-optimized distance metrics, so comparisons use the host CPU's vector instructions. Mapping the saved .usearch file avoids a duplicate copy on the heap.

I'm sizing for a library of about 25,000 screenshots, with headroom to 50,000 vectors.

Cosine similarity across that many is expected to resolve well under a millisecond (index lookup only, excluding OCR and ranking). The end-to-end target is still under 10 ms.

The cost is tying the design to a single library. I haven't run it against a real screenshot library yet.

Index quietly, and keep everything on the machine

Inference would stay on the device, and a three-step throttle would keep indexing from competing with whatever the user is doing.

The alternatives are both bad: external vision APIs would expose proprietary codebases, cloud console screens, customer data, and tokens to third-party servers, and Python-heavy vision pipelines can eat gigabytes of RAM and spin up the fans.

The throttle runs in this order:

1. FSEvents file monitoring: The daemon would subscribe to low-overhead macOS filesystem notifications instead of polling directory trees.
2. Perceptual hashing: A fast 64-bit perceptual hash would discard duplicate captures and near-identical window frames before any neural inference runs.
3. Neural Engine dispatch: Quantized INT8 weights would be scheduled on the ANE, which keeps inference off the CPU and GPU cores the user's IDE and compilers need.

Here's the latency budget I'm working to. These are rounded estimates, and the OCR figure is the one I'm least sure of:

Indexing pipeline latency budget (rounded design estimates):
├── Image load and downsample: about 2 ms
├── Perceptual hash check:     about 0.2 ms
├── Vision model inference:    about 14 ms (ANE)
├── OCR text extraction:       about 6 ms (least certain)
└── USearch index insertion:   about 0.3 ms
───────────────────────────────────
Estimated total:               about 22.5 ms per screenshot

The privacy side is a short offline-first contract:

• No remote network calls: The binary is designed to make no outgoing network connections.
• Secret flagging: Screenshots with detected SSH keys or private certs would get an encrypted metadata tag.
• One data folder: All indexes and thumbnails would live in ~/.trackshot/, so backup, encryption, or deletion is a single-folder job.

The cost is that the design has to earn its speed on the user's own laptop, with no cloud to fall back on.

My test: "Would I be comfortable running this with the Wi-Fi off and a compile running?"

What Trackshot won't do, and what I haven't measured

A concept is easy to flatter, so here's where I expect it to fall short, or where I simply don't know.

• It won't run beyond macOS. The design leans on FSEvents, Apple Vision OCR, and the Neural Engine, so it's a macOS, Apple Silicon design.
• It won't find text the OCR can't read. Exact-string matches depend on the OCR stream reading the text in the first place.
• It won't scrub secrets. It would only flag screenshots holding detected keys or certs, and flagging isn't removing.
• It won't phone a cloud model when the local one misses. That's deliberate, and I don't know yet how the local model compares.
• It won't hit its numbers because I wrote them down. Nothing here has been measured, and the weights, the OCR time, and the memory target could all move.

These are the targets the design is built around:

If I had to protect one decision, it's pairing vectors with OCR tokens and fusing them by rank.

My second bet is that on-device should win on speed and privacy for personal search, because nothing leaves the machine and there's no network round trip.

The next step is to build the indexing daemon and test these budgets against a real screenshot library.

Next time you can't find a screenshot, notice whether you remember how it looked or what it said. That's the two indexes. For a related project, see Claudey, a local proxy that absorbs provider outages so a coding agent never has to.