September 15, 2026 at 12:00:00 AM UTC

malachite dedup batching — applyWrites deletes instead of one deleteRecord per record

pkgsmalachitecroft-click-coretealatprototypescript

pkgs

croft-click-core: batched dedup deletions via applyWrites

removeDuplicateRecords previously deleted one record at a time — a dedicated com.atproto.repo.deleteRecord call per duplicate with a 100ms pacing delay between each. Running the plan against the 3,395 duplicates from the September audit meant 3,395 HTTP round-trips plus ~5.6 minutes of artificial delay.

The executor now chunks deletions into com.atproto.repo.applyWrites#delete writes, 200 per call — the PDS hard limit (InvalidRequestError beyond that). Same pattern the publisher already uses for creates: transient network errors retry with exponential backoff (3 attempts, 1s/2s/4s).

applyWrites is atomic — one bad rkey fails the whole batch. On batch failure after retries, the executor falls back to per-record deleteRecord calls so salvageable deletions still go through, preserving the previous "remove what we can" behaviour.

DELETE costs 1 rate-limit point per record either way (5,000/hour, 35,000/day), so batching saves HTTP requests, not quota. swapCommit is left unset — nothing else writes during a dedup run, so batches stay independent.

Both the CLI (malachite deduplicate) and malachite-web delete through this shared executor, so both get the speedup. Full 252-test suite passes.


all entries