September 23, 2026 at 12:00:00 AM UTC
atperson jetstream full-speed ingestion
atpersonjetstreamatprotoperformancec
Live Jetstream ingestion ran at a fraction of the achievable rate: a 200k-event catch-up session observed 5 frames in 60 seconds. Three transport changes close that gap.
Changes
- Drain inside
fetch_batch. AWOULD_BLOCKfrom Wolfram while connected and idle no longer returns to the caller — the batch keeps polling until the feed goes quiet, the budget is spent, or reconnect backoff is advertised. The CLI loop drops its 25ms inter-batch sleep and only sleeps on advertised backoff. - Cross-batch
max-eventsaccounting. The CLI loop appliedmax-eventsper batch, so an event-bounded run never terminated. Frames are now counted cumulatively across batches. - Dictionary-compressed binary frames, on by default. The official Jetstream zstd dictionary is fetched via the public
network.bsky.jetstream.getZstdDictionaryquery on the endpoint's service host and passed to Wolfram's connect options.ATPERSON_JETSTREAM_COMPRESS=0opts out; fetch failure or a build without libzstd degrades to uncompressed JSON instead of failing the run. --kindsfilter. Operator-selected event-kind predicate for the v2subscribeEventsendpoint (--kinds <file>orATPERSON_JETSTREAM_KINDS_FILE), validated against the four Jetstream v2 kinds, max four unique values. Default remains no predicate, preserving#sync/#identity/#accountprotocol evidence.
Verification
Both build configs green: core 54/54, network 60/60. New offline tests cover the kinds loader (parse/dedup, unknown kind, limit) and client constructor validation (kinds > 4 rejects; dictionary without libzstd rejects, with it succeeds).
PR: ewanc26/atperson#138
all entries