Distributed throttle
Share one rate budget across workers by pointing the throttle at a shared store.
Share one rate limit across every worker or server by pointing the throttle at a
shared store. By default a stitch keeps its throttle counters in the in-memory
store, so the rate budget is per process: with pool: 'host' every stitch in
that process already pools into one budget per host (see
Throttle), but the in-memory store keeps that
pooling within a single process — each worker gets its own. Point the throttle at
a shared store and one budget spans every worker that uses it.
Example
import { } from 'stitchapi';
import type { StitchStore } from 'stitchapi';
// A shared store — e.g. @stitchapi/redis's redisStore() — see The pluggable store.
declare const : StitchStore;
const = ({
: 'https://api.example.com',
: '/search',
: { : '10/s', : 'host' },
: , // the budget now spans all workers using this store
});Options
throttle sets the limits — rate (e.g. '10/s'), concurrency, and
pool — while store decides where the counters live. The default in-memory
store is single-process, so each worker enforces 10/s on its own and the fleet
runs at 10/s × workers. A shared store holds the rate counter every worker
increments, so the whole fleet shares one 10/s budget — and grants are
even-spaced (about one every 100ms, i.e. window / limit), the same pacing
the in-process limiter uses, so attaching a store doesn't turn a smooth limit into
boundary bursts.
concurrency goes fleet-wide with it: concurrency: 5 over a shared store means
five calls in flight across every worker, not five each. Nothing extra to
configure — a store that supports it just does it.
A fleet-wide slot is a lease, so it has an expiry. A worker that crashes
mid-call never gives its slot back, and throttle.lease (default 30s) is how
long the fleet waits before reclaiming it. Size it above your slowest call:
too long strands a dead worker's slots, too short and a call still running
loses its slot to the next caller, briefly putting the fleet over the cap.
Streaming never holds a slot at all, so this only bounds buffered calls — a
lease comfortably above timeout.total can't be outlived.
Two behaviours differ from the in-process limiter, inherently: a blocked call
polls rather than being handed the slot (no worker can wake another), so
strict FIFO fairness is gone; and a release is fire-and-forget, because the
expiry already covers a lost one. Backed by memoryStore,
@stitchapi/redis and @stitchapi/deno-kv; a store without leases keeps
per-process concurrency.
How exact the fleet-wide pacing is depends on your store. A store with a
pacing cell — memoryStore, @stitchapi/redis, @stitchapi/deno-kv — paces
the whole fleet on one continuous cursor, so every worker draws from one
schedule no matter when it starts.
A store without one (@stitchapi/cloudflare-kv, which is eventually
consistent and has no atomic read-compute-write to build a cell from) falls
back to a shared counter plus a per-worker cursor. That holds the budget at a
window boundary and keeps each worker's own pacing exact, but a fleet that
starts mid-window runs at up to rate × workers until the schedule catches
up with the clock. Nothing to configure either way — the throttle uses the
cell when the store has it.
pool decides what shares a budget: 'stitch' (the default) gives each stitch
its own counter keyed by name, while 'host' keys by origin so every stitch
hitting api.example.com draws from one budget. Workers share a budget only when
they point at the same store and resolve to the same key — same pool, same
stitch name or host.
The same shared store also backs auth sessions, so one store gives you both a fleet-wide rate budget and a shared session — see Shared sessions.