Many of our customers are bursty and latency-sensitive at the same time. But when we’re hit with thousands of documents in seconds, we have to watch for the noisy-neighbor problem: a small set of customers flooding the system and spiking everyone’s latency.
Our first instinct was full tenant isolation. We ran three worker pools: shared, per-tenant overflow, and dedicated (we’ll set dedicated aside for this post). Each customer was allocated fixed capacity in the warm shared pool. For every submitted job, we’d send it to the shared pool’s queue if that org had capacity available, and to the cold per-tenant queue if it didn’t, so a job’s fate was decided at submission time. This protected the shared pool, but it also dumped jobs onto a potentially cold queue with highly variable start times. And by the time a per-tenant pool warmed up, the burst was usually over, leaving idle workers to spin back down.
The deeper we looked, the clearer it got. Full isolation was both increasing our customers’ latency and wasting compute, and our existing orchestration primitives weren’t built to express the scheduling policies we actually wanted.
What we actually wanted
Fixed per-customer capacity is the wrong primitive for an autoscaling fleet. If the fleet can grow, then “how much should this customer be allowed to run right now?” isn’t a constant. It’s a function of the live fleet and the customer’s recent behavior. What we wanted, per customer, was simple to say out loud:
- a guaranteed floor of capacity they can always count on,
- a ceiling that rises when the fleet has room to spare and tightens when it’s saturated,
- and the property that when we’re compute-constrained, the spiky customers get throttled first while steady ones are left alone.
So we adopted a small task queue we could make our own
We looked at heavier workflow systems, including Temporal. They’re all very powerful, but were difficult for us to adopt: they’re complex systems hard to maintain for on-prem deployments and often add more overhead than we wanted.
Instead we adopted streaq, a small open source async task queue built on Redis. We picked it precisely because it was simple: a clean core small enough that we were confident we could shape it into what we needed. So we vendored it and modified it heavily, while holding a hard line: streaq stays a clean, standalone library with no business logic in it. All of our scheduling and traffic-management policy lives in our code, on top of the primitives the library gives us. That separation has helped us keep our code clean, maintainable, and highly performant.
What streaq is, and everything we changed
streaq is an async task queue built on Redis: typed Python APIs, workers that live inside our own processes, and a surface area small enough to understand end to end. The async, event-loop model gives us very high concurrency and suits our IO-heavy orchestration, but it has one sharp edge: a single blocking call can stall everything. So we never run CPU-bound parse compute on the loop. Heavy compute goes to a separate, kill-able subprocess pool while the loop stays free to coordinate.
On top of that core we have made a lot of changes: new capabilities plus a long tail of production hardening. The highlights, by area:
Concurrency & traffic control
- The per-key concurrency throttle, with fair per-priority waiter queues that never barge or busy-poll.
- Multi-key admission in a single round trip, holding earlier slots when a later key is full so we never livelock.
- Per-job dynamic limits, so a customer’s ceiling can move without redeploying workers.
- Generic resource backpressure: memory, CPU, and pressure-stall sensors scale each worker’s prefetch up and down on their own.
- Pluggable priority-admission gating, the basis for the batch lane.
Moving large data around
- Transparent large-payload offload: multi-megabyte arguments and results go to object storage (or a shared volume) and travel through Redis as a small reference, so they never get inlined and stall the loop, plus lazy resolution and rolling-deploy-safe handling.
Reliability & hardening
- Widened, jittered, bounded work-reclaim sweeps so one slow tick can’t trigger false re-delivery.
- An idempotency guard so a re-delivered task never repeats its side effects, plus safe handling of unknown arguments and empty results during rolling deploys, and cleanup of orphaned schedules.
Observability & autoscaling
- Spans and lifecycle events on every hot path, and a cached metrics endpoint that drives Kubernetes autoscaling on worker-slot utilization.
Throttles as product logic
The core building block is a concurrency limit attached to a key: allow at most N jobs to run concurrently for that key, park the rest in a fair queue, and wake them the instant a slot frees up.
Fair-queue throttling in motion: each tenant’s work parks in its own queue and drains into one shared pool as slots free, with a separate lane for batch traffic. Press play or drag the scrubber.
This primitive has allowed us to express complex fair queueing policies. Here are some policies we’ve implemented:
- A floor and a flexible ceiling, per org. Every customer gets guaranteed concurrency, plus a ceiling that grows as their traffic becomes sustained and as the fleet scales up. We answer “how much should this customer run right now?” against the live fleet, ensuring our policies scale with our infrastructure.
- Ceilings that compress under pressure. When the shared pool saturates, every customer’s ceiling tightens by the same proportion, so we shed load fairly instead of letting the fastest enqueuer win. When the pool is idle, bursty customers’ capacity expands well past their floor to maximize throughput and resource utilization.
- One key, or many. A key can be an org, a job, or both, so we can separately cap how much one customer runs at once and how far any single document is allowed to fan out.
- Downstream sub-tenant throttling. A customer can hand us a tenant id for their own end-customer, and we throttle on it as a second key, so one noisy downstream tenant can’t degrade the rest of that account. The old isolation model had nowhere to put this: it could protect our customers from each other, but not our customers’ customers.
Inside the throttle primitive
Admission runs as a single atomic step in Redis: check how many jobs hold the key, admit if there’s room, otherwise append this job to a fair waiter list. When a job finishes, the same atomic step wakes the next waiters in line (highest priority first, first-in-first-out within a priority) and hands each its slot directly. A task only ever waits when we want it to: because admission and wake are one atomic operation, there’s no window where a slot sits free while a job sits parked, and no polling interval adding latency to a waiting job.
A single job can need several keys at once (its org and its tenant, say). We acquire them together in one round trip and hold what we’ve already got if one is full, so we never deadlock or livelock.
All of this sits on the hot path, so it has to be cheap. The decision is one atomic Redis call; the live fleet size is cached and never fetched inline; the per-customer rate signal is a single fixed-size read. Deciding where and when a job runs can’t be allowed to become the bottleneck itself.
How the floor and ceiling move
The ceiling is computed per job from three things: the customer’s recent submission rate, the average job duration, and the live size of the fleet. The intuition is Little’s Law: a customer sustaining R submissions/min at D seconds each needs about R×D/60 jobs in flight, so that’s roughly what we grant, never below their floor:
- Earned base:
base = max(floor, ⌈ R × D / 60 ⌉), where R = recent submissions/min and D = avg job duration in seconds - Ceiling:
ceiling = base + max(burst_floor, base × burst_pct) - Under pressure:
limit = ceiling × clamp(pool_target ÷ in_flight, f_min, f_max)
When total in-flight work exceeds what the fleet can serve, that last factor drops below 1 and every customer’s ceiling tightens by the same proportion, so pressure is shared fairly. When there’s spare capacity, the factor lets bursty customers expand past their floor to use it.
Floors are configurable per customer, and larger customers can be granted dedicated capacity that’s added on top, so “give this account more headroom” is a config change, not a private island.
The customer-facing version of these controls is documented at docs.reducto.ai/reference/throttling.
Then we made it smooth — and cheap
Owning the scheduler let us go after cost and latency from both ends.
A batch lane for work that isn’t urgent. Customers can opt specific jobs into a batch queue: a firm 12-hour SLA in exchange for a discount. Batch work only runs on capacity that’s already warm and idle, and it never triggers a scale-up on its own. That’s a real win-win. It’s cheaper for latency-insensitive workloads, and it means we no longer provision for batch spikes as if they were interactive traffic.
Scaling that anticipates the day. Our traffic has a strong daily and weekly shape, so we warm capacity ahead of the curve using weeks of arrival data, and we scale on worker-slot utilization as a leading indicator before the queue starts backing up, not after.
A hard performance push. A lot of our latency under load turned out to be contention: lock, CPU, and network contention that only showed up when many jobs ran at once. Owning the scheduling layer, with detailed tracing through the whole pipeline, finally let us see it, stress it, and kill it. We also made the genuinely heavy work much faster: document rendering and other compute-intensive steps, plus a large inference speedup. Better visibility plus faster code let us right-size aggressively, cutting warm provisioning several-fold while latency kept dropping.
What it bought us
We didn’t get here with one dramatic rewrite. We made the scheduler more precise, then shifted traffic onto it carefully enough that each new bottleneck became visible and fixable: shadow traffic first, then real traffic in small percentage steps, watching the metrics, fixing sharp edges (including one full rollback), and ramping back up until every region was at 100%.
Over the rollout, platform-wide P50 parse latency dropped roughly 3.9x and P99 roughly 11x. For one latency-sensitive customer, P50 fell about 3.3x and their worst-case latency about 10x.
This is the kind of platform work that keeps pulling us in at Reducto: take an annoying product constraint, follow it all the way down into the infrastructure, and then own the layer where the right abstraction should have existed.
If this is the kind of problem you want to own, come build platform at Reducto.