Congestive collapse: why our API got slower the harder we pushed it

Every response in our load test came back 200 OK. The error rate was zero. And at 192 concurrent callers the API was doing a third of the work it did at four.

This is the story of finding that, fixing it on ezychat.io, the multi-tenant AI platform I build and run, and writing down exactly where the fix stops working.

Measure at production shape, or don’t bother

The API runs on AWS ECS Fargate in tasks of 0.25 vCPU and 512 MiB. That number matters more than anything else in this post.

My first load-test runs used the plain development container. Three identical runs returned 88.0, 64.6 and 89.5 requests per second. That’s a ±28% noise floor, wider than most of the effects worth measuring. The container was sharing a laptop’s CPU with PostgreSQL, Redis, the worker and the load generator itself.

So the test harness got a Compose overlay that pins the API to the production task’s shape: a quarter of a core, 512 MiB. A cgroup quota is a guaranteed share of a core, not “whatever is left”. The numbers became production-shaped, and they became repeatable.

The data had to be production-shaped too. The seed has 50 tenants, 400,000 conversations and 2 million messages, and one tenant holds 55% of all of it. Real multi-tenant data is skewed, and a skewed tenant is where a missing index hides.

First, the boring wins

The first pass found what load tests usually find: queries doing far more work than they should.

Change	Before	After
Inbox list (composite index matching its filter and sort)	310 ms	12.9 ms
Contacts list (indexes on the foreign keys its lateral joins use)	3,782 ms	18.5 ms
API throughput, full request mix at 16 callers	35.4 req/s	70.4 req/s
API p50 latency, same run	349 ms	121 ms

The inbox query had been scanning about 220,000 rows and sorting on disk for every request. We also added indexes on 57 foreign keys. Besides speeding up reads, that stopped a privacy erasure (deleting one contact under Malaysia’s PDPA) from scanning conversation history.

After that pass, PostgreSQL never went above about 26% of its two vCPUs. The API task sat at 100% of its CPU quota from the first concurrent request. The database was no longer the bottleneck in any scenario.

Then the surprise

Here is the API under rising concurrency, after the index work. This was a separate run from the numbers above, so compare rows within each table rather than across them:

Concurrent callers	Throughput	p50	p99
4	157.6 req/s	20.8 ms	107 ms
16	130.1 req/s	84.0 ms	632 ms
32	98.7 req/s	208.5 ms	1,624 ms
192	55.0 req/s	2,508.8 ms	13,119 ms

Throughput fell as load rose. That’s congestive collapse.

One event loop in one process on a quarter of a core gains nothing from more concurrent work. Extra requests don’t run faster; they queue. Past the knee, the queue is pure loss. At 192 callers the server did a third of the work it managed at four and made everyone wait thirteen seconds for it. Most of that work was wasted anyway, because a real client has given up long before its response is written.

And nothing alerted, because nothing failed. An error-rate alarm watches for 5xx. This was a wall of very slow 200s.

Why the limits we already had didn’t catch it

The API already had several bounds, from the edge inwards: a WAF rate rule, per-tenant/user/IP throttles, a request-body size cap, and a database connection-pool bulkhead that returns 503 when requests pile up on the database. None of them covered this case:

• The pool bulkhead only fires when requests are blocked waiting for a database connection. After the index work, queries returned in about 12 ms. The queue was in the event loop, not the pool.
• The per-tenant throttle is an abuse ceiling, sized an order of magnitude above real usage. In this run, 192 callers across 50 tenants averaged about 66 requests a minute each, nowhere near it. Correctly so: this wasn’t abuse, it was load.
• More uvicorn workers per task would only contend. A 0.25 vCPU task can’t use even one core. The real lever is more tasks, which is what the CPU autoscaler does. But a new task takes about a minute to start, and something has to protect the process during that minute.

Admission control, with a limit nobody configures

The fix is admission control: cap how much work the process accepts, queue a little, and reject the rest with a typed error that clients and load balancers understand.

The interesting part is the number. My first version used a hand-picked concurrency limit of 24. On the real 0.25 vCPU task, the right number turned out to be 5. A count of in-flight requests means something different on every machine, so any constant is wrong in a different direction in each environment.

So there is no concurrency setting to tune. The configured value is a latency target: the 200 ms p95 from our non-functional requirements. The controller adjusts concurrency to hold it, with the same rule TCP uses for congestion: additive increase while there is headroom, multiplicative decrease when there isn’t (AIMD). Work beyond the limit waits in a small bounded queue. Overflow gets 503 with Retry-After.

A simplified sketch of the idea as ASGI middleware:

import asyncio, time
from starlette.responses import JSONResponse

class AdmissionControl:
    """Concurrency limit derived from a latency target (AIMD)."""

    def __init__(self, app, target_s=0.2, max_queue=16, max_wait_s=1.0):
        self.app, self.target = app, target_s
        self.limit, self.inflight, self.waiting = 4.0, 0, 0
        self.max_queue, self.max_wait = max_queue, max_wait_s
        self.cond = asyncio.Condition()

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":
            return await self.app(scope, receive, send)
        async with self.cond:
            if self.inflight >= int(self.limit) and self.waiting >= self.max_queue:
                return await self.reject(scope, receive, send)
            self.waiting += 1
            try:
                await asyncio.wait_for(
                    self.cond.wait_for(lambda: self.inflight < int(self.limit)), self.max_wait)
            except asyncio.TimeoutError:
                return await self.reject(scope, receive, send)
            finally:
                self.waiting -= 1
            self.inflight += 1
        started = time.monotonic()
        try:
            await self.app(scope, receive, send)
        finally:
            elapsed = time.monotonic() - started
            async with self.cond:
                self.inflight -= 1
                if elapsed <= self.target:
                    self.limit = min(64, self.limit + 1 / self.limit)   # additive increase
                else:
                    self.limit = max(1, self.limit * 0.9)               # multiplicative decrease
                self.cond.notify_all()

    async def reject(self, scope, receive, send):
        await JSONResponse({"detail": "overloaded"}, 503, headers={"Retry-After": "1"})(scope, receive, send)

It’s a sketch, not a drop-in: a production version also needs per-route exemptions (health checks), metrics, and care around streaming responses.

What it bought, honestly

The capacity of that task is roughly 25 to 30 requests per second. Measured against it:

Offered load	Effect
Up to capacity	No shedding, no change
About 2× capacity	Throughput ~26 → ~29 req/s, p99 6.5 s → 3.7 s
About 10× capacity	Little help: shed requests took 4.5 s at p50, as slow as served ones

The 10× row is the important caveat. At that point the process is starved at the socket layer. Uvicorn must still accept and parse every request before any application code, including this middleware, gets to look at it. Rejecting is only as cheap as the process is idle, and at 10× it isn’t idle. No in-process mechanism can fix that.

The protections that do work at 10× sit elsewhere. The WAF and per-tenant throttles catch the common case, one tenant misbehaving. The CPU autoscaler (a 60% target, between 2 and 6 tasks) adds capacity for real growth. Admission control covers the minute it takes a new task to start. Shedding at the edge, with a connection cap or a smaller listen backlog, is the next step. It’s deliberately untried rather than guessed at.

What admission control does buy, and why it stays:

• Work in the process is bounded to the limit plus the queue, not to whatever the network offers.
• Failure is explicit and typed (503 with Retry-After), which clients, SDK retry policies and load balancers all act on. The alternative was a pool-exhaustion error after fifteen seconds.
• The ceiling tracks the hardware instead of a constant somebody picked on a laptop.

What I’d tell anyone load-testing an async API

1. Pin the CPU. Test at the production task’s shape, or your numbers are noise.
2. Skew your seed data. One huge tenant finds the missing indexes a uniform seed never will.
3. Watch throughput and p99 together. Error rate alone will happily report a collapsing service as healthy.
4. Derive limits from a target, not a constant. The right concurrency on your laptop is wrong in production.
5. Write down where a fix stops working. “Little help at 10×” is as useful to the next engineer as the 2× win.

If your API slows down under load, leaks between tenants or costs too much to run, that’s the backend and AWS hardening work I take on.