Designing a rate limiter, from first idea to production

At three in the morning, one customer’s integration gets stuck in a retry loop and sends the same request four hundred times a second. Nothing is wrong with your code, and nobody is being malicious. But every customer shares the same database and the same pool of workers, so a minute later everyone else is getting timeouts too.

A rate limiter is the code that stops that from happening. It counts what each caller does and says no, quickly and politely, when someone asks for more than their share. It’s also one of the most common system design interview questions, because a good answer touches almost everything: data structures, concurrency, distributed state, failure handling and product judgment.

This article builds one from nothing. By the end you’ll be able to:

• explain what a rate limit is and choose sensible numbers for one;
• implement the five standard algorithms and say which one fits which job;
• make a limiter correct across many servers, and avoid the two bugs almost everyone ships first;
• walk through the design in an interview, with real capacity numbers;
• read a production limiter and know why each line is there.

If you’ve never built a limiter, start at Beginner. If you have, skip to Advanced: that’s where most production bugs live. The last section walks through the limiters I run in ezychat.io, a multi-tenant customer-messaging product, including the bugs we fixed.

Beginner: what a rate limiter is for

One caller can hurt everyone

Most backends are shared. Every customer’s requests land on the same servers, wait in the same queues and read from the same database. That’s what makes software cheap to run, and it’s also why one caller’s bad day can become everybody’s bad day.

A rate limiter caps how much any one caller can ask for. You’ll want one for four reasons:

1. Abuse. Someone tries ten thousand passwords against one account, scrapes your whole catalogue, or creates fake accounts by the thousand. A login limit of ten attempts a minute turns a password-guessing attack that takes an hour into one that takes months.
2. Fairness. A customer with a buggy script shouldn’t be able to use up the capacity everyone else is paying for.
3. Cost. Some requests cost real money: a text message, a call to an AI model, a large export. A loop that calls a paid API can produce a bill before anyone notices.
4. Protecting what’s behind you. Your database can handle so many queries per second. Third-party APIs you call have their own limits and will block you if you exceed them.

Rate limiting is one of three questions every request has to answer. Authentication asks who you are. Authorization asks whether you’re allowed to do this. Rate limiting asks whether you’re allowed to do it again, right now.

What a limit actually says

You already live with rate limits. Your bank card has a daily cash withdrawal limit: you can take out, say, RM1,500 per day. That one sentence has the three parts every rate limit has:

• An amount: RM1,500, or 100 requests.
• A window: per day, or per minute.
• An identity: your card, or a user, an API key, an IP address. This is called the key, because it’s what the limiter looks each count up by.

A software limit reads the same way: “100 requests per minute per user.” Two more decisions finish the definition:

• What counts. Usually one request is one unit. Sometimes requests have different costs: a search might cost 1 and a bulk export 50. AI APIs often limit tokens (pieces of text) per minute as well as requests.
• What happens over the limit. Most limiters reject the request straight away. Some put it in a queue and serve it later. A few slow it down.

What the caller sees: 429 Too Many Requests

HTTP has a status code for exactly this: 429 Too Many Requests. A good 429 response tells the caller how long to wait:

HTTP/1.1 429 Too Many Requests
Retry-After: 18
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 18
Content-Type: application/json

{"error": "rate_limited", "detail": "too many requests; please slow down"}

Retry-After: 18 means “try again in 18 seconds.” A well-behaved client waits that long. Clients that ignore it and retry immediately just get more 429s, and if every client retries at the same moment you get a stampede. So good clients add backoff (wait longer after each failure) and jitter (a small random amount, so they don’t all retry together).

Don’t confuse 429 with 503 Service Unavailable. A 429 says “you are over your limit.” A 503 says “I am overloaded right now,” whoever you are. They need different reactions from the client, so use the right one.

Where a limiter can sit

A request passes through several layers on its way to your code, and a limiter can sit at any of them:

• The client can limit itself, and good SDKs do. But you can never rely on it: anyone can write their own client.
• A CDN or web application firewall (WAF) sees requests before they reach your servers, so rejecting there is almost free. It only knows the IP address, though, not which user or customer is behind it.
• The load balancer can cap connections, which protects against a different problem (too many open sockets) but doesn’t know about callers.
• An API gateway knows the API key, and often which plan it’s on, so it can enforce “free plan: 60 a minute, paid: 600.”
• Your service knows the most: the logged-in user, their organization, which feature they’re using and how expensive it is.
• Upstreams. You can also limit what you send to other systems, to stay inside their limits.

Real systems usually limit at more than one layer, because each layer catches something the others can’t see. We’ll come back to that in Expert.

Your first rate limiter

Here’s a complete limiter in about fifteen lines. It allows limit requests per window seconds for each caller:

import time


class FixedWindowLimiter:
    """Allow `limit` requests per `window` seconds for each caller."""

    def __init__(self, limit, window, clock=time.monotonic):
        self.limit = limit
        self.window = window
        self.clock = clock
        self.counts = {}  # (caller, window number) -> requests so far

    def allow(self, caller):
        window_number = int(self.clock() // self.window)
        key = (caller, window_number)
        count = self.counts.get(key, 0)
        if count >= self.limit:
            return False
        self.counts[key] = count + 1
        return True

The trick is window_number. Dividing the current time by the window length and rounding down gives the same number for every moment inside one window. With a 60-second window, every second from 0 to 59 gives window 0, every second from 60 to 119 gives window 1, and so on. So “how many requests in this window” is just “how many requests with this window number.”

The clock is passed in so we can control time in tests. Let’s try it with a fake clock and a limit of 3 per minute:

now = 0.0
limiter = FixedWindowLimiter(limit=3, window=60, clock=lambda: now)

for i in range(1, 6):
    print("alice request", i, limiter.allow("alice"))
print("bob request 1", limiter.allow("bob"))

now = 60.0  # one minute later: a new window
print("alice, next minute", limiter.allow("alice"))

# alice request 1 True
# alice request 2 True
# alice request 3 True
# alice request 4 False
# alice request 5 False
# bob request 1 True
# alice, next minute True

Alice gets three requests, then two refusals. Bob has his own count, so he isn’t affected by Alice. A minute later Alice starts fresh.

This limiter works, and for a single small server it’s genuinely fine. It has three problems, which the rest of this article fixes one at a time:

1. It leaks memory. Old windows are never deleted. Every caller adds one entry per minute, forever.
2. It only works on one server. Run it on four servers behind a load balancer and each one counts separately.
3. It lets bursts through at the edge of each window. That one’s subtle, and it’s where Intermediate begins.

class FixedWindowLimiter:
    """Same behavior, but each caller keeps only its current window."""

    def __init__(self, limit, window, clock):
        self.limit, self.window, self.clock = limit, window, clock
        self.counts = {}  # caller -> (window number, count)

    def allow(self, caller):
        window_number = int(self.clock() // self.window)
        stored_window, count = self.counts.get(caller, (window_number, 0))
        if stored_window != window_number:
            count = 0  # an old window: overwrite it instead of keeping it forever
        if count >= self.limit:
            return False
        self.counts[caller] = (window_number, count + 1)
        return True


now = 0.0
limiter = FixedWindowLimiter(limit=3, window=60, clock=lambda: now)
for minute in range(1000):
    now = minute * 60.0
    limiter.allow("alice")
print("entries stored for alice after 1,000 minutes:", len(limiter.counts))
# entries stored for alice after 1,000 minutes: 1

Intermediate: five algorithms, and how to choose

Every rate limiter answers the same question: has this caller used up their allowance? The algorithms differ in how they remember the past, and that choice decides four things: how accurate the limit is, how much memory each caller costs, how bursts are treated, and how much work each request takes.

Fixed window

This is what we just built. Time is cut into fixed blocks (each minute, each hour), and each caller gets a counter per block.

It’s as cheap as a limiter gets: one counter per caller, one lookup and one increment per request. Its weakness is the moment one window ends and the next begins:

now = 0.0
limiter = FixedWindowLimiter(limit=100, window=60, clock=lambda: now)

allowed = 0
now = 59.5  # half a second before the minute ends
allowed += sum(limiter.allow("mallory") for _ in range(100))
now = 60.5  # half a second after the next minute starts
allowed += sum(limiter.allow("mallory") for _ in range(100))
print("allowed in one second:", allowed)

# allowed in one second: 200

“100 per minute” turned into 200 in one second. In general, a fixed window can let through twice the limit in a short burst straddling a boundary. If the limit exists to protect a database that can take 100 queries in a minute, that burst is exactly what you were trying to prevent.

There’s a second, quieter problem. Every caller’s window resets at the same moment, at the top of the minute. Clients that were refused all retry at 1:00:00 together, and your traffic gets a spike every minute on the minute.

Sliding log

The most direct fix is to stop using blocks at all. Keep a log of when each allowed request happened, and count how many are in the last 60 seconds, measured back from now:

from collections import deque


class SlidingLogLimiter:
    """Remember the time of every allowed request in the last `window` seconds."""

    def __init__(self, limit, window, clock):
        self.limit = limit
        self.window = window
        self.clock = clock
        self.logs = {}  # caller -> deque of timestamps, oldest first

    def allow(self, caller):
        now = self.clock()
        log = self.logs.setdefault(caller, deque())
        while log and log[0] <= now - self.window:  # forget anything older than the window
            log.popleft()
        if len(log) >= self.limit:
            return False
        log.append(now)
        return True

The same boundary attack:

now = 0.0
limiter = SlidingLogLimiter(limit=100, window=60, clock=lambda: now)
allowed = 0
now = 59.5
allowed += sum(limiter.allow("mallory") for _ in range(100))
now = 60.5
allowed += sum(limiter.allow("mallory") for _ in range(100))
print("allowed in one second:", allowed)
now = 119.6  # 60.1 s after the first burst: those 100 entries have aged out
print("a minute later:", sum(limiter.allow("mallory") for _ in range(100)))

# allowed in one second: 100
# a minute later: 100

The sliding log is exact: at no moment are there more than 100 allowed requests in any 60-second span. The price is memory. It stores one timestamp per allowed request, so a limit of 1,000 per hour means up to 1,000 timestamps per caller. In Redis this is usually a sorted set (ZADD to record, ZREMRANGEBYSCORE to forget, ZCARD to count), which costs several commands and a lot more memory than a counter.

Use it when the limit is small and must be exact: five password resets an hour, three payout requests a day.

Sliding window counter

This is the clever compromise, and what most large systems use. Keep the fixed-window counters, but when you check, also look at the previous window and count only the part of it that’s still inside the last 60 seconds:

class SlidingWindowCounter:
    """Two counters per caller: this window and the previous one."""

    def __init__(self, limit, window, clock):
        self.limit = limit
        self.window = window
        self.clock = clock
        self.counts = {}  # (caller, window number) -> count

    def estimate(self, caller):
        now = self.clock()
        current = int(now // self.window)
        elapsed = (now % self.window) / self.window  # how far through this window, 0.0 to 1.0
        previous_count = self.counts.get((caller, current - 1), 0)
        current_count = self.counts.get((caller, current), 0)
        # The part of the previous window still inside "the last 60 seconds" is (1 - elapsed).
        return previous_count * (1 - elapsed) + current_count, current

    def allow(self, caller):
        estimated, current = self.estimate(caller)
        if estimated >= self.limit:
            return False
        self.counts[(caller, current)] = self.counts.get((caller, current), 0) + 1
        return True

now = 0.0
limiter = SlidingWindowCounter(limit=100, window=60, clock=lambda: now)
allowed = 0
now = 59.5
allowed += sum(limiter.allow("mallory") for _ in range(100))
now = 60.5
allowed += sum(limiter.allow("mallory") for _ in range(100))
print("allowed in one second:", allowed)
now = 90.0  # halfway through the second window
print("estimate at t=90:", round(limiter.estimate("mallory")[0], 1))
print("allowed at t=90:", sum(limiter.allow("mallory") for _ in range(100)))

# allowed in one second: 101
# estimate at t=90: 51.0
# allowed at t=90: 49

The boundary attack now gets 101 through instead of 200, and halfway through the next window the limiter correctly allows about half a window’s worth. It costs two counters per caller instead of a log.

It’s an estimate, and you should know exactly where it’s wrong. It assumes the previous window’s requests were spread evenly across it. If they actually all came in the last few seconds (as in our attack), the formula underweights them and lets a little extra through. If they all came at the start, it overweights them and refuses a little early. For ordinary traffic that’s spread out, the error is small. You’ll see it in the simulator below if you start a steady stream halfway through a window.

Token bucket

The token bucket thinks about the problem differently. Instead of counting what you’ve used, it tracks what you have left:

Each caller has a bucket that holds up to capacity tokens and refills at rate tokens per second. A request takes one token. No token, no request. Two numbers describe the whole policy: the capacity is the biggest burst you’ll allow, and the rate is the long-run average.

class TokenBucket:
    """Each caller has a bucket of up to `capacity` tokens that refills at `rate` per second."""

    def __init__(self, capacity, rate, clock):
        self.capacity = capacity
        self.rate = rate
        self.clock = clock
        self.buckets = {}  # caller -> (tokens, time we last topped it up)

    def allow(self, caller, cost=1):
        now = self.clock()
        tokens, last = self.buckets.get(caller, (self.capacity, now))
        tokens = min(self.capacity, tokens + (now - last) * self.rate)  # refill lazily
        if tokens < cost:
            self.buckets[caller] = (tokens, now)
            return False
        self.buckets[caller] = (tokens - cost, now)
        return True


now = 0.0
bucket = TokenBucket(capacity=10, rate=1.0, clock=lambda: now)  # 10 burst, 1 per second sustained
print("burst of 15 at t=0:", sum(bucket.allow("app") for _ in range(15)))
now = 5.0
print("7 more at t=5:", sum(bucket.allow("app") for _ in range(7)))
now = 60.0
print("one expensive call (cost 8) at t=60:", bucket.allow("app", cost=8))

# burst of 15 at t=0: 10
# 7 more at t=5: 5
# one expensive call (cost 8) at t=60: True

Notice there’s no background job adding tokens every second. The bucket refills lazily: when a request arrives, we work out how many tokens would have dripped in since the last visit. That keeps it to two numbers per caller and no timers.

Token buckets are popular for public APIs (Amazon’s API Gateway documents its throttling as a token bucket) because they match how real clients behave: quiet for a while, then a burst. The cost parameter makes it natural to charge more for expensive operations.

One thing to be clear about: a token bucket intends to allow bursts. In any one-minute span, it can admit a full bucket plus a minute of refill. If you need “never more than N in any window,” a token bucket isn’t the right tool.

Leaky bucket: a queue or a meter

“Leaky bucket” means two different things, and people mix them up.

The leaky bucket as a queue. Requests pour into a bucket with a hole in the bottom. They leak out, to be processed, at a constant rate. If the bucket is full, new requests spill over and are rejected. The point isn’t to count; it’s to smooth. However bursty the input, the output is perfectly steady. That’s what you want in front of something with a hard rate, like a messaging provider that accepts 20 sends per second and blocks you if you go faster. The cost is waiting time, and a queue to hold.

The leaky bucket as a meter. Without the queue, it’s mathematically the same as a token bucket seen from the other side. A tidy way to build it is GCRA (the generic cell rate algorithm, from telephone networks). It stores a single number per caller: the time its next request is “due.” A request arriving too far ahead of schedule is refused. The redis-cell module implements rate limiting this way.

class GCRA:
    """Leaky bucket as a meter. One number per caller: when its next request is 'due'."""

    def __init__(self, limit, window, burst, clock):
        self.interval = window / limit                 # each request's share of the window
        self.tolerance = self.interval * (burst - 1)   # how far ahead of schedule a caller may run
        self.clock = clock
        self.tat = {}  # caller -> theoretical arrival time

    def allow(self, caller):
        now = self.clock()
        tat = max(self.tat.get(caller, now), now)
        if tat - now > self.tolerance:
            return False
        self.tat[caller] = tat + self.interval
        return True


now = 0.0
meter = GCRA(limit=60, window=60, burst=5, clock=lambda: now)  # 1 per second, bursts of 5
print("10 at once:", [meter.allow("app") for _ in range(10)].count(True))
now = 3.0
print("at t=3:", [meter.allow("app") for _ in range(10)].count(True))


def leaky_queue(arrivals, capacity, interval):
    """Leaky bucket as a queue: accept while there is room, release one every `interval` s."""
    leave_times = []
    for t in arrivals:
        waiting = sum(1 for leave in leave_times if leave >= t)
        if waiting >= capacity:
            continue  # bucket full: this request spills over and is rejected
        leave_times.append(max(t, leave_times[-1] + interval) if leave_times else t)
    return leave_times


print("leave times:", leaky_queue([0.0] * 8, capacity=5, interval=1.0))

# 10 at once: 5
# at t=3: 3
# leave times: [0.0, 1.0, 2.0, 3.0, 4.0]

Eight requests arrive at once. The queue keeps five and lets them out one per second; the other three are refused.

Compare the five

Now try them yourself. The simulator runs each algorithm for real, with a limit per 4-second window so you don’t have to wait a minute to see a window end.

Here’s the same comparison as a table. The boundary-attack column comes from running the simulator’s code on identical traffic: a full limit of 5 in the 0.7 seconds before a window boundary, and 5 more in the 0.7 seconds after. It counts the most requests that reached the server in any 4 seconds.

In words: the fixed window is the cheapest and the loosest. The sliding log is exact and the most expensive. The sliding window counter is close to exact for two counters. The token bucket lets a full bucket through at once because that’s its job. The leaky bucket accepts a burst into its queue but releases it at a steady rate, so the server never sees the burst at all.

If you have to pick one with no other information, pick the sliding window counter for limits and the token bucket when you want to allow bursts on purpose.

Choosing the key

The algorithm decides how you count. The key decides whom. Getting the key wrong is the most common way a limiter hurts the people it’s meant to protect.

IP address. The only identity an anonymous request has, so it’s right for login pages, signup and public forms. But many people can share one IP: an office behind one router, a university, a mobile carrier that puts thousands of phones behind each address. A limit tight enough to stop an attacker can lock out a whole building. With IPv6 it’s the opposite problem: one home connection is usually given a whole block of addresses (commonly a /64, more addresses than the entire IPv4 internet), so key on that network prefix, not the full address.

User ID. Right once someone is logged in. It follows the person across networks and doesn’t punish their colleagues.

API key. Right for integrations, and the natural place to hang plan-based limits.

Tenant (organization or account). Right for protecting a shared system from any one customer. But on its own it has a trap: one looping script inside a big customer uses up the whole organization’s allowance, and their colleagues get locked out. The fix is two limits, a tighter one per user and a looser one per organization.

Combinations. Often the best key is two things: (user, endpoint) so a busy search box doesn’t block saving a form, or (organization, feature) so an expensive feature has its own budget.

One more trap with IPs. Behind a load balancer, your server sees the load balancer’s address, not the client’s. The client’s address arrives in the X-Forwarded-For header, which every proxy appends to. But the client can put anything it likes in that header before it reaches your proxy:

def client_ip(forwarded_for, peer_ip, trusted_proxies):
    """The address that connected to our outermost trusted proxy."""
    if trusted_proxies == 0:
        return peer_ip  # nothing in front of us: the TCP peer is the client
    hops = [h.strip() for h in forwarded_for.split(",") if h.strip()]
    if len(hops) < trusted_proxies:
        return peer_ip
    return hops[-trusted_proxies]  # count from the right: the end the client can't write


# The attacker sends "X-Forwarded-For: 1.2.3.4". The load balancer appends the real address.
header = "1.2.3.4, 203.0.113.9"
print("leftmost (spoofable):", header.split(",")[0].strip())
print("trusted, 1 proxy:   ", client_ip(header, peer_ip="10.0.1.7", trusted_proxies=1))

# leftmost (spoofable): 1.2.3.4
# trusted, 1 proxy:    203.0.113.9

If your limiter takes the leftmost address, an attacker gets a fresh limit on every request just by changing a header. Count from the right, as many hops as you have proxies you control.

Telling the caller what’s left

Callers behave better when they can see their limit. Three headers are common:

• Retry-After (standard HTTP): how long to wait, sent with a 429.
• RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset: the limit, what’s left, and seconds until it resets. These come from an IETF draft for standard rate limit headers; later revisions of the draft combine them into fewer fields, so check what your clients expect.
• X-RateLimit-*: the older, non-standard style many big APIs still use. GitHub’s API, for example, sends x-ratelimit-limit, x-ratelimit-remaining and x-ratelimit-reset, where reset is a Unix timestamp rather than a number of seconds.

import math


def limit_headers(limit, used, seconds_to_reset):
    remaining = max(limit - used, 0)
    headers = {
        "RateLimit-Limit": str(limit),
        "RateLimit-Remaining": str(remaining),
        "RateLimit-Reset": str(math.ceil(seconds_to_reset)),
    }
    if remaining == 0:
        headers["Retry-After"] = str(math.ceil(seconds_to_reset))
    return headers


print(limit_headers(limit=100, used=42, seconds_to_reset=17.3))
print(limit_headers(limit=100, used=100, seconds_to_reset=17.3))

# {'RateLimit-Limit': '100', 'RateLimit-Remaining': '58', 'RateLimit-Reset': '18'}
# {'RateLimit-Limit': '100', 'RateLimit-Remaining': '0', 'RateLimit-Reset': '18', 'Retry-After': '18'}

Round the wait up. Telling a client to retry in 17 seconds when the window actually resets in 17.3 earns it one more 429.

Advanced: one limit across many servers

Why local counters break

Your service runs on four servers behind a load balancer. Each one runs the limiter from earlier with a limit of 100 per minute. A caller’s requests get spread across all four, so each server sees a quarter of them, and the caller gets up to 400 a minute before any server says no. Add servers for capacity and you quietly raise every limit.

You could make the load balancer send each caller to the same server every time (sticky routing by key). That works until a server restarts and loses its counts, or the cluster grows and callers move. The usual answer is to keep the counters somewhere every server can see.

Redis: a hash map everyone shares

That somewhere is almost always Redis: an in-memory store that’s essentially a hash map you reach over the network. Two of its features make it ideal for counters:

• INCR key adds one and returns the new value, atomically: if two servers increment at the same moment, the result is +2, never +1.
• EXPIRE key seconds makes Redis delete the key automatically. A window’s counter can clean itself up when the window is over, which fixes our memory leak for free.

Counters are small. I measured it on Redis 7.2: 200,000 keys shaped like rl:api_user:100000000:29512345, each holding a small integer with a 120-second expiry, used 117 bytes per counter. A million active callers with two counters each (sliding window counter) is about 234 MB, which fits on one modest server.

The race: read, check, write

Here’s the obvious way to use Redis, and it’s wrong:

import threading
import redis

r = redis.Redis(port=6399)
r.delete("rl:demo")
LIMIT = 10
allowed = []
start = threading.Barrier(50)


def handle_request():
    start.wait()                         # release all 50 at the same instant
    count = int(r.get("rl:demo") or 0)   # 1. read
    if count < LIMIT:                    # 2. check
        r.incr("rl:demo")                # 3. write
        allowed.append(1)


# 50 requests arriving at the same moment, like 50 app workers serving one caller.
threads = [threading.Thread(target=handle_request) for _ in range(50)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print("limit:", LIMIT, "allowed:", len(allowed), "counter:", r.get("rl:demo").decode())

I ran it six times against a local Redis. It printed allowed: 13, 10, 12, 11, 16 and 13, against a limit of 10.

Each request reads the count, decides, then writes. Between one request’s read and its write, other requests read the same old value, all decide there’s room, and all go through. This is a check-then-act race, and it’s the same bug as two people booking the last seat on a flight. The demo below slows it down so you can watch it happen:

The tempting fix is to increment first and look at the result, since INCR itself is atomic:

count = r.incr(key)        # atomic: every request gets a different number
if count == 1:
    r.expire(key, 60)      # first request in this window sets the expiry
if count > LIMIT:
    reject()

That really does fix the race: fifty concurrent INCRs return 1 through 50, and exactly ten of them are 10 or less. It’s the fixed-window recipe you’ll find in many tutorials. It also contains the most dangerous bug in this article.

The key that never expires

INCR and EXPIRE are two separate round trips to Redis. What if the first one succeeds and the second never happens? The connection drops, the process is killed during a deploy, Redis fails over to a replica in between:

import redis

r = redis.Redis(port=6399)
key = "rl:login:203.0.113.9"
r.delete(key)

count = r.incr(key)          # round trip 1: the counter exists now...
# ...the connection drops here, before round trip 2 ever runs:
# r.expire(key, 60)
print("count:", count, "ttl:", r.ttl(key))  # -1 means "no expiry, ever"

# count: 1 ttl: -1

A TTL of -1 means the key will never be deleted. And look at the if count == 1 line: the expiry is only ever set by the first request in a window. That request is gone. Every later request sees 2, 3, 4…, never sets an expiry, and the counter climbs forever. Once it passes the limit, that caller is rate limited permanently. There’s no error anywhere: from the outside, one customer just can’t log in any more, and nothing in your logs says why.

There are three ways out:

1. Create the key with its expiry in one command, then increment. SET key 0 EX 60 NX creates the counter with an expiry only if it doesn’t exist; INCR keeps an existing expiry. If you crash between the two, you’ve left a zero with an expiry, which is harmless.
2. A transaction. MULTI, INCR, EXPIRE, EXEC runs both commands together. But inside a transaction you can’t look at a value and decide, so it can’t express “only increment if under the limit.”
3. A Lua script. Redis runs a script from start to finish as one operation, with no other command in between. It can read, decide and write. This is what production limiters use.

Atomic scripts

Here’s the sliding window counter as a Redis script. The read, the decision, the increment and the expiry all happen in one atomic step:

import time
import redis

SLIDING_WINDOW = """
local current_key, previous_key = KEYS[1], KEYS[2]
local limit, ttl, elapsed = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])

local current  = tonumber(redis.call('GET', current_key) or '0')
local previous = tonumber(redis.call('GET', previous_key) or '0')
if previous * (1 - elapsed) + current >= limit then
    return 0
end
redis.call('INCR', current_key)
redis.call('EXPIRE', current_key, ttl)   -- same script, so never a counter without an expiry
return 1
"""

r = redis.Redis(port=6399)
script = r.register_script(SLIDING_WINDOW)


def allow(identity, limit, window):
    now = time.time()
    number = int(now // window)
    elapsed = (now % window) / window
    keys = [f"rl:{identity}:{number}", f"rl:{identity}:{number - 1}"]
    return bool(script(keys=keys, args=[limit, window * 2, elapsed]))


results = [allow("demo-user", limit=5, window=60) for _ in range(7)]
print(results)
number = int(time.time() // 60)
print("ttl:", r.ttl(f"rl:demo-user:{number}"))

# [True, True, True, True, True, False, False]
# ttl: 120

The expiry is two windows, because the current window’s counter is still needed as the previous window for one more window after it ends.

I ran the 50-thread race again with a script doing the read, check and increment. Six runs, six times allowed: 10.

A few things to know about scripts:

• They block Redis while they run. Redis executes commands one at a time, which is exactly what makes a script atomic, and also why it must be tiny. Four commands is fine; a loop over thousands of keys is not.
• Send the script once, then call it by its hash. register_script in redis-py does this for you (SCRIPT LOAD once, EVALSHA after), so each request sends a short hash instead of the whole script text.
• They cost a bit more than a single command. On my laptop (an M1 Pro, one Redis 7.2 instance, 50 client connections, random keys), redis-benchmark measured about 66,000 of these scripts per second, against about 99,000 plain INCRs per second. Plan capacity with numbers measured on your own hardware, but expect a script to cost noticeably more than one command.
• In Redis Cluster, every key a script touches must live on the same node. Put the identity in braces, rl:{user:42}:29512345, so all of one caller’s keys hash to the same slot.

Clocks and windows

Our script computes the window number from the app server’s clock. If one server’s clock is two seconds fast, it starts the next window two seconds early, and the counts get muddled. Servers synced with NTP are usually within a few milliseconds of each other, which doesn’t matter for one-minute windows but starts to matter for one-second ones. If it matters for you, read the time inside the script with Redis’s TIME command, so there’s only one clock.

The top-of-the-minute stampede from earlier has a cheap fix too: give each key its own window offset, for example a hash of the key modulo the window length, so callers’ windows don’t all reset at the same moment.

Fewer round trips: lease, then spend locally

Every request now costs one round trip to Redis, typically well under a millisecond inside one data center. Usually that’s fine. When it isn’t, because Redis is far away or you’re handling a huge request rate, a server can lease allowance in chunks and spend it locally:

import time
import redis

r = redis.Redis(port=6399)
calls = 0


class LeasedLimiter:
    """Claim allowance from the shared counter in chunks, then spend it locally."""

    def __init__(self, identity, limit, window, chunk):
        self.identity, self.limit, self.window, self.chunk = identity, limit, window, chunk
        self.local = 0             # requests this server may still admit without asking Redis
        self.lease_window = None
        self.exhausted = False     # the shared allowance for this window is gone

    def allow(self):
        global calls
        window = int(time.time() // self.window)
        if window != self.lease_window:          # a new window voids old leases
            self.local, self.lease_window, self.exhausted = 0, window, False
        if self.local == 0:
            if self.exhausted:
                return False                     # no need to ask again until the window ends
            key = f"lease:{self.identity}:{window}"
            granted_up_to = r.incrby(key, self.chunk)   # one round trip claims a whole chunk
            r.expire(key, self.window * 2)
            calls += 1
            self.local = max(0, min(self.chunk, self.limit - (granted_up_to - self.chunk)))
            if self.local == 0:
                self.exhausted = True
                return False
        self.local -= 1
        return True


servers = [LeasedLimiter("api-key-42", limit=100, window=60, chunk=10) for _ in range(4)]
admitted = sum(servers[i % 4].allow() for i in range(130))
print("admitted:", admitted, "of 130, Redis calls:", calls)

# admitted: 100 of 130, Redis calls: 14

Four servers admitted exactly the limit with 14 calls to Redis instead of 130. The trade-off: a server that leases ten and only uses three is sitting on seven that nobody else can have, so callers can be refused slightly early, and a server that crashes takes its unused lease with it. (Like the earlier example, this sends INCRBY and EXPIRE separately, to keep it short. A real version should put them in one script.)

When one Redis isn’t enough

At some point one Redis instance runs out of throughput. Redis Cluster splits keys across nodes by hashing each key into one of 16,384 slots, so different callers’ counters land on different machines and the load spreads out.

The exception is a hot key: one enormous customer whose counter gets a large share of all traffic, all on one node. The fixes are to split that counter into several sub-keys (rl:{org:7}:a, rl:{org:7}:b, …), increment a random one, and add them up when checking; or to use leasing for that one key.

Fail open or fail closed

Sooner or later Redis will be unreachable for a few seconds. Your limiter has to decide what to do with requests it can’t count:

• Fail open: allow everything until Redis is back. Nobody is wrongly refused, but there’s no limit for a while.
• Fail closed: refuse everything. The limit holds, but a cache problem has become a total outage.

Decide per limit, by asking what it protects. A limit that exists for fairness or to stop runaway scripts should almost always fail open, with another layer (such as a WAF) still standing. A limit that guards something dangerous or expensive, like password attempts, payouts or paid text messages, should fail closed or fall back to a local in-memory limit set at the global limit divided by the number of servers.

Either way, give the Redis call a short timeout. A limiter that waits five seconds for a dead Redis has made every request five seconds slower, which is worse than having no limiter.

import redis

TOKEN_BUCKET = """
local key = KEYS[1]
local capacity, rate, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])

local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + math.max(0, now - ts) * rate)

local allowed = 0
if tokens >= cost then
    tokens = tokens - cost
    allowed = 1
end
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
-- A full bucket is the same as no bucket, so let Redis forget it once it would be full again.
redis.call('EXPIRE', key, math.ceil(capacity / rate) + 1)
return {allowed, tostring(tokens)}
"""

r = redis.Redis(port=6399)
bucket = r.register_script(TOKEN_BUCKET)
r.delete("tb:app")


def allow(now, cost=1):
    allowed, tokens = bucket(keys=["tb:app"], args=[10, 1.0, now, cost])
    return bool(allowed), float(tokens)


print("burst of 15 at t=1000:", sum(allow(1000.0)[0] for _ in range(15)))
print("t=1005:", [allow(1005.0)[0] for _ in range(7)].count(True))
print("ttl:", r.ttl("tb:app"))

# burst of 15 at t=1000: 10
# t=1005: 5
# ttl: 11

Expert: designing it for real

The interview walk-through

“Design a rate limiter” is a classic interview question because there’s no single right answer, only trade-offs you can explain or can’t. Here’s a structure that works, with the numbers filled in.

1. Requirements. Ask before you draw anything. A reasonable set:

• Functional: limit requests per API key according to its plan; allow per-endpoint overrides; return 429 with Retry-After and remaining-quota headers; change limits without a deploy.
• Non-functional: add no more than a millisecond or two at p99; never take the API down if the limiter fails; be accurate to within a few percent; handle 50,000 requests per second at peak from 10 million API keys, of which about 1 million are active in any minute.

2. Estimates. Do the arithmetic out loud:

• Throughput: one script call per request is 50,000 scripts per second. One Redis instance on my laptop managed about 66,000 a second, so one instance would be running close to its limit at peak with no headroom. Plan for a small cluster, three or four shards, and measure on the real hardware.
• Memory: 1 million active keys × 2 counters × 117 bytes ≈ 234 MB. Memory is not the problem. Throughput is.
• Latency: one round trip inside the data center, well under a millisecond. That fits the budget.

3. API. Inside the service, a single call: check(key, cost=1) returns allowed, remaining and reset_seconds. Limits live in a rules table (plan → limit, window, burst), cached in each server’s memory and refreshed every few seconds, so changing a limit is a data change, not a deploy.

4. High-level design.

The limiter runs as middleware in each app server (or in the API gateway). It reads the caller’s plan from the cached rules, calls the Redis script, and either passes the request on with rate limit headers or answers 429 itself.

5. Deep dives. The interviewer will pick some of these. Have an answer ready for each:

• Which algorithm? Sliding window counter for plan limits: two counters per key, no boundary burst. Token bucket if the product wants to allow bursts.
• Atomicity: a Lua script, so read, check, increment and expiry are one step. Mention both bugs from the Advanced section; it shows you’ve run one of these.
• Hot keys: split the counter of the one huge customer, or lease.
• Failure: fail open, with a local fallback limit, and a short Redis timeout.
• Observability: count 429s by rule and plan, and alert on sudden changes in either direction. A rule that suddenly rejects nothing may be broken; one that suddenly rejects a lot may be breaking a customer. Log the first rejection per key per window, not every one, or an attack will flood your logs too.

6. Bottlenecks and follow-ups. Redis throughput (shard), hot keys (split), cross-region (next section), and limits based on cost rather than request count (a cost argument).

Across regions

If your API runs in two regions, where do the counters live? There are three answers, each with a cost:

1. Split the limit. Each region enforces half. Simple and fast, but a customer whose traffic all lands in one region gets half their limit.
2. One global store. Exact, but every request pays a cross-region round trip, tens to hundreds of milliseconds. Rarely acceptable on the hot path.
3. Count locally, share asynchronously. Each region counts its own traffic and periodically tells the others. Fast and nearly fair, but for the length of the sync interval a caller can exceed the global limit. That’s eventual consistency: the regions agree in the end, not at every instant.

The question to ask is how much brief over-admission costs. For a fairness limit, very little: pick option 3. For a limit on something that costs real money per unit, maybe option 1 with generous headroom.

Layered defense

No single limiter can do everything, because each layer sees different things:

• The edge (CDN or WAF) sees only IPs, but rejects for almost nothing. It’s the right place to stop floods: thousands of requests a minute from one address.
• The gateway sees API keys and plans. It’s the right place for the commercial limits you put in your pricing page.
• The service sees users, tenants and features. It’s the right place for “one user can’t use up their organization’s allowance” and “this expensive feature gets its own budget.”
• Around your dependencies, you limit yourself, so you stay inside what a database or vendor can take.

Each layer catches what the outer ones can’t see, and outer layers keep the inner ones from being overwhelmed. You’ll see all four in the production section.

Rate limits aren’t concurrency limits

A rate limit counts requests per unit of time. It says nothing about how long each request takes. A caller well within “100 per minute” can still bring down a server if each request holds a database connection for ten seconds. What protects you there is a concurrency limit: a cap on how many requests are in progress at once, with the rest queued briefly or refused.

Throttling what you call

So far we’ve limited people calling us. Some of the most important limiters point the other way: they stop you from calling a paid or limited service too fast. AI model APIs are the clearest case. Vendors cap requests and tokens per minute, answer 429 when you go over, and bill for everything you send.

A limiter in front of a vendor typically:

• keeps a budget per model per minute, shared across all your servers in Redis, so you hit your own limit before the vendor’s;
• gives priority lanes: a customer waiting for a reply may use the whole budget, while background jobs get only a share and wait their turn;
• pauses a model for as long as the vendor’s Retry-After says when it does get a 429, instead of sending the next request into the same refusal;
• treats a revoked or refused key differently from a busy one: pause everything, then send a single probe request now and then to see if it works again.

Fairness between tenants

The same idea applies to work queues. If all customers share one queue, one customer who submits ten thousand jobs pushes everyone else to the back. A per-tenant limit on how much work each customer can start per minute fixes this. The best version doesn’t throw extra work away: it defers it with a short delay, so the busy customer’s jobs still run, just behind everyone else’s.

Testing a limiter

Limiters fail quietly, so test them deliberately:

• Inject the clock. Every limiter in this article takes a clock function, so tests can jump time to just before and just after a window boundary with no sleeping.
• Test concurrency against real Redis. The 50-thread race above is a test: assert that exactly limit requests are allowed.
• Test that every key expires. After a test run, scan the limiter’s keys and assert none has a TTL of -1. That one assertion would have caught the permanent-lockout bug.
• Test failure. Stop Redis and assert the limiter fails open (or closed) the way you decided, within its timeout.
• Roll out in shadow mode. Before enforcing a new limit, log who it would have rejected for a week. You’ll find the customers it would have broken before they do.

In production: the limiters in ezychat.io

ezychat.io is a multi-tenant customer-messaging product: AI and people answering customers on WhatsApp, Instagram, Messenger and web chat. Every customer shares the same API servers. The comment at the top of our tenant rate limiter says why that makes limits essential:

One customer’s runaway retry loop, misconfigured integration or scraper is therefore a platform-wide outage, not a degraded experience for that customer.

Here’s how the limits are layered, and the decisions behind them. The numbers are the defaults in the code; I’ve left out the security thresholds, since publishing them would only help someone stay under them.

The same comment is clear about what these limits are not:

It is an abuse guard, not a product limit. Every plan markets unlimited conversations and replies, so a limit a working team could reach by working would be a broken promise, not a safeguard.

So the defaults sit roughly an order of magnitude above observed legitimate use, and a throttled caller gets a 429 with Retry-After and nothing else: no billing, no warning banner.

Bug one: the counter that never expired

The first version of the shared limiter used INCR and then a separate EXPIRE. Today every limiter in the codebase goes through one function, consume, which replaced it. The comment that replaced it describes exactly the failure from the Advanced section:

If the EXPIRE failed — a dropped connection, a failover between the two round trips — the key was left with no TTL at all. Nothing would ever clear it, the counter kept climbing past the limit forever, and that identity was rate limited permanently, with no error anywhere to explain why.

The fix was to do both inside one Lua script, so a counter without an expiry can’t exist even for an instant.

Bug two: the boundary

The same rewrite fixed the fixed window’s boundary burst. The comment again:

A caller who spends its whole quota in the last second of one window and again in the first second of the next lands 2x the limit in barely over a second, which is precisely the burst shape the limit exists to stop.

This is the script that runs today, a sliding window counter that also returns the estimated count for the rate limit headers:

local current_key, previous_key = KEYS[1], KEYS[2]
local limit   = tonumber(ARGV[1])
local ttl     = tonumber(ARGV[2])
local elapsed = tonumber(ARGV[3])

local current  = tonumber(redis.call('GET', current_key) or '0')
local previous = tonumber(redis.call('GET', previous_key) or '0')
local estimated = previous * (1.0 - elapsed) + current

if estimated >= limit then
    return {1, math.floor(estimated + 0.5)}
end

current = redis.call('INCR', current_key)
redis.call('EXPIRE', current_key, ttl)
return {0, math.floor(previous * (1.0 - elapsed) + current + 0.5)}

Keys look like rl:{bucket}:{identity}:{window}, the TTL is two windows, and Retry-After is whatever is left of the current window, plus one second, never less than one.

Failing open, on purpose

consume fails open: if Redis can’t be reached, the request is allowed. The docstring spells out the trade:

The limiter exists to shed abuse, and refusing all traffic because the abuse counter is unavailable converts a cache incident into a total outage. The trade is explicit — during a Redis outage there is no rate limiting, and the WAF’s rate-based rule is the remaining layer.

That WAF rule is attached to the load balancer and blocks any single IP that sends far more requests in a rolling five minutes than any real user would. It knows nothing about users or customers, which is exactly why it still works when the application’s limiter doesn’t. Every other limiter in the codebase, from the AI throttle to queue fairness, makes the same fail-open choice for the same reason.

Choosing keys, and the seventh attempt

The original per-endpoint limiter keyed everything on the client’s IP. Its docstring now explains when that’s right and when it isn’t:

Per client IP. The right key for anonymous and pre-auth surfaces — signup, login, the public booking page, the webchat ingress — where the IP is the only identity there is. Wrong for an authenticated per-org action: every org behind one corporate NAT would share a single budget.

What forced the change was brand derivation, a feature that reads a customer’s website and everything the organization already has, then proposes a draft brand profile. One run is several model calls and up to four outbound fetches, charged to one organization. With an IP key, the seventh attempt from an office address got a 429, whoever made it, and the customer just saw a generic error. It now uses rate_limit_org, keyed on the organization, at 12 per hour. Account actions use a per-user key for the reasons in this comment:

Per user, not per IP: an office behind one NAT must not share a budget, and a stolen session must not get a fresh one by changing address.

For anonymous requests, the client IP comes from X-Forwarded-For, counted from the right by the number of proxies we run:

def _client_ip(request: Request) -> str:
    settings = get_settings()
    if settings.trusted_proxy_count > 0:
        parts = [
            p.strip() for p in request.headers.get("x-forwarded-for", "").split(",") if p.strip()
        ]
        if len(parts) >= settings.trusted_proxy_count:
            return parts[-settings.trusted_proxy_count]
    return request.client.host if request.client else "unknown"

Per user before per organization

Every API call passes a middleware that counts it twice: against the user (600 a minute) and against their organization (1,200 a minute). Anonymous calls count against their IP (240 a minute). The order matters:

# Per-user first: it is the tighter of the two, so a single looping
# session is stopped before it has spent the organisation's budget.
user = await consume("api_user", str(claims.user_id), settings.api_rate_limit_user_per_minute, 60)
if not user.allowed:
    return user
org = await consume("api_org", str(claims.org_id), settings.api_rate_limit_org_per_minute, 60)

A per-organization limit alone would turn one looping browser tab into an outage for the whole customer. Checking the user first stops the loop before it touches the shared budget. A token that fails verification doesn’t get a free pass either: it falls through to the anonymous IP bucket, so a flood of forged tokens is still counted.

What we never rate limit

Three paths are exempt, and each reason is a lesson in what a 429 actually does to the caller:

• /health, the load balancer’s health check. A 429 there reads as “this server is unhealthy,” and the load balancer drains it. A throttle becomes an outage.
• /webhooks, where Meta delivers incoming WhatsApp, Instagram and Messenger messages. Meta retries failed deliveries, then disables a subscription that keeps failing, and re-enabling it is a manual step. Throttling it risks silently losing every incoming message.
• /billing/webhook, where Stripe sends subscription events. Being unauthenticated, it would land in the anonymous per-IP bucket, shared by every event Stripe sends from that address for every customer. Stripe retries for about three days and then gives up, after which a paying customer could silently keep free-plan limits.

Those endpoints are bounded differently: their signatures are checked before any database work, and the webhook ingress runs as its own service with its own capacity.

Limiting what we call: the AI throttle

Production AI runs on Gemini, and the vendor has its own per-minute quotas. A throttle checks three rules before every request the Gemini adapter sends, including retries, because each one spends the same quota:

1. A budget per model per minute, counted in Redis across every process with the same consume function. A live call, where a customer is waiting for a reply, may use the whole budget and waits up to five seconds for room before failing over to the backup model. Background work (scoring, summaries, sweeps) stops at 60% of the budget and is held back at once, because it will come round again on its own.
2. A pause when the vendor rate-limits a model, for as long as the vendor said (30 seconds if it named no time, at most five minutes), so the next request doesn’t land in the same refusal.
3. A pause of every model when the vendor refuses the key. The docstring records why: on 3 October 2026 the vendor denied the project, and scoring asked again every thirty seconds for every room, about 120 refused calls an hour from one room, for nothing. Now every model waits five minutes, then exactly one request goes out as a probe. If it’s refused, the pause repeats and one log line is written for the alarm to count; if it’s answered, everything resumes.

The throttle’s state is keyed by a fingerprint of the API key, not the vendor’s name, so a test on a different project’s key can never pause production.

Fairness on a shared queue

Comment replies run on one shared, first-in-first-out queue. A customer who publishes a campaign across many pages can put hundreds of jobs in front of everyone else. Each organization may start 60 batches a minute, roughly 1,500 comments, which is far above any real campaign. Past that, its batches defer in milliseconds (no model call, no database read) and come back after 30 seconds. Nothing is dropped: the busy customer’s work is delayed, which is the right outcome for the customer that caused the flood.

A limit that isn’t an error

When an agent types a reply, the composer tells the customer’s app “typing…”. It fires on keystrokes, so a stuck browser tab could send thousands of these. The limit is one per conversation every five seconds, and when it’s hit, the endpoint just answers {"sent": false}. Being rationed is an ordinary answer here, not a failure, so there’s no 429 and no error in the logs.

Shapes a request counter can’t see

Not every abuse shape is a request rate. A WebSocket is one request that then stays open, holding a task and a Redis connection for as long as it lives, so a request limiter counts it once no matter how long it’s held or how many are opened. That needs a different tool: a cap on concurrent connections per user. The general habit is worth more than the example: list the resources a caller can hold, not just the requests they can send, and make sure each one has its own limit or a written reason why not yet.

What I’d copy

• Do the check, the increment and the expiry in one atomic step. The counter that never expired came from splitting them.
• Choose fail-open or fail-closed per limit, write the reason next to the code, and make sure another layer stands when you fail open.
• Key by who pays. IP only when there’s nothing better; the organization for costs it carries; the user so one person can’t spend everyone’s allowance.
• Know what a 429 does to each caller. Health checks and webhooks are callers too, and some of them punish you for refusing.
• Limit yourself, not just your users, in front of every paid or rate-limited dependency.

Cheat sheet

• A limit is an amount, a window and a key: “100 per minute per user.”
• Over the limit: 429 Too Many Requests with Retry-After. 503 means “I’m overloaded,” not “you’re over.”
• Fixed window: one counter. Up to 2× the limit in a burst at a boundary.
• Sliding log: exact, one timestamp per request. For small limits that must be exact.
• Sliding window counter: two counters and a weighted estimate. The default for API limits.
• Token bucket: capacity is the burst, rate is the average. Bursts by design; handles weighted costs.
• Leaky bucket: as a queue, it smooths output for a dependency; as a meter (GCRA), it’s one number per caller.
• Keys: IP for anonymous traffic; user, API key or organization once you know them; often two at once.
• X-Forwarded-For: count from the right, by the number of proxies you trust.
• Many servers: shared counters in Redis, or every server multiplies the limit.
• Atomicity: read, check, increment and expire in one Lua script.
• The silent killer: a counter without an expiry locks a caller out forever.
• Redis capacity: about 117 bytes per counter; a script costs more than a single command. Measure on your hardware.
• Failure: fail open for fairness limits, with another layer still standing; fail closed for dangerous ones.
• Regions: split the limit, centralize it, or sync asynchronously. Decide what brief over-admission costs.
• Not the same thing: rate limits count per unit of time; concurrency limits count requests in flight.