Caching, from first idea to production

When you opened this page, it most likely came from a CloudFront edge near you, not from the storage bucket it lives in. The edge already had a copy, checked it was still allowed to use it, and handed it over. That’s caching: keeping a copy of an answer close to where it’s needed, so you don’t have to fetch or compute it again.

It’s the cheapest performance work in backend engineering, and the source of some of its strangest bugs: a price that won’t update, a permission change that takes a day to apply, a database that falls over the moment one popular key expires. All of them come from a few decisions you can learn to make on purpose.

By the end of this article you’ll be able to:

• say when a cache will help and when it won’t, with numbers;
• pick a caching pattern, an invalidation method and an eviction policy, and defend the choice;
• write HTTP Cache-Control headers that are fast and still correct after a deploy;
• prevent stampedes, stale writes and cross-tenant leaks, the three bugs that hurt most in production;
• size a cache from its hit ratio and walk through a caching design in an interview.

Start at Beginner if “cache hit” is a new phrase. If you already put Redis in front of things, jump to Advanced: that’s where the production bugs live. The last section walks through the real caches in ezychat.io, a multi-tenant customer-messaging product I’m the primary engineer of, and the CDN setup of this site, with the code comments that explain each choice.

Beginner: what a cache is and when it pays

Why cache at all?

Because the same expensive answer gets asked for again and again, and keeping a copy nearby costs far less than producing it each time. Here are real numbers. The ones marked “laptop” I measured for this article on an M1 Pro; the others are the widely quoted orders of magnitude from Jeff Dean’s “latency numbers every programmer should know”.

The middle of that table surprised me. On my laptop, Postgres answered a primary-key read faster than Redis answered a GET. Both costs are mostly the network stack and the client library; the database’s own work for a primary-key read is tiny. So Redis in front of a cheap indexed read doesn’t make it faster. It takes load off the database, which matters when thousands of requests ask at once.

The bottom row is where caching shines: a 339 ms aggregate that becomes a 0.1 ms GET is a 3,000-fold saving. The rule of thumb: a cache saves whatever the miss costs. Cache the expensive things (aggregates, big joins, calls to other services, rendered pages, model calls). Cheap things you cache only for load.

Hits, misses and TTL

A cache is a store of answers you’ve already worked out, looked up by a key that describes the question. When the answer is there, that’s a hit. When it isn’t, that’s a miss: you do the slow work, then usually store the result for next time. The share of lookups that are hits is the hit ratio.

A cached answer can go out of date: the exchange rate moves, the product gets a new price. An answer that no longer matches the source of truth is stale. The simplest defense is a time to live (TTL): every entry carries an expiry time, and after it you treat the entry as missing.

Here’s a complete cache with a TTL. The clock is passed in so we can move time forward in the example:

import time


class TTLCache:
    """Remember answers for `ttl` seconds."""

    def __init__(self, ttl, clock=time.monotonic):
        self.ttl = ttl
        self.clock = clock
        self.entries = {}  # key -> (value, time it expires)
        self.hits = self.misses = 0

    def get(self, key):
        entry = self.entries.get(key)
        if entry is not None and self.clock() < entry[1]:
            self.hits += 1
            return entry[0]
        self.misses += 1
        self.entries.pop(key, None)  # expired or never there
        return None

    def set(self, key, value):
        self.entries[key] = (value, self.clock() + self.ttl)


now = 0.0
cache = TTLCache(ttl=60, clock=lambda: now)


def exchange_rate(currency):
    rate = cache.get(currency)
    if rate is None:  # a miss: do the slow thing, then remember it
        print("  asking the bank for", currency)
        rate = {"USD": 4.21, "SGD": 3.27}[currency]  # made-up rates
        cache.set(currency, rate)
    return rate


print(exchange_rate("USD"))
print(exchange_rate("USD"))
print(exchange_rate("SGD"))
now = 61.0  # a minute later, the USD answer has expired
print(exchange_rate("USD"))
print("hits:", cache.hits, "misses:", cache.misses)

#   asking the bank for USD
# 4.21
# 4.21
#   asking the bank for SGD
# 3.27
#   asking the bank for USD
# 4.21
# hits: 1 misses: 3

The second USD lookup never asked the bank. SGD had its own entry, because it’s a different key. After 61 seconds the USD entry was gone, so the bank was asked again. A TTL is a promise about staleness: “this answer is at most 60 seconds old.” Picking the number is a product decision as much as a technical one.

Your language already has one

Python ships a cache in the standard library. functools.lru_cache remembers a function’s results by its arguments, up to a size limit:

from functools import lru_cache


@lru_cache(maxsize=2)
def shipping_zone(postcode):
    print("  looking up", postcode)
    return "east" if postcode.startswith("8") else "west"


for code in ["50450", "50450", "88000", "10200", "50450"]:
    shipping_zone(code)
print(shipping_zone.cache_info())

#   looking up 50450
#   looking up 88000
#   looking up 10200
#   looking up 50450
# CacheInfo(hits=1, misses=4, maxsize=2, currsize=2)

The last lookup of 50450 missed even though we’d seen it. The cache had room for two entries, so when 10200 arrived, something had to go. Choosing what goes is eviction, and lru_cache throws out whatever was used least recently (LRU). More on that in Intermediate.

Where caches live

A request for a web page passes several places that can keep a copy, and a hit at any of them skips everything further down:

• The browser keeps files it was told it may keep. A hit sends no request at all.
• A CDN (content delivery network, such as CloudFront or Cloudflare) keeps copies on servers in many cities, close to users.
• App memory is a dictionary inside your server process, like lru_cache. Fastest, but each process has its own copy and they don’t agree.
• A shared cache such as Redis or Memcached is one store every server can reach, for the price of a network round trip.
• The database’s buffer pool keeps hot pages of tables and indexes in RAM (Postgres calls it shared_buffers). It’s free, and saves the disk read but not the query.

What makes data worth caching

Before you cache anything, ask: how wrong can this be, and for how long? “Never, not even for a second” means don’t cache it, or only with invalidation you can prove. “A minute is fine” means you have a TTL. Good candidates are:

• Read far more often than written. A product page read a thousand times between price changes.
• The same answer for many requests, or at least for many requests from the same user.
• Expensive to produce: an aggregate, a call to another service, a rendered page, a model call.
• Small enough to store many of.

Poor candidates: a balance at the moment of payment, one-off reports, anything unique to one request, and anything where an old answer is a security problem, unless your invalidation can’t lag (ezychat.io caches permissions this way; see the last section).

import functools
import time


def ttl_cache(seconds, clock=time.monotonic):
    """Like functools.cache, but every answer expires after `seconds`."""
    def decorate(fn):
        entries = {}  # args -> (value, expires_at)

        @functools.wraps(fn)
        def wrapper(*args):
            now = clock()
            hit = entries.get(args)
            if hit is not None and now < hit[1]:
                return hit[0]
            value = fn(*args)
            entries[args] = (value, now + seconds)
            return value

        wrapper.entries = entries
        return wrapper
    return decorate


now = 0.0
calls = 0


@ttl_cache(30, clock=lambda: now)
def opening_hours(shop):
    global calls
    calls += 1
    return f"{shop}: 9am-6pm"


for t in (0, 10, 29, 31, 40):
    now = float(t)
    opening_hours("kl-central")
print("calls to the slow function:", calls)

# calls to the slow function: 2

Intermediate: patterns, invalidation, eviction and HTTP

Every cache design answers four questions: who fills the cache, what happens on a write, how stale entries get out, and what leaves when it’s full. The first two are the pattern, the third is invalidation, the fourth is eviction.

Cache-aside: the default

In cache-aside (also called lazy loading), the application talks to both the cache and the database. On a read it checks the cache; on a miss it reads the database and stores the result. On a write it updates the database, then deletes the cached copy.

class Database:
    """A stand-in for Postgres that counts how often it's asked."""

    def __init__(self):
        self.rows = {"p1": {"name": "Kopi beans", "price": 3500}}
        self.reads = self.writes = 0

    def read(self, key):
        self.reads += 1
        return self.rows.get(key)

    def write(self, key, row):
        self.writes += 1
        self.rows[key] = row


db = Database()
cache = {}  # stands in for Redis


def get_product(product_id):
    """Cache-aside: the application checks the cache, and fills it on a miss."""
    key = f"product:{product_id}"
    if key in cache:
        return cache[key]
    row = db.read(product_id)
    cache[key] = row  # in Redis: SET key value EX ttl
    return row


def update_price(product_id, price):
    row = dict(db.read(product_id), price=price)
    db.write(product_id, row)                 # 1. the database is the truth: write it first
    cache.pop(f"product:{product_id}", None)  # 2. then delete the cached copy, don't overwrite it


for _ in range(1000):
    get_product("p1")
print("after 1,000 reads, database reads:", db.reads)
update_price("p1", 3200)
print("price after update:", get_product("p1")["price"], "| database reads:", db.reads)

# after 1,000 reads, database reads: 1
# price after update: 3200 | database reads: 3

A thousand reads cost the database one query. The update read the row once more to change it, and the next read refilled the cache with the new price.

Two details in update_price matter more than they look.

Write the database first, then touch the cache. If you delete the key first, another request can miss, read the old row (your write hasn’t happened yet) and put it straight back. Then your write lands, and the cache holds the old price until the TTL ends.

Delete; don’t set. Two writers racing can set their values in the opposite order from their database commits, leaving the loser’s value cached. A delete is safe in any order: the next reader fetches whatever the database says now. The write path also doesn’t need to know how the cached form is built.

Cache-aside is my default. Only data someone asks for gets cached, the cache can be emptied at any time without losing anything, and if it’s down the application can still read the database.

Read-through and write-through

In read-through, the cache itself loads a missing entry, by calling a loader you gave it. Java’s Caffeine LoadingCache works this way, and so does a CDN, which fetches from your origin when it misses. It behaves like cache-aside, with the loading logic in one place.

In write-through, every write goes to the cache and the database together, before the write returns:

class WriteThrough:
    """Every write goes to the cache and the database before it returns."""

    def __init__(self, db):
        self.db, self.cache = db, {}

    def set(self, key, value):
        self.db[key] = value      # the database first: if this fails, the cache never lies
        self.cache[key] = value

    def get(self, key):
        return self.cache[key] if key in self.cache else self.db.get(key)


profiles = WriteThrough(db={})
profiles.set("user:7", {"name": "Aisyah"})
print("write-through, cache and database agree:", profiles.cache == profiles.db)

# write-through, cache and database agree: True

Reads after a write are fresh hits. The costs: every write pays twice, and you cache everything written, read or not. It suits data with a single owner and writer.

Write-behind

Write-behind (or write-back) accepts the write into the cache and saves it to the database later, usually in batches. It turns many small writes into a few big ones:

class WriteBehind:
    """Writes land in the cache now and reach the database later, in batches."""

    def __init__(self, db):
        self.db, self.cache, self.dirty = db, {}, set()
        self.db_writes = 0

    def set(self, key, value):
        self.cache[key] = value
        self.dirty.add(key)       # remember to save it; no database work yet

    def flush(self):              # run by a background worker every second or so
        for key in self.dirty:
            self.db[key] = self.cache[key]
            self.db_writes += 1
        self.dirty.clear()


db = {}
views = WriteBehind(db)
for n in range(1, 1001):          # a popular page is viewed 1,000 times in a second
    views.set("views:home", n)
views.flush()
print("database value:", db["views:home"], "| database writes:", views.db_writes)

views.set("views:home", 1001)     # if the process dies now, this write is gone
print("cache says", views.cache["views:home"], "but database says", db["views:home"])

# database value: 1000 | database writes: 1
# cache says 1001 but database says 1000

A thousand writes became one. The price is in the last line: until the flush, the newest value exists only in the cache, and a crash loses it. Fine for view counters and analytics; I wouldn’t use it for anything a customer would notice losing.

Refresh-ahead

Refresh-ahead reloads a popular entry before it expires, from a background job or from a read that notices expiry is near. It wastes a few refreshes, and it’s the cleanest cure for the stampede in Advanced.

Invalidation: three tools

Invalidation is how stale entries get out. You have three tools, and good designs combine them.

1. TTL. The entry expires on its own. No coordination, always works, but blunt: stale for up to the full TTL after a change.
2. Explicit delete. The code that changes the data deletes the key. Precise, but only if every writer does it: the admin panel, the import script, the migration, the other service. Miss one and the cache is wrong until the TTL saves you.
3. Versioned keys. Put a version number in the key and bump it when the data changes. Readers build the key from the current version, so they stop finding the old entry:

cache = {}
versions = {"catalog:shop-9": 1}  # in production this number lives in the database row


def catalog_key(shop_id):
    return f"catalog:{shop_id}:v{versions[f'catalog:{shop_id}']}"


def get_catalog(shop_id):
    key = catalog_key(shop_id)
    if key not in cache:
        cache[key] = f"catalog built at version {versions[f'catalog:{shop_id}']}"
    return cache[key]


print(get_catalog("shop-9"))
versions["catalog:shop-9"] += 1   # any product edit bumps the version in the same transaction
print(get_catalog("shop-9"))
print("keys in cache:", sorted(cache))  # the old entry is never read again; its TTL cleans it up

# catalog built at version 1
# catalog built at version 2
# keys in cache: ['catalog:shop-9:v1', 'catalog:shop-9:v2']

Versioned keys shine when one change should invalidate many entries: bump one shop’s version and every cached search, page and count for it is invalid at once. The price is reading the current version first.

My rule: always set a TTL, even when you invalidate explicitly. Explicit invalidation makes the cache correct today; the TTL limits the damage from the writer you haven’t thought of. The last section quotes what happened in ezychat.io when that backstop was a day long.

Eviction: who leaves when the cache is full

When the cache is full and a new entry arrives, an eviction policy chooses a victim. Three cover most of what you’ll meet:

• LRU (least recently used) evicts what hasn’t been read for longest, betting that recent use predicts the next.
• LFU (least frequently used) evicts what’s been read the fewest times. It protects favourites, but notices slowly when they change.
• FIFO (first in, first out) evicts the oldest entry, however popular. Simplest, usually worst.

An LRU cache is a hash map plus a doubly linked list: the map finds an entry in one step, and the list keeps entries in order of use, so moving one to the front and dropping one from the back are one step each too.

How much does the policy matter? I simulated it. Real traffic is skewed: a few keys get most requests. A common model is the Zipf distribution, where the second most popular key gets half the requests of the first, the third a third, and so on. Here are 50,000 such requests over 1,000 keys:

import random
from bisect import bisect
from collections import Counter, OrderedDict, deque
from itertools import accumulate


def zipf_stream(n_keys, skew, length, seed=1):
    """Key i (0 = most popular) is requested with probability proportional to 1 / (i + 1) ** skew."""
    rng = random.Random(seed)
    weights = [1 / (i + 1) ** skew for i in range(n_keys)]
    cum = list(accumulate(weights))
    return [bisect(cum, rng.random() * cum[-1]) for _ in range(length)]


def lru_hits(stream, size):
    cache, hits = OrderedDict(), 0
    for key in stream:
        if key in cache:
            hits += 1
            cache.move_to_end(key)
        else:
            cache[key] = True
            if len(cache) > size:
                cache.popitem(last=False)
    return hits / len(stream)


def fifo_hits(stream, size):
    cache, order, hits = set(), deque(), 0
    for key in stream:
        if key in cache:
            hits += 1
        else:
            cache.add(key)
            order.append(key)
            if len(cache) > size:
                cache.discard(order.popleft())
    return hits / len(stream)


def lfu_hits(stream, size):
    counts, hits = {}, 0  # key -> uses while cached
    for key in stream:
        if key in counts:
            hits += 1
            counts[key] += 1
        else:
            if len(counts) >= size:
                del counts[min(counts, key=counts.get)]
            counts[key] = 1
    return hits / len(stream)


stream = zipf_stream(1000, 1.0, 50_000)
top = Counter(stream).most_common(10)
print("share of requests for the top 10 keys:", round(sum(c for _, c in top) / len(stream), 3))
for size in (10, 50, 100, 200):
    print(f"cache {size:>3} of 1000 keys:", "LRU", round(lru_hits(stream, size), 3),
          "LFU", round(lfu_hits(stream, size), 3), "FIFO", round(fifo_hits(stream, size), 3))
uniform = zipf_stream(1000, 0.0, 50_000)
print("uniform, cache 100:", "LRU", round(lru_hits(uniform, 100), 3))

# share of requests for the top 10 keys: 0.391
# cache  10 of 1000 keys: LRU 0.211 LFU 0.346 FIFO 0.182
# cache  50 of 1000 keys: LRU 0.46 LFU 0.565 FIFO 0.406
# cache 100 of 1000 keys: LRU 0.577 LFU 0.66 FIFO 0.521
# cache 200 of 1000 keys: LRU 0.695 LFU 0.747 FIFO 0.644
# uniform, cache 100: LRU 0.103

Three lessons. Skew is what makes small caches work: room for 10% of the keys answered 58% of requests under LRU, against 10% on uniform traffic. On stable popularity, LFU beats LRU beats FIFO. And the gap between policies is smaller than the gain from doubling the cache.

Now let popularity move halfway through, the way it does when a product launches or a story breaks:

before = zipf_stream(1000, 1.0, 25_000, seed=1)
after = [(k + 500) % 1000 for k in zipf_stream(1000, 1.0, 25_000, seed=2)]  # new favourites
shifted = before + after
for name, fn in (("LRU", lru_hits), ("LFU", lfu_hits), ("FIFO", fifo_hits)):
    total = fn(shifted, 100)
    first = fn(before, 100)
    second_half = (total * len(shifted) - first * len(before)) / len(after)
    print(f"{name}: before the shift {first:.3f}, after it {second_half:.3f}")

# LRU: before the shift 0.578, after it 0.578
# LFU: before the shift 0.659, after it 0.053
# FIFO: before the shift 0.522, after it 0.521

LFU collapsed from 66% to 5%. The old favourites hold large counts, so every newcomer arrives with a count of 1 and is evicted next. The cache is frozen full of yesterday’s news. That’s why production LFU implementations age their counters: Redis decays an idle key’s counter as time passes, and Caffeine’s W-TinyLFU keeps a frequency sketch that it periodically halves. Plain LFU, as taught, is a trap.

Try both effects yourself. All three policies see the same requests, and the bars at the bottom compare them:

HTTP caching: headers that browsers and CDNs obey

Browsers and CDNs are caches you instruct with the Cache-Control response header. The directives you’ll use most:

The most useful trick in the table is splitting max-age from s-maxage. max-age=0, s-maxage=86400 says: browsers, check every time; CDN, keep it a day. The CDN is close to the user, so checking is cheap, and the CDN is the cache you can tell to forget.

Checking is called revalidation. The server sends a fingerprint of the response, an ETag. Next time the browser sends it back in If-None-Match, and if nothing changed the server answers 304 Not Modified with no body:

import hashlib
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.error import HTTPError
from urllib.request import Request, urlopen

BODY = b"<h1>Price list</h1>"
ETAG = '"' + hashlib.sha256(BODY).hexdigest()[:16] + '"'


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.headers.get("If-None-Match") == ETAG:
            self.send_response(304)                     # "what you have is still right"
            self.send_header("ETag", ETAG)
            self.end_headers()
            return
        self.send_response(200)
        self.send_header("ETag", ETAG)
        self.send_header("Cache-Control", "public, max-age=0, must-revalidate")
        self.send_header("Content-Length", str(len(BODY)))
        self.end_headers()
        self.wfile.write(BODY)

    def log_message(self, *args):
        pass


server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
url = f"http://127.0.0.1:{server.server_port}/prices"

first = urlopen(url)
print(first.status, first.headers["ETag"], len(first.read()), "bytes")
try:
    urlopen(Request(url, headers={"If-None-Match": first.headers["ETag"]}))
except HTTPError as revalidated:  # urllib reports a 304 as an "error"
    print(revalidated.code, revalidated.headers["ETag"], len(revalidated.read()), "bytes")
server.shutdown()

# 200 "918a726bc29fc23a" 19 bytes
# 304 "918a726bc29fc23a" 0 bytes

A 304 still costs a round trip, but no body, and often none of the expensive work.

Files that never change once published can skip revalidation entirely. Build tools put a hash of the content in the file name (app.3f9a2c.js), so a changed file gets a new URL and each URL’s content is permanent: cache it for a year with immutable. The HTML that references it is what changes, and that stays on a short leash. This site uses exactly that split.

To remove something from a CDN, you invalidate (purge) it by path or wildcard, and on some CDNs by tag. A purge takes seconds to minutes to reach every edge, so for assets prefer new URLs: a new name can’t be stale anywhere.

from collections import OrderedDict


def trace(policy, requests, size=3):
    cache, hits = OrderedDict(), []
    for key in requests:
        hit = key in cache
        hits.append(hit)
        if hit and policy == "lru":
            cache.move_to_end(key)
        if not hit:
            cache[key] = True
            if len(cache) > size:
                cache.popitem(last=False)
    return ["hit" if h else "miss" for h in hits], list(cache)


requests = [1, 2, 3, 1, 4, 2, 5, 1]
print("LRU: ", *trace("lru", requests))
print("FIFO:", *trace("fifo", requests))

# LRU:  ['miss', 'miss', 'miss', 'hit', 'miss', 'miss', 'miss', 'miss'] [2, 5, 1]
# FIFO: ['miss', 'miss', 'miss', 'hit', 'miss', 'hit', 'miss', 'miss'] [4, 5, 1]

Advanced: stampedes, races and the arithmetic of hit ratios

What is a cache stampede?

A cache stampede (or thundering herd) happens when a popular entry expires and many requests miss at once. Each does what cache-aside says: read the database and fill the cache. If a key is read 200 times a second and takes half a second to rebuild, about 100 requests run the same query before the first finishes. The database slows down under that load, which keeps the key missing for longer.

Here it is with 50 threads standing in for 50 concurrent requests:

import threading
import time

db_queries = 0
count_lock = threading.Lock()


def slow_query():
    """The expensive thing we cache: a 200 ms report query."""
    global db_queries
    with count_lock:
        db_queries += 1
    time.sleep(0.2)
    return "top sellers"


def naive(cache, key):
    value = cache.get(key)
    if value is None:
        value = slow_query()
        cache[key] = value
    return value


fill_lock = threading.Lock()


def single_flight(cache, key):
    value = cache.get(key)
    if value is not None:
        return value
    with fill_lock:                  # one caller computes; the rest queue here
        value = cache.get(key)       # check again: someone may have filled it while we waited
        if value is None:
            value = slow_query()
            cache[key] = value
    return value


def stampede(get, n=50):
    global db_queries
    db_queries, cache = 0, {}        # the hot key has just expired
    start = threading.Barrier(n)

    def request():
        start.wait()
        get(cache, "report:top-sellers")

    threads = [threading.Thread(target=request) for _ in range(n)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    return db_queries


print("naive:        ", stampede(naive), "database queries for 50 requests")
print("single-flight:", stampede(single_flight), "database query for 50 requests")

# naive:         50 database queries for 50 requests
# single-flight: 1 database query for 50 requests

Single-flight (or request coalescing) means one caller does the work and the rest wait for its result. The second cache.get inside the lock matters: without it, every waiter would rebuild the value in turn once it got the lock. Go’s golang.org/x/sync module ships this as singleflight; in production use one lock per key, not one for everything.

A lock in one process only coalesces that process’s requests; twenty servers still send twenty queries. To coalesce across servers, take the lock in Redis:

import threading
import time
import uuid

import redis

r = redis.Redis(port=6399)
RELEASE = r.register_script("""
if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end
return 0
""")
queries = 0


def build_report():
    global queries
    queries += 1
    time.sleep(0.2)
    return b"top sellers"


def get_report(key="report:top-sellers", ttl=60):
    value = r.get(key)
    if value is not None:
        return value
    token = uuid.uuid4().hex
    # Only one caller across every server gets the lock; it expires on its own if that caller dies.
    if r.set(f"lock:{key}", token, nx=True, px=5000):
        try:
            value = build_report()
            r.set(key, value, ex=ttl)
            return value
        finally:
            RELEASE(keys=[f"lock:{key}"], args=[token])  # delete the lock only if it's still ours
    for _ in range(50):              # everyone else waits briefly for the winner's result
        time.sleep(0.02)
        value = r.get(key)
        if value is not None:
            return value
    return build_report()            # the winner is too slow: give up waiting and compute it


r.delete("report:top-sellers", "lock:report:top-sellers")
start = threading.Barrier(50)
threads = [threading.Thread(target=lambda: (start.wait(), get_report())) for _ in range(50)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print("database queries for 50 concurrent misses:", queries)

# database queries for 50 concurrent misses: 1

Every part of that lock handles a failure. nx=True lets only one caller create it. px=5000 makes it expire, so a crashed holder can’t block the key forever. The release script deletes the lock only if it still holds our token, so a slow winner can’t delete someone else’s. And waiters give up after a second and compute it themselves, because a cache that makes requests wait forever is worse than none.

A gentler variant serves the stale value while one caller refreshes it: give entries a soft expiry that starts a refresh and a hard one after which the value is gone. That’s stale-while-revalidate, applied inside your application.

Early refresh without a lock

Locks have moving parts. An alternative is to refresh before expiry and pick the refresher at random, so that usually only one request does it. The best-known version is XFetch, from “Optimal Probabilistic Cache Stampede Prevention” (Vattani, Chierichetti and Lowenstein, VLDB 2015). Each read rolls a die, weighted so the chance of refreshing rises as expiry approaches:

import math
import random


def should_refresh_early(now, expiry, delta, beta=1.0, rng=random.random):
    """XFetch: refresh before `expiry` with a probability that rises as it gets close.

    delta is how long the recompute takes. 1 - rng() is in (0, 1], so its log is <= 0
    and the subtraction moves `now` forward by a random, usually small, amount."""
    return now - delta * beta * math.log(1 - rng()) >= expiry


rng = random.Random(7).random
expiry, delta = 60.0, 0.5   # cached at t=0 for 60 s; rebuilding takes 0.5 s
for seconds_left in (10, 3, 1, 0.5, 0.1):
    trials = 100_000
    hits = sum(should_refresh_early(expiry - seconds_left, expiry, delta, rng=rng) for _ in range(trials))
    print(f"{seconds_left:>4} s before expiry: {hits / trials:.4f} chance each request refreshes")

#   10 s before expiry: 0.0000 chance each request refreshes
#    3 s before expiry: 0.0027 chance each request refreshes
#    1 s before expiry: 0.1351 chance each request refreshes
#  0.5 s before expiry: 0.3678 chance each request refreshes
#  0.1 s before expiry: 0.8202 chance each request refreshes

The chance is exp(-seconds_left / (delta × beta)). Scaling by delta, the rebuild time, is the clever part: a slow rebuild starts earlier. beta above 1 refreshes earlier still. A busy key is almost always refreshed by one of its own readers before it expires, and a quiet key expires normally, which is fine: a quiet key can’t stampede.

Pure XFetch doesn’t guarantee one refresher: requests arriving while the first refresh runs can roll the die too. I simulated 2,000 refresh cycles at 200 requests a second with a half-second rebuild: on average 2.65 recomputes per refresh, 18 in the worst cycle. Far better than a hundred, and a “refresh in progress” flag brings it to one, which is what the demo does:

No protection sends a query per waiting request. The lock sends one, but everyone arriving during the rebuild waits for it. Early refresh sends a few more queries than the lock, because it refreshes before it has to, and in exchange nobody waits. For a hot key where latency matters, that’s the trade I’d take.

Negative caching

What should the cache do when the answer is “not found”? Caching nothing is harmless until a broken client, a crawler or an attacker asks for the same missing key a thousand times a second. Negative caching stores the absence too, with a shorter TTL:

MISSING = object()   # "we looked, and there's nothing": different from "we haven't looked"
db_reads = 0
users = {"42": "Aisyah"}
cache = {}           # key -> (value, expires_at)


def find_user(user_id, now, ttl=300, negative_ttl=30):
    global db_reads
    entry = cache.get(user_id)
    if entry and now < entry[1]:
        value = entry[0]
        return None if value is MISSING else value
    db_reads += 1
    value = users.get(user_id)
    if value is None:
        cache[user_id] = (MISSING, now + negative_ttl)   # remember the absence, briefly
        return None
    cache[user_id] = (value, now + ttl)
    return value


for second in range(100):   # a broken client asks for user 999 every second
    find_user("999", now=float(second))
print("database reads for 100 lookups of a missing user:", db_reads)
users["999"] = "Ravi"       # the user signs up at t=100
print("at t=100:", find_user("999", now=100.0), "| at t=121:", find_user("999", now=121.0))

# database reads for 100 lookups of a missing user: 4
# at t=100: None | at t=121: Ravi

A hundred lookups cost four reads. The cost shows in the last line: Ravi signed up at t=100 and stayed invisible until the negative entry expired. Keep the negative TTL short, and delete the key on create, as you would on update.

One rule I’d never bend: don’t cache errors as answers. A timeout isn’t “not found,” and caching it turns a one-second blip into a minute of wrong answers.

Hot keys

Sharding spreads keys across servers, but each key lives on one. A hot key (the home page, a celebrity’s profile, a flash-sale product) can max out one node while the rest sit idle. The fixes, in the order I’d reach for them:

1. A tiny in-process cache in front, for a second or two. A key read 50,000 times a second across 20 servers becomes 20 Redis reads a second.
2. Replicate the key as product:7#0 to product:7#7: read a random copy, write and delete all eight.
3. Find them first with redis-cli --hotkeys, which works only under an LFU eviction policy, since that’s what makes Redis track frequency.

The race between update and invalidate

Cache-aside has one race every team meets eventually. A reader misses, reads the database, then pauses (a garbage collection, a slow network). Meanwhile a writer updates the row and deletes the key. Then the reader wakes and writes the old value into the cache:

db = {"price:p1": 3500}
cache = {}

# Each step is one thing one actor does. The order is the one a busy server can produce.
steps = [
    ("reader", "cache miss on price:p1"),
    ("reader", "reads the database: 3500"),
    ("writer", "updates the database to 3200"),
    ("writer", "deletes price:p1 from the cache"),
    ("reader", "writes what it read (3500) into the cache"),
]
read_value = None
for who, what in steps:
    if what.startswith("reads"):
        read_value = db["price:p1"]
    elif what.startswith("updates"):
        db["price:p1"] = 3200
    elif what.startswith("deletes"):
        cache.pop("price:p1", None)
    elif what.startswith("writes"):
        cache["price:p1"] = read_value
    print(f"{who:>6}: {what}")

print("database:", db["price:p1"], "| cache:", cache["price:p1"], "<- stale until the TTL runs out")

# reader: cache miss on price:p1
# reader: reads the database: 3500
# writer: updates the database to 3200
# writer: deletes price:p1 from the cache
# reader: writes what it read (3500) into the cache
# database: 3200 | cache: 3500 <- stale until the TTL runs out

No single step is a bug, the window is small, and nobody can reproduce it when it happens. Four defenses:

• A TTL, so the wrong value can’t last forever. This is the minimum.
• Delete after the transaction commits. Deleting while it’s still open lets a reader refill the key from the old, still-committed row. ezychat.io has a long comment about exactly this, quoted in the last section.
• Versions. Store the row’s version with the value and refuse to overwrite a newer one, or use versioned keys so the late write lands where nobody reads.
• Leases. Facebook’s paper “Scaling Memcache at Facebook” (NSDI 2013) describes them: a miss hands the reader a lease token, a delete cancels outstanding leases, and a set without a valid lease is refused.

Leases are simple to model:

import itertools


class LeasingCache:
    """A miss hands out a lease; a delete cancels it; a set needs a lease that is still valid."""

    def __init__(self):
        self.values, self.leases = {}, {}
        self.tokens = itertools.count(1)

    def get(self, key):
        if key in self.values:
            return self.values[key], None
        token = next(self.tokens)
        self.leases[key] = token
        return None, token

    def delete(self, key):
        self.values.pop(key, None)
        self.leases.pop(key, None)   # any reader holding a lease now holds a dead one

    def set(self, key, value, token):
        if self.leases.get(key) != token:
            return False             # the data changed after this reader looked: refuse the old value
        del self.leases[key]
        self.values[key] = value
        return True


db = {"price:p1": 3500}
cache = LeasingCache()
_, lease = cache.get("price:p1")        # reader: miss, gets lease 1
old = db["price:p1"]                    # reader: reads 3500
db["price:p1"] = 3200                   # writer: updates the database
cache.delete("price:p1")                # writer: invalidates, which kills lease 1
print("reader's late set accepted?", cache.set("price:p1", old, lease))
value, lease = cache.get("price:p1")    # the next reader misses and fills it correctly
print("next reader's set accepted?", cache.set("price:p1", db["price:p1"], lease), "| cached:", cache.get("price:p1")[0])

# reader's late set accepted? False
# next reader's set accepted? True | cached: 3200

The paper uses leases against stampedes too, by handing out a lease per key only every so often.

Serialization isn’t free

Every object you put in Redis is serialized (turned into bytes) and parsed on the way out. For small values that’s nothing; for 200 products it isn’t:

import json
import pickle
import timeit

# A typical cached API answer: 200 products with a handful of fields each.
products = [
    {"id": i, "name": f"Product {i}", "price_cents": 1000 + i, "tags": ["coffee", "beans"], "in_stock": i % 3 != 0}
    for i in range(200)
]
as_json = json.dumps(products).encode()
as_pickle = pickle.dumps(products, protocol=pickle.HIGHEST_PROTOCOL)  # trusted, local data only

for name, dumps, loads, blob in (
    ("json", lambda: json.dumps(products).encode(), lambda: json.loads(as_json), as_json),
    ("pickle", lambda: pickle.dumps(products, protocol=pickle.HIGHEST_PROTOCOL), lambda: pickle.loads(as_pickle), as_pickle),
):
    n = 2000
    encode_us = timeit.timeit(dumps, number=n) / n * 1e6
    decode_us = timeit.timeit(loads, number=n) / n * 1e6
    print(f"{name:>6}: {len(blob):>6} bytes, encode {encode_us:.0f} us, decode {decode_us:.0f} us")

#   json:  20647 bytes, encode 113 us, decode 113 us
# pickle:   8353 bytes, encode 38 us, decode 54 us

Decoding that JSON took 113 µs on my laptop, more than the 96 µs round trip that fetched it, so the hit cost twice what the latency table suggested. For big values on a hot path, measure the decode, cache smaller pieces, or keep a parsed copy in process memory.

Pickle is faster and smaller, and I still wouldn’t use it in a shared cache: unpickling runs code chosen by whoever wrote the bytes, so anyone who can write to your Redis can run code in every server that reads it. JSON can only produce data.

The quieter problem is schema drift: you deploy code that renames a field while the cache is full of entries in the old shape, and the new code crashes reading them. Put a schema version in the key (ent:v3:…) and bump it when the shape changes, so new code never looks at old entries. ezychat.io learned this one the hard way; the comment is in the last section.

How big should the cache be?

With hit ratio h, a hit costing t_hit and a miss costing t_hit + t_miss (you checked the cache first):

def effective_ms(hit_ratio, hit_ms, miss_ms):
    """Average time per read. A miss pays for the cache check and the database."""
    return hit_ratio * hit_ms + (1 - hit_ratio) * (hit_ms + miss_ms)


reads_per_second = 20_000
for h in (0.0, 0.80, 0.90, 0.99, 0.999):
    print(f"hit ratio {h:>5.1%}: {effective_ms(h, hit_ms=0.5, miss_ms=20):5.2f} ms per read, "
          f"database sees {reads_per_second * (1 - h):>6,.0f} reads/s")

# hit ratio  0.0%: 20.50 ms per read, database sees 20,000 reads/s
# hit ratio 80.0%:  4.50 ms per read, database sees  4,000 reads/s
# hit ratio 90.0%:  2.50 ms per read, database sees  2,000 reads/s
# hit ratio 99.0%:  0.70 ms per read, database sees    200 reads/s
# hit ratio 99.9%:  0.52 ms per read, database sees     20 reads/s

Read the last column. Latency barely improves past 99%, but database load drops tenfold with every extra nine: 200 queries a second at 99%, 20 at 99.9%.

That’s also the most dangerous property of a good cache. At 99.9% the database is sized for 20 reads a second; empty the cache (a restart, a flush, a failover) and it gets 20,000. The cache is now load-bearing, and its failure is your database’s failure.

To estimate a hit ratio you need the working set and the skew. LRU over 100,000 keys and a million requests (cold start included):

Skew matters more than size: at skew 1.0, a cache of 1% of the keys beats 10% at 0.8. Measure your access log before buying memory, and measure entries too: on Redis 7.2, a 1 KB JSON value with a 32-byte key and a TTL used 1,336 bytes (MEMORY USAGE), about 300 bytes of overhead.

Redis: maxmemory and eviction policies

Redis caps its memory with maxmemory, and maxmemory-policy decides what happens at the cap:

• noeviction: refuse writes. This is the open-source default, which surprises people who think of Redis as a cache.
• allkeys-lru, allkeys-lfu, allkeys-random: evict any key.
• volatile-lru, volatile-lfu, volatile-random, volatile-ttl: evict only keys that have a TTL (volatile-ttl picks the one closest to expiring).

Here’s what the first two do when about 20 MB of entries are pushed into a 5 MB budget:

import json

import redis

r = redis.Redis(port=6399)
r.flushall()
value = json.dumps({"id": 1, "name": "x" * 1000})          # about 1 KB of JSON
r.set("product:1", value, ex=300)
print("bytes Redis uses for one 1 KB entry:", r.memory_usage("product:1"))

r.config_set("maxmemory", "5mb")
for policy in ("noeviction", "allkeys-lru"):
    r.flushall()
    r.config_set("maxmemory-policy", policy)
    r.config_resetstat()
    stored = 0
    try:
        for i in range(20_000):                              # about 20 MB of data into a 5 MB budget
            r.set(f"product:{i}", value, ex=300)
            stored += 1
    except redis.exceptions.ResponseError as e:
        print(f"{policy}: write {stored + 1} failed: {str(e)[:40]}...")
    info = r.info()
    print(f"{policy}: keys kept {r.dbsize()}, evicted {info['evicted_keys']}")
r.config_set("maxmemory", "0")

# bytes Redis uses for one 1 KB entry: 1336
# noeviction: write 2910 failed: command not allowed when used memory > '...
# noeviction: keys kept 2909, evicted 0
# allkeys-lru: keys kept 2902, evicted 17080

With noeviction, writes started failing after about 2,900 entries (the count varies a little by run), so the app must handle errors from a component whose job is to be disposable. With allkeys-lru, Redis quietly dropped 17,080 old entries.

Redis’s LRU is approximate: it samples maxmemory-samples keys (5 by default) and evicts the best candidate. Its LFU keeps an 8-bit logarithmic counter per key and decays it over time (lfu-decay-time, a minute by default), the aging our plain LFU lacked.

The policy gets serious when one Redis does more than caching. Queues, sessions and locks often share the node, usually without a TTL:

import json

import redis

r = redis.Redis(port=6399)
value = json.dumps({"id": 1, "name": "x" * 1000})
r.config_set("maxmemory", "5mb")
for policy in ("allkeys-lru", "volatile-lru"):
    r.flushall()
    r.config_set("maxmemory-policy", policy)
    r.rpush("queue:emails", *[f"job {i}" for i in range(100)])   # work, not cache: no TTL
    for i in range(20_000):
        r.set(f"product:{i}", value, ex=300)                    # cache entries: all have a TTL
        if i % 100 == 0:
            r.get(f"product:{i // 2}")                          # keep some cache keys warm
    print(f"{policy}: queued jobs left {r.llen('queue:emails')}")
r.config_set("maxmemory", "0")
r.config_set("maxmemory-policy", "noeviction")

# allkeys-lru: queued jobs left 0
# volatile-lru: queued jobs left 100

Under allkeys-lru, the untouched queue was evicted and 100 jobs vanished without an error. Under volatile-lru, only keys with a TTL could go. My recommendation: a dedicated cache gets allkeys-lru (or allkeys-lfu for stable popularity); a shared node gets a volatile-* policy, a TTL on every cache write and an alarm on evictions. Better still, give queues their own instance.

import asyncio


class SingleFlight:
    """Concurrent callers asking for the same key share one call to `fn`."""

    def __init__(self):
        self.running = {}  # key -> asyncio.Future for the call in progress

    async def do(self, key, fn):
        if key in self.running:
            return await asyncio.shield(self.running[key])   # join the call already running
        future = asyncio.get_running_loop().create_future()
        self.running[key] = future
        try:
            result = await fn()
            future.set_result(result)
            return result
        except Exception as e:
            future.set_exception(e)                         # every waiter sees the same error
            raise
        finally:
            del self.running[key]                           # the next miss starts a fresh call


calls = 0


async def load_report():
    global calls
    calls += 1
    await asyncio.sleep(0.2)
    return {"top": ["kopi", "teh"]}


async def main():
    group = SingleFlight()
    results = await asyncio.gather(*(group.do("report", load_report) for _ in range(100)))
    print("callers:", len(results), "| calls to load_report:", calls, "| all equal:", all(r == results[0] for r in results))
    await group.do("report", load_report)
    print("a later miss starts a new call:", calls)


asyncio.run(main())

# callers: 100 | calls to load_report: 1 | all equal: True
# a later miss starts a new call: 2

Expert: layers, attackers, scale and LLMs

Multi-layer caches and consistency

Big systems stack caches: an in-process L1 per server for the hottest keys, a shared L2 such as Redis behind it, then the database. L1 absorbs hot keys; L2 holds the long tail and keeps servers in agreement.

Staleness adds up. An L1 entry copied from L2 can hold a value that was already five minutes old. Worst-case staleness is the sum of the TTLs along the path, five minutes and five seconds here. Say that number when someone asks how fresh the data is.

Invalidate from the bottom up: database, then L2, then L1s. Clear L1 first and a server can refill it from the L2 copy you haven’t deleted yet.

Fan-out messages get lost. Redis pub/sub is fire-and-forget, so a server that’s reconnecting never hears the message. Short L1 TTLs are the backstop. For a stronger guarantee, send invalidations through something durable, such as a stream read with offsets or the database’s change feed.

Users notice one more thing: reading their own writes. Someone saves their profile, reloads, and sees the old name from another server’s L1. Send the writer’s next read past the caches, or have the writing server update its own L1.

Cache poisoning and other security problems

A cache serves one stored answer to many people, so whoever controls what’s stored controls what everyone sees. That’s cache poisoning. James Kettle’s 2018 PortSwigger research, “Practical Web Cache Poisoning,” showed that headers a CDN leaves out of its cache key (such as X-Forwarded-Host) can still change what the origin generates: one malicious request gets cached under the normal URL and served to every later visitor. The defense is to leave nothing that changes the response out of the key: strip unknown headers at the edge and list the ones that matter in Vary.

Its sibling, web cache deception (Omer Gil, Black Hat USA 2017), tricks a CDN into caching a private page, for example via /account/settings/x.css on a server that ignores the trailing part. The CDN sees .css and caches the victim’s page for the next visitor. The origin should mark private responses private or no-store, and the CDN should obey the origin rather than guess from extensions.

Inside your application, the same mistake looks ordinary: a key that forgets someone. Cache the dashboard under dashboard instead of dashboard:{org_id}:{user_id} and tenant A sees tenant B’s numbers. Leave the permission scope out of a search cache and a junior employee sees what only managers should. When the key gets long, hash it, but hash all of it.

Also bound keys built from user input (hash or length-limit them), or an attacker fills your memory with junk.

Distributed caches and sharding

One Redis node eventually runs out of memory or throughput. The naive way to spread keys is hash(key) % N, until the day you add a node:

import hashlib


def node_for(key, nodes):
    h = int.from_bytes(hashlib.md5(key.encode()).digest()[:8], "big")
    return h % nodes


keys = [f"product:{i}" for i in range(100_000)]
moved = sum(node_for(k, 4) != node_for(k, 5) for k in keys)
print(f"adding a fifth cache node moves {moved / len(keys):.0%} of keys")

# adding a fifth cache node moves 80% of keys

Moving 80% of keys means 80% of the cache misses at once: a stampede triggered by adding capacity. Consistent hashing puts keys and nodes on a ring so that adding a node moves only about one key in N. Redis Cluster hashes keys into 16,384 fixed slots and moves whole slots between nodes; braces in a key ({org:7}:dashboard) make only that part count, keeping related keys together.

Replicas add read throughput, but lag their primary by a few milliseconds: one more source of staleness to add to the sum.

Cache warming

An empty cache is cold. After a key-format change, a flush, a failover or a new region, every request misses. At a 99.9% hit ratio that’s a thousandfold jump in database load. Ways to survive it:

• Ask the capacity question first. If the database can’t serve full traffic, a cold cache is an outage you haven’t had yet.
• Warm from evidence: load the top few thousand keys from yesterday’s logs before taking traffic. Under skew, they’re most of the hits.
• Shift traffic gradually, 5% then 20%, so misses arrive at a rate the database can take.
• Coalesce: single-flight makes a cold start cost one query per key, not per request.
• Protect the database anyway with a connection pool sized to what it can take and fast 503s when it’s full.

Measuring and alerting on hit ratio

Redis counts hits and misses for you:

import redis

r = redis.Redis(port=6399)
r.flushall()
r.config_resetstat()
for i in range(100):
    r.set(f"product:{i}", "x", ex=300)
for i in range(400):
    r.get(f"product:{i % 125}")      # 100 of the 125 keys asked for exist

stats = r.info("stats")
hits, misses = stats["keyspace_hits"], stats["keyspace_misses"]
print(f"hits {hits}, misses {misses}, hit ratio {hits / (hits + misses):.1%}")

# hits 325, misses 75, hit ratio 81.2%

That number is server-wide: it mixes every cache, lock and session on the node, and a 30% cache can hide behind a 99% one. Count hits and misses in your application, per cache name. What I’d alert on:

• A drop in a cache’s hit ratio against the same hour last week: usually a changed key format, a shorter TTL or evictions.
• Evictions, if the node also holds anything that isn’t a cache.
• Database reads per second, the number the cache exists to keep down.
• The CDN’s hit ratio. A drop after a deploy often means a new query parameter or header is splitting the cache key.

A hit ratio can mislead: 99% on cheap keys and 50% on expensive ones may do less good than the reverse. When it matters, weight misses by cost.

Caching in LLM apps

Model calls are slow and billed per token, and three very different things get called caching.

Provider prompt caching happens at the provider. When requests share a long prefix (system prompt, tool definitions, a document), the provider reuses its processed form and charges less. OpenAI and Gemini do it automatically for long repeated prefixes; Anthropic’s API asks you to mark where the cacheable prefix ends. No invalidation and no wrong answers; the only way to lose it is to put something that changes, like a timestamp, near the start. Stable parts first, per-request parts last.

Exact response caching stores the answer under a key built from the full request. It’s safe for deterministic-in-spirit tasks (extract these fields, classify this message) when the key holds everything that shaped the answer: model ID, prompt version, tools, retrieved context, tenant. Leave one out and a prompt change is ignored for a whole TTL.

Semantic caching returns the answer to a similar cached question, found by embedding. It raises the hit ratio, and it’s where I’d be most careful, because similar isn’t same:

import math
from collections import Counter


def similarity(a, b):
    """Cosine similarity of word counts: a crude stand-in for comparing embeddings."""
    x, y = Counter(a.lower().split()), Counter(b.lower().split())
    dot = sum(x[w] * y[w] for w in x)
    return dot / (math.sqrt(sum(v * v for v in x.values())) * math.sqrt(sum(v * v for v in y.values())))


cached_question = "can i cancel my booking for tomorrow"
for question in ("can i cancel my booking for tomorrow please",
                 "can i change my booking for tomorrow",
                 "what time do you open tomorrow"):
    print(f"{similarity(cached_question, question):.2f}  {question}")

# 0.94  can i cancel my booking for tomorrow please
# 0.86  can i change my booking for tomorrow
# 0.15  what time do you open tomorrow

Real embeddings beat word counts, but the problem stays: “cancel” and “change” are one word apart and need different answers, and a threshold loose enough for rephrasings will catch some different questions. Add the other costs (an embedding and a vector search on every lookup, a shared index that can leak across tenants, answers that go stale with the knowledge base) and my default is: prompt caching always, exact-match where the task allows, semantic only for FAQ-style answers, per tenant, with a strict threshold and evals that include near-miss questions.

The interview walk-through

“Add caching” shows up inside almost every design interview. A full pass, with numbers:

The prompt: a product catalogue API for an online store. 5 million products, about 2 KB of JSON each. 40,000 reads a second at peak, about 50 product updates a second. Prices must be correct within a minute. Stock is checked again at checkout.

1. Requirements and the staleness budget. Product details may be minutes old, price at most 60 seconds; stock is advisory on the page and authoritative at checkout. Writing these down is half the design.

2. Estimates.

• Data: 5M × 2 KB = 10 GB, about 11.5 GB with Redis overhead. That fits a small cluster, so cache everything that’s requested.
• Misses: with everything cacheable, misses come from expiry and invalidation. Each key that’s read often enough to stay cached costs about one miss per TTL. Say a million products are that popular: with a 10-minute TTL that’s 1,000,000 / 600 ≈ 1,700 misses a second; with a one-hour TTL, about 280, plus 50 a second from updates. The longer TTL is affordable because writes invalidate explicitly.
• Database: about 330 reads a second instead of 40,000. Easy for one Postgres primary with a replica.

3. Design. Cache-aside on Redis (three shards, one replica each), keyed product:{id}:v{schema}, TTL one hour. Writes update Postgres, then delete the key after commit. A small in-process L1 (two seconds) in each API server for the hottest products.

4. The price problem. With a one-hour TTL, a lost invalidation shows a wrong price for an hour, breaking the 60-second rule. Deliver invalidations reliably (an outbox row in the same transaction, applied by a worker), and also give price its own key with a 30-second TTL as the backstop.

5. Deep dives the interviewer may pick.

• Flash sale: one product gets 20,000 reads a second. L1 absorbs most of it; single-flight plus early refresh on that key stops a stampede when it expires.
• Redis outage: the database can’t take 40,000 reads a second, so fail open behind a concurrency limit, serve L1 entries past their TTL, and shed load with 503.
• New region or flushed cluster: warm the top keys from logs, shift traffic gradually.
• CDN: anonymous product pages can be cached at the edge with s-maxage=30 and stale-while-revalidate, which takes most read traffic off the API entirely.
• Observability: per-cache hit ratio, database reads per second, evictions, and price-staleness sampling.

In production: the caches behind ezychat.io and this site

Two sets of real decisions: the CDN headers on this site, which you can check yourself, and the caches inside ezychat.io, whose code comments record why each TTL is what it is. Excerpts are trimmed from the real code.

This site: two Cache-Control policies and one invalidation

tarif.dev is static HTML in an S3 bucket behind CloudFront. The deploy workflow uploads it in two passes with different headers, then clears the CDN:

# Hashed build assets never change: cache them for a year.
- name: Upload fingerprinted assets
  run: |
    aws s3 sync dist/_astro s3://tarif-dev-portfolio/_astro \
      --delete \
      --cache-control "public, max-age=31536000, immutable"

# Everything else (HTML, feeds, sitemap, OG images, CV) must revalidate,
# so visitors and crawlers always get the current version.
- name: Upload pages and files
  run: |
    aws s3 sync dist s3://tarif-dev-portfolio \
      --delete \
      --exclude "_astro/*" \
      --cache-control "public, max-age=0, s-maxage=86400, must-revalidate"

- name: Invalidate CloudFront
  run: |
    aws cloudfront create-invalidation \
      --distribution-id ${{ secrets.CLOUDFRONT_DISTRIBUTION_ID }} \
      --paths "/*"

• /_astro/ files get a year and immutable. Astro puts a content hash in every built file’s name (the home page portrait is portrait.DPDHmL-F_1KEcFr.jpg), so a URL’s content never changes. immutable tells browsers not to revalidate even on reload; Firefox and Safari honor it, and Chrome reached the same result by changing how reload works.
• Everything else gets max-age=0, s-maxage=86400, must-revalidate. HTML can’t carry a hash in its URL, so browsers check every time, and because S3 sends an ETag the check is a cheap 304. CloudFront may keep its copy for a day, so the check is answered at the edge. must-revalidate forbids serving a stale copy without checking.
• The invalidation makes the day-long s-maxage safe. Without it, CloudFront could serve yesterday’s HTML for a day after a deploy. One wildcard path is simpler than working out which pages changed, and since the workflow also runs daily so future-dated posts appear on their day, “everything may have changed” is the honest assumption.
• Assets, then pages, then invalidation. New HTML names new asset files, so those must be in the bucket first.

You can watch it work. While writing this, I requested the home page twice: x-cache: Miss from cloudfront, then x-cache: Hit from cloudfront. Sending its ETag back in If-None-Match got HTTP/2 304, x-cache: Hit from cloudfront and age: 9: the edge answered from a nine-second-old copy without touching S3. The portrait came back with cache-control: public, max-age=31536000, immutable.

One trade-off I accept: --delete removes the previous build’s hashed files, so an old tab that lazily requests an old file would miss. The site uses full page loads, so every navigation brings fresh HTML that names the new files; a long-lived single-page app should keep old assets around for a while.

ezychat.io: the model policy, cached for a minute

Which model handles which step of an AI reply is data in ezychat.io: a model_policies table changed only by database migrations, as the production note on pinning model IDs explains. Every AI reply resolves it, so it’s read constantly and written almost never:

_POLICY_TTL = 60


async def get_active_policy(tier: str) -> ResolvedPolicy:
    """Redis-cached active policy row for a tier; fail-open to the DB."""
    try:
        redis = get_redis()
        raw = await redis.get(_policy_key(tier))
        if raw is not None:
            return ResolvedPolicy.from_json(raw)
        policy = await _load_from_db(tier)
        await redis.set(_policy_key(tier), policy.to_json(), ex=_POLICY_TTL)
        return policy
    except Exception:
        return await _load_from_db(tier)

The docstring explains why a TTL, not explicit invalidation:

model_policies is platform-global reference data written only by migrations/platform tooling, so the cache is a short-TTL row snapshot per tier (the authz _VER_TTL precedent: out-of-band writers can only be picked up by expiry, and 60s bounds that window) rather than a version-key pair.

A migration runs outside the application, so no app code is there to delete the key. The TTL is the invalidation, and 60 seconds is the promise: a new policy serves within a minute, with no redeploy. An invalidate_policy_cache() exists for in-app tooling, and its docstring is clear about which mechanism carries the weight: “expiry covers the migration path regardless.”

Is it a stampede risk? There are two keys, one per tier, read by every AI reply, so they’re as hot as keys get. But a miss costs one indexed single-row read, so a few dozen concurrent misses barely register. That’s why I’d add no lock here: stampede protection earns its complexity when the rebuild is expensive.

Versioned snapshots for permissions and plans

Permissions and plan entitlements are read on nearly every request. Both caches use the version-key design from Intermediate: the database holds a version per organization that every relevant change bumps, a small Redis key caches that version, and a cached snapshot is used only if it was built from the same version. In-app changes delete the version key, so every snapshot for that organization stops matching at once.

The version key has a short TTL, and the comment above it records why:

The version key must stay short-lived: in-app permission mutations delete it via invalidate_org, but migrations (and any other out-of-band change) can only bump organizations.perm_version in the database. A long-lived cached version would keep validating stale snapshots against itself until it expired — with 24h this left every org’s permissions frozen for up to a day after a permission-seeding migration. 60s bounds that window to a minute at the cost of one single-row read per org per minute.

That’s the TTL-backstop rule, learned from the version that didn’t have a short one.

The entitlement cache also puts the shape of its JSON in the key (ent:v3:{org_id}). Its comment is the schema-drift story from Advanced:

nothing tracked the layout of the JSON until a cached entry outlived a code change that renamed a limits key, and a stale entry validated itself against a perfectly current ev and then raised KeyError deep inside a request.

So the schema rides in the key: an entry written by older code is not looked up at all, it simply ages out on its own TTL.

The comment logs each bump; for version 3 it notes that without one, “every org with a warm cache raises KeyError on the first comment answered, for up to the cache TTL.”

Invalidate after the commit, not before

The module that switches product features on and off per organization caches their state under one key per organization. Its invalidation helper describes the update/invalidate race better than I can:

Invalidating before the commit is not enough, and the gap is not theoretical. db.app_session() holds the transaction open until the FastAPI dependency closes — i.e. after the handler has returned — so between an in-handler invalidate() and that commit there are several more round trips (readiness recompute, entitlement read, disable counts, serialization). Any concurrent reader in that window — another request’s route guard, the nav shell, the outbox consumer’s beat tick — misses the cache, reads the old still-committed row, and repopulates the key with ex=60.

The consequence it spells out: for up to a minute, a module the admin just disabled reports as enabled, and the public booking page keeps taking bookings. So the write path deletes twice, eagerly and from an SQLAlchemy after_commit hook, “which is the one that actually closes the window.” The module also explains why it skips version keys: one writer path and one key per organization, “so a direct DELETE of that single key on every write is already precise invalidation.”

Keys that carry everything that changes the answer

The retrieval pipeline behind ezychat.io’s AI answers has two caches, both lessons in key design. The query-embedding cache rules out the approximate matching discussed in Expert:

Exact-match by construction: the key is the sha256 of the normalized query text plus the embedding model id, so identical questions reuse their vector and nothing approximate is ever served. Normalization is case-fold + whitespace-collapse only — anything smarter (stemming, typo folding) would make two different queries share a vector, which is exactly the accuracy risk this cache design forbids.

The result cache stores which knowledge-base chunks answered a question, under ai:rr:{org_id}:{kb_version}:{model}:{scope}:{qhash}. The organization keeps tenants apart. The kb_version is bumped “in the same transaction” as any change retrieval can see, so no cached result outlives an edit. The scope token exists because “two employees with different collection grants must never share cached results.” Degraded (unreranked) results are never cached; empty ones are, because, as its handbook page puts it, “a repeat miss costs nothing”: negative caching, invalidated by the version in the key.

Other caches follow the habit. The WhatsApp template list is keyed by organization and channel, because “one tenant’s approved names must never be served for another’s.” A cache of model answers for tuning sessions keys on a hash of the whole prompt (“The prompt is its own version”), because a hand-assembled key would be “a second description of the prompt, and the first time the two disagreed the cache would answer a question nobody asked.”

Fail open, because the database is the truth

The Redis caches quoted here all fall back to the database (or the vendor) when Redis has trouble. The rate limiter’s docstring states it as a codebase-wide rule, “matching every other Redis path in this codebase (caches, circuit breaker, debounce),” and the entitlement module says what it means for caches:

“Fail-open to DB” is availability language, not security: DB resolution is the ground truth, so a Redis outage degrades latency, never correctness.

That’s only true because the caches are built to be losable. The booking availability cache is “advisory only”: the booking is always checked again inside the database transaction, “the only source of truth for whether a slot is actually free.” A cache you can lose without being wrong is a cache you can fail open on.

The same module has a real latency number. Availability search used to fetch each (resource, day) entry with its own GET, and “the round trips — ~2.7 ms each — were the dominant cost of an availability query.” One MGET and one write-back pipeline made it “two, whatever the window.” When a request needs many entries, batching can matter more than the hit ratio.

One Redis, three jobs

In ezychat.io’s Terraform, the Redis replication group’s description is “sessions, queues, and caches,” and the alarm on evictions fires on the first one. Its comment:

Redis evictions mean the node is discarding keys to stay inside maxmemory. On this stack that is not a cache miss — the Celery broker lives here, so an eviction can silently drop queued work.

That’s the queue experiment from Advanced as a production concern. The cache writes in this section all pass an expiry (ex=), so cache entries clean themselves up whatever happens to memory.

Prompt caching on Gemini

Production AI on ezychat.io runs on Gemini, which caches long repeated prompt prefixes implicitly. From the adapter:

Gemini uses implicit caching keyed on the request prefix, so the adapter’s whole job is preserving the frozen-first block ordering ModelRequest already guarantees — frozen system blocks become the head of systemInstruction, tools follow deterministically. No explicit CachedContent objects at this stage: the stable block sizes in this pipeline are below the point where explicit context caching pays for its storage-per-hour cost.

The prompt compiler’s docstring promises “Zero volatile bytes: no timestamps, no org names, nothing that varies between renders” in the prefix, with per-turn context “after the cache boundary.” Because one changing byte would silently zero the discount, the daily cost ledger tracks a cache-read share per organization (cached over input tokens), and names what a busy organization stuck near zero means: “a volatile byte before the cache boundary silently zeroing the prompt-cache hit rate.” The price sheet notes cached input at a tenth of the input price for the Gemini models it lists.

Exact response caching appears where the task allows. Reading a document image is “a pure function of the pixels and the checklist it was read against,” so readings are cached by a hash of both, per organization, because “one afternoon of testing spent forty model calls re-reading four documents.”

What I’d copy

• Write the staleness budget next to every TTL, and keep a TTL even where you invalidate explicitly. The day-long backstop on permissions is the cautionary tale.
• Delete after commit, not before, and delete rather than set.
• Put everything that changes the answer into the key: tenant, permission scope, data version, schema version, model and prompt version. Hash the key if it gets long, but hash all of it.
• Fail open only when the cache is advisory. If losing the cache can make you wrong, it isn’t a cache any more.
• Use one cache policy per kind of file at the CDN: fingerprinted forever, everything else revalidated, and an invalidation on deploy.
• Alarm on evictions whenever cache entries share memory with anything you can’t afford to lose.

Cheat sheet