Designing real-time chat, from first idea to production

Ben types “on my way” and taps send. On Aisha’s phone, across town, the words appear a fraction of a second later. She didn’t refresh anything. Her app didn’t ask. The message just arrived.

That moment is harder to build than it looks, because the web works the other way round: a browser asks, a server answers, and the server can’t speak first. Real-time chat is the set of designs that turn that around, and keep it working when the phone enters a tunnel, when ten thousand people share a room, and when a server restarts mid-afternoon.

By the end of this article you’ll be able to:

• explain polling, long polling, server-sent events and WebSockets, and pick between them with numbers;
• design the message model: ids, sequence numbers, receipts, presence and typing, history and catch-up;
• scale WebSocket servers horizontally with a pub/sub layer, and choose between fan-out on write and fan-out on read;
• handle the failures most tutorials skip: slow readers, dead connections, reconnect storms;
• walk through a chat system design in an interview, with estimates you can defend.

If you’ve never opened a WebSocket, start at Beginner. If you’ve built a chat app on one server, skip to Advanced, where the second server makes everything interesting. The last section is the real-time inbox of ezychat.io, the multi-tenant messaging product I build and run, quoted from its source.

Beginner: what “real-time” means and how a message gets pushed

What does real-time mean in a chat app?

For a chat app, real-time means one thing to the person using it: a message shows up without them doing anything. No pull-to-refresh, no reload button, no waiting for the next time the app checks.

This is “soft” real-time: nobody is hurt if a message takes 800 milliseconds instead of 200, unlike the “hard” real-time of brake controllers. But people notice. A reply that appears while you’re still looking feels like a conversation; one that appears ten seconds later feels like email. Receipts (the ticks), presence (the green dot) and typing indicators ride on the same machinery.

The request/response problem

Ordinary HTTP works like a shop counter. You ask, the shopkeeper hands it over, and that’s the end of it. The shopkeeper can’t run into the street to tell you a delivery has arrived; they wait until you come back and ask. When Ben’s message reaches the server, Aisha’s browser isn’t asking for anything, so the server has the message and nobody to give it to.

There are four standard ways out, from the crudest to the one most chat apps use.

Short polling: ask again and again

The simplest fix: the browser asks every few seconds. “Anything new? No. Anything new? No. Anything new? Yes, one message.”

It works everywhere and it’s trivial to build. Its two costs pull against each other: ask every 5 seconds and a message waits up to 5 seconds; ask every second and you make five times as many requests, nearly all empty, each carrying a full set of HTTP headers.

Long polling: ask, and let the server wait

Long polling keeps the question but changes the answer. The browser asks “anything new?”, and if there isn’t, the server doesn’t answer yet. It holds the request open until a message arrives (or until 30 seconds or so pass, when it answers “nothing” so proxies don’t kill the connection). The moment the answer comes back, the browser asks again.

Messages now arrive almost as soon as they’re sent, and a quiet conversation costs one request every 30 seconds. The catch is busy conversations: every message ends a request, and anything that arrives before the next request lands has to wait. A busy conversation turns long polling back into fast polling.

Server-sent events: one response that never ends

Server-sent events (SSE) are a browser standard for a response that keeps streaming. The browser opens one request with EventSource, and the server writes events down it whenever it likes, each as a few lines of text:

id: 42
data: {"from": "ben", "text": "on my way"}

The browser reconnects by itself and sends the last id it saw in a Last-Event-ID header, so the server can pick up where it left off. We’ll build the same idea by hand later.

SSE only goes from server to browser; to send, the browser makes an ordinary HTTP request. One limit to know: over HTTP/1.1 browsers allow six connections per site, and each open stream holds one, so several tabs can run out. Over HTTP/2 the streams share one connection.

WebSockets: a connection that stays open both ways

A WebSocket starts life as an HTTP request with an Upgrade: websocket header. The server answers 101 Switching Protocols, and from then on the same TCP connection carries frames in both directions, for as long as both sides keep it open. No more headers per message: a small frame from the server costs 2 bytes of framing (frames from the browser add a 4-byte mask, so 6).

That’s what most chat apps use, and what this article uses from here on.

Which one should you use?

Here are the four side by side. “Overhead” means bytes spent on everything except the message itself.

To put numbers on that, I wrote out a typical browser request for one poll and counted the bytes:

poll_request = (
    "GET /api/messages?after=1041 HTTP/1.1\r\n"
    "Host: chat.example.com\r\n"
    "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36\r\n"
    "Accept: application/json\r\n"
    "Accept-Language: en-GB,en;q=0.9\r\n"
    "Accept-Encoding: gzip, deflate, br\r\n"
    "Referer: https://chat.example.com/\r\n"
    "Cookie: session=" + "x" * 43 + "\r\n"
    "\r\n"
)
empty_response = (
    "HTTP/1.1 200 OK\r\n"
    "Date: Fri, 13 Nov 2026 09:00:00 GMT\r\n"
    "Content-Type: application/json\r\n"
    "Content-Length: 15\r\n"
    "Cache-Control: no-store\r\n"
    "\r\n"
    '{"messages":[]}'
)
print("one poll that finds nothing:", len(poll_request) + len(empty_response), "bytes")


def ws_frame_header(payload_len, from_client):
    """RFC 6455: 2 bytes, plus 2 or 8 for longer payloads, plus a 4-byte mask on client frames."""
    size = 2 if payload_len < 126 else 4 if payload_len < 65536 else 10
    return size + (4 if from_client else 0)


print("WebSocket frame header, server to client:", ws_frame_header(60, from_client=False), "bytes")
print("WebSocket frame header, client to server:", ws_frame_header(60, from_client=True), "bytes")

# one poll that finds nothing: 536 bytes
# WebSocket frame header, server to client: 2 bytes
# WebSocket frame header, client to server: 6 bytes

A poll that finds nothing costs 536 bytes. A message on an open WebSocket costs 2. (Counting the same way, the WebSocket handshake itself is 582 bytes, paid once per connection.) Real cookies are often far longer than 43 characters, so 536 is on the kind side. HTTP/2 compresses repeated headers and would shrink the polling cost a lot; it can’t shrink the waiting.

Then I modelled one minute of chat: twelve messages, including a burst of three at 15 seconds, a 100 ms round trip, and heartbeats every 20 seconds on the open connections:

ONE_WAY = 0.05          # seconds, client to server or back (a 100 ms round trip)
POLL_BYTES = 536        # one request and its empty answer, counted above
UPGRADE_BYTES = 582     # one WebSocket handshake, counted the same way
HEARTBEAT = 20          # seconds between keep-alive pings on an open connection (8 bytes each)


def short_poll(arrivals, duration, every=5.0):
    """Ask every `every` seconds. Each answer carries whatever arrived since the last one."""
    polls = [k * every for k in range(int(duration / every) + 1)]
    latency = []
    for t in arrivals:
        sent = next(p for p in polls if p + ONE_WAY >= t)   # the first poll to reach the server after it
        latency.append(sent + 2 * ONE_WAY - t)
    return len(polls), len(polls) * POLL_BYTES, latency


def long_poll(arrivals, duration, hold=30.0):
    """The server holds each request until there is something to say, or `hold` seconds pass."""
    requests, latency, pending = 0, [], list(arrivals)
    at_server = ONE_WAY                                     # when the first held request arrives
    while at_server < duration:
        requests += 1
        if pending and pending[0] <= at_server + hold:
            answer = max(at_server, pending[0])             # answer as soon as something is there
        else:
            answer = at_server + hold                       # nothing came: an empty answer, ask again
        sent = [t for t in pending if t <= answer]
        latency += [answer + ONE_WAY - t for t in sent]
        pending = [t for t in pending if t > answer]
        at_server = answer + 2 * ONE_WAY                    # back to the client, and the next request out
    return requests, requests * POLL_BYTES, latency


def push(arrivals, duration, opening, per_message):
    """SSE or WebSocket: one connection, then every message is pushed down it as it arrives."""
    heartbeats = int(duration // HEARTBEAT)
    return 1, opening + per_message * len(arrivals) + 8 * heartbeats, [ONE_WAY] * len(arrivals)


# Twelve messages in a minute, including a burst of three at t = 15 s.
arrivals = [2.0, 2.4, 9.1, 15.0, 15.03, 15.06, 27.8, 33.0, 41.5, 48.2, 55.0, 58.9]
duration = 60.0
rows = {
    "short poll, 5 s": short_poll(arrivals, duration),
    "short poll, 1 s": short_poll(arrivals, duration, every=1.0),
    "long poll": long_poll(arrivals, duration),
    "SSE": push(arrivals, duration, POLL_BYTES, per_message=14),        # "id: 42\ndata: " ... "\n\n"
    "WebSocket": push(arrivals, duration, UPGRADE_BYTES, per_message=2),  # a 2-byte frame header
}
print(f"{'':17}{'requests':>9}{'overhead':>10}{'avg delay':>11}{'worst':>9}")
for name, (requests, overhead, latency) in rows.items():
    avg = sum(latency) / len(latency)
    print(f"{name:17}{requests:>9}{overhead:>8} B{avg * 1000:>8.0f} ms{max(latency) * 1000:>6.0f} ms")

#                   requests  overhead  avg delay    worst
# short poll, 5 s         13    6968 B    1934 ms  5040 ms
# short poll, 1 s         61   32696 B     434 ms  1040 ms
# long poll               12    6432 B      59 ms   120 ms
# SSE                      1     728 B      50 ms    50 ms
# WebSocket                1     630 B      50 ms    50 ms

Polling every 5 seconds made one message wait over five seconds. Polling every second cut the wait and multiplied the bytes by almost five. Long polling looks nearly as good as a WebSocket here, but notice the worst case: the burst at 15 seconds caught it between requests. The demo below runs the same model, and the “Busy” stream shows where long polling falls apart.

With the busy stream, long polling makes 30 requests in a minute, more than polling every 5 seconds does, because every message ends a request. SSE and the WebSocket still make one.

My rule: use a WebSocket for chat, because chat sends both ways and the client sends often (typing, receipts, the messages themselves). Use SSE when only the server talks, like a notification bell or an AI answer streaming in, because it’s plain HTTP and reconnects for free. Use polling when “live” means “within a minute”, and don’t feel bad about it.

Your first WebSocket chat

Here’s a complete chat server in Python with the websockets library (pip install websockets), plus two people talking to it, in one file you can run:

import asyncio
import json

from websockets.asyncio.client import connect
from websockets.asyncio.server import serve

connected = set()  # every open socket on this server


async def chat(websocket):
    connected.add(websocket)
    try:
        async for raw in websocket:              # wait for the next frame from this client
            message = json.loads(raw)
            frame = json.dumps({"from": message["name"], "text": message["text"]})
            for other in list(connected):        # push it to everyone, the sender included
                await other.send(frame)
    finally:
        connected.discard(websocket)             # the socket closed: forget it


async def person(name, says=None):
    async with connect("ws://localhost:8765") as ws:
        if says:
            await asyncio.sleep(0.1)             # let the other person connect first
            await ws.send(json.dumps({"name": name, "text": says}))
        print(name, "got", await ws.recv())


async def main():
    async with serve(chat, "localhost", 8765):
        await asyncio.gather(person("aisha"), person("ben", says="on my way"))


asyncio.run(main())

# ben got {"from": "ben", "text": "on my way"}
# aisha got {"from": "ben", "text": "on my way"}

About thirty lines, and it really is real-time: Aisha’s client never asks for anything, it just waits on recv(), and the server pushes Ben’s message to her the moment it arrives.

The important idea is connected. The server keeps a set of open connections in memory, and sending a message means looping over it. That one line is the whole problem the rest of this article solves:

1. The set lives in one process. Run two servers and each has half the users.
2. Nothing is stored. If Aisha’s phone is offline for a second, she misses the message forever.
3. await other.send(frame) waits for each socket in turn, so one slow phone delays everyone after it in the loop.
4. There are no ids, so a retry would show the message twice, and there’s no order, so two messages can swap places.

Intermediate: messages, ordering, receipts and history

The toy server forwards text. A real one stores messages, numbers them, tracks who has seen what, and lets a phone that was away catch up. This section builds each piece.

What does a message need?

Every field below exists because of a bug you get without it:

Why does the server number the messages? Because clocks disagree. If Ben’s phone is 40 seconds fast, ordering by the time each phone says it sent puts his replies before the questions they answer. The server is the one place both messages pass through, so its order is the order, and a counter makes it gap-free and cheap to compare: “I have up to 41” completely describes what a phone has.

Why per conversation and not global? A global counter makes every message on the platform pass through one place. A counter per conversation only serializes one conversation, whose rate is limited by how fast people type: strict order where people can see it, no coordination where they can’t.

Discord takes a different route: its ids are Snowflakes, 64-bit numbers with a timestamp in the top bits, so they sort by time with no counter at all. That’s an approximate global order; a per-conversation counter is an exact local one. I’d pick the counter for chat, because “is there a gap?” becomes a subtraction.

Sending: optimistic, with a client id

When Ben taps send, his phone shouldn’t wait for the server before showing the message. It shows the bubble at once with a little clock, and sends it with an id it just made up:

If the acknowledgement doesn’t come back, the phone resends, with the same client id. That’s the idempotency key pattern, and it’s what stops a flaky train connection from posting “on my way” three times:

import sqlite3
import uuid

db = sqlite3.connect(":memory:", isolation_level=None)
db.executescript("""
    CREATE TABLE conversations (id TEXT PRIMARY KEY, last_seq INTEGER NOT NULL DEFAULT 0);
    CREATE TABLE messages (
        id            TEXT PRIMARY KEY,              -- the server's id for the message
        conversation  TEXT NOT NULL,
        seq           INTEGER NOT NULL,              -- 1, 2, 3... within one conversation
        sender        TEXT NOT NULL,
        client_msg_id TEXT NOT NULL,                 -- made by the sender's device, once per message
        body          TEXT NOT NULL,
        created_at    TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
        UNIQUE (conversation, seq),
        UNIQUE (conversation, sender, client_msg_id)
    );
    INSERT INTO conversations (id) VALUES ('c1');
""")


def send(conversation, sender, client_msg_id, body):
    """Store a message once, however many times the device sends it. Returns the ack."""
    db.execute("BEGIN IMMEDIATE")                    # one writer at a time per database here
    try:
        seen = db.execute(
            "SELECT id, seq FROM messages WHERE conversation = ? AND sender = ? AND client_msg_id = ?",
            (conversation, sender, client_msg_id)).fetchone()
        if seen:                                     # a retry: answer with the original
            db.execute("COMMIT")
            return {"client_msg_id": client_msg_id, "id": seen[0], "seq": seen[1], "duplicate": True}
        seq = db.execute(
            "UPDATE conversations SET last_seq = last_seq + 1 WHERE id = ? RETURNING last_seq",
            (conversation,)).fetchone()[0]           # the next number, under the row's write lock
        message_id = str(uuid.uuid4())
        db.execute(
            "INSERT INTO messages (id, conversation, seq, sender, client_msg_id, body) VALUES (?, ?, ?, ?, ?, ?)",
            (message_id, conversation, seq, sender, client_msg_id, body))
        db.execute("COMMIT")
        return {"client_msg_id": client_msg_id, "id": message_id, "seq": seq, "duplicate": False}
    except Exception:
        db.execute("ROLLBACK")
        raise


first = send("c1", "ben", "ben-phone-0001", "on my way")
print(first["seq"], first["duplicate"])
again = send("c1", "ben", "ben-phone-0001", "on my way")   # the ack was lost, so the phone resends
print(again["seq"], again["duplicate"], again["id"] == first["id"])
print(send("c1", "aisha", "aisha-web-0001", "see you there")["seq"])
print(db.execute("SELECT seq, sender, body FROM messages ORDER BY seq").fetchall())

# 1 False
# 1 True True
# 2
# [(1, 'ben', 'on my way'), (2, 'aisha', 'see you there')]

The retry got back the original message, with the original id and seq, and the table has one row. The ack carries the client_msg_id so the phone knows which pending bubble to update: it swaps the clock for a tick and moves the bubble to its real position by seq.

In PostgreSQL, lock the conversation row first (SELECT ... FOR UPDATE), then check for the client id, then take the next number. Two copies of one send then queue on the lock, and the second one finds the first one’s row. The unique constraint on (conversation, sender, client_msg_id) is the backstop for any path that forgets.

Delivery and read receipts

The three ticks are three different facts, and they come from three different places:

1. Sent: the server stored it. This comes from the ack.
2. Delivered: the recipient’s device received it. The device sends a “delivered” frame back, and the server relays it to the sender.
3. Read: the recipient opened the chat and the message was on screen.

Two rules keep receipts honest. First, they only move forward. Receipts travel over the network like everything else, so a “delivered” can arrive after a “read”. Applying them in arrival order would turn a blue tick grey again. Second, in a group, don’t store one receipt per member per message. Store one cursor per member: “Chen has read up to 41”. A message is read by everyone whose cursor is at or past its seq.

STATES = {"sent": 1, "delivered": 2, "read": 3}


def apply_status(message, status):
    """Receipts only move forward. A late 'delivered' must not undo a 'read'."""
    if STATES[status] > STATES[message["status"]]:
        message["status"] = status
    return message["status"]


m = {"seq": 7, "status": "sent"}
for update in ["read", "delivered", "delivered"]:   # how they happened to arrive
    print(update, "->", apply_status(m, update))


# In a group, store one cursor per member, not one receipt per member per message.
read_upto = {}  # (conversation, member) -> highest seq they have read


def mark_read(conversation, member, seq):
    key = (conversation, member)
    read_upto[key] = max(read_upto.get(key, 0), seq)   # a cursor never moves backwards


def read_by(conversation, seq, members):
    return [m for m in members if read_upto.get((conversation, m), 0) >= seq]


members = ["aisha", "ben", "chen", "dev"]
mark_read("g1", "aisha", 41)
mark_read("g1", "ben", 38)
mark_read("g1", "ben", 12)                             # an old tab reporting late
mark_read("g1", "chen", 41)
print("seq 40 read by:", read_by("g1", 40, members))
print("ben's cursor:", read_upto[("g1", "ben")])

group, messages = 50, 1000
print("receipt rows, one per member per message:", group * messages)
print("cursor rows, one per member:", group)

# read -> read
# delivered -> read
# delivered -> read
# seq 40 read by: ['aisha', 'chen']
# ben's cursor: 38
# receipt rows, one per member per message: 50000
# cursor rows, one per member: 50

The cursor is the sequence number paying for itself. Because messages are numbered without gaps, “read up to 41” covers 41 messages in one integer, and unread counts become arithmetic, which is one of the exercises below.

Presence: online means “heard from recently”

A green dot looks like a fact, but the server can’t actually know whether someone is online. It only knows when it last heard from them. So presence is built on heartbeats and expiry: every connected device says “still here” every 20 seconds or so, the server stores “online” with a 60-second lifetime, and if the heartbeats stop, the entry simply expires. Nobody has to say goodbye, which is good, because a phone that loses signal never does.

class Presence:
    """Online means 'heartbeated recently'. Like Redis SET key EX ttl: nobody has to say goodbye."""

    def __init__(self, ttl, clock):
        self.ttl, self.clock, self.expires = ttl, clock, {}

    def heartbeat(self, user):
        came_online = not self.is_online(user)
        self.expires[user] = self.clock() + self.ttl
        return came_online                       # True on an offline -> online transition

    def is_online(self, user):
        return self.expires.get(user, 0) > self.clock()


class TypingThrottle:
    """Send 'typing' at most once every `gap` seconds per conversation, however fast the keys go."""

    def __init__(self, gap, clock):
        self.gap, self.clock, self.last = gap, clock, {}

    def should_send(self, conversation):
        now = self.clock()
        if now - self.last.get(conversation, float("-inf")) < self.gap:
            return False
        self.last[conversation] = now
        return True


now = 0.0
presence = Presence(ttl=60, clock=lambda: now)
print("first heartbeat, came online:", presence.heartbeat("aisha"))
now = 20.0
print("20 s later, came online:", presence.heartbeat("aisha"))
now = 79.0
print("phone went into a tunnel at 20 s; online at 79 s:", presence.is_online("aisha"))
now = 81.0
print("online at 81 s:", presence.is_online("aisha"))

throttle = TypingThrottle(gap=5, clock=lambda: now)
sent = 0
for k in range(120):                          # 120 keystrokes over 18 seconds of typing
    now = 100.0 + k * 0.15
    sent += throttle.should_send("c1")
print("keystrokes: 120, typing events sent:", sent)

# first heartbeat, came online: True
# 20 s later, came online: False
# phone went into a tunnel at 20 s; online at 79 s: True
# online at 81 s: False
# keystrokes: 120, typing events sent: 4

The cost of expiry is honesty in one direction only: someone whose phone died at 20 seconds still looks online until 80. Shorten the lifetime and the dot is more accurate, but every user’s heartbeat costs more. A minute is a common compromise.

The came_online flag matters more than it looks. The moment someone goes from offline to online is when the system should do something: deliver queued work, re-route a support ticket to them, tell their contacts. Detecting that edge on the same write that refreshes the expiry means no second lookup and no race. The production section shows exactly this, with Redis SET ... GET.

Typing indicators: ephemeral, and rate limited

“Aisha is typing…” is the most disposable data in the system. It’s never stored, it’s useless five seconds later, and a lost one doesn’t matter. So treat it that way:

• Send it over the socket, never to the database. No row, no outbox, no retry.
• Expire it on the receiving side. The recipient’s app hides the indicator a few seconds after the last “typing” frame, so a “stopped typing” frame that never arrives can’t leave it stuck.
• Rate limit it. A keystroke is not an event: the throttle above turned 120 keystrokes into 4 frames. Enforce it on the server too, since a wedged client won’t respect yours, and when the limit is hit just say “not sent”. Being rationed is the normal case for typing.

Storing messages and loading history

A chat screen opens on the newest messages and loads older ones as you scroll up. That’s pagination backwards in time, and there are two ways to do it.

Offset pagination says “skip the newest 900,000 and give me the next 50”. It’s what LIMIT 50 OFFSET 900000 does, and the database really does walk past all 900,000 rows to find where to start.

Keyset pagination (also called cursor pagination) says “give me the 50 messages before seq 100,001”. With an index on (conversation, seq), the database jumps straight to that seq and reads 50 rows. It costs the same on page 2 as on page 20,000.

I measured both on a million messages in one conversation, in SQLite on my laptop (an M1 Pro):

import sqlite3
import time

db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE messages (
    conversation TEXT, seq INTEGER, sender TEXT, body TEXT,
    PRIMARY KEY (conversation, seq)) WITHOUT ROWID""")
db.executemany("INSERT INTO messages VALUES ('c1', ?, 'ben', 'hello there, this is a message')",
               ((s,) for s in range(1, 1_000_001)))


def newest_page():
    return db.execute("SELECT seq FROM messages WHERE conversation = 'c1' "
                      "ORDER BY seq DESC LIMIT 50").fetchall()


def offset_page(page):                     # "page 18,000 of the history"
    return db.execute("SELECT seq FROM messages WHERE conversation = 'c1' "
                      "ORDER BY seq DESC LIMIT 50 OFFSET ?", (page * 50,)).fetchall()


def keyset_page(before_seq):               # "the 50 messages before the oldest one I have"
    return db.execute("SELECT seq FROM messages WHERE conversation = 'c1' AND seq < ? "
                      "ORDER BY seq DESC LIMIT 50", (before_seq,)).fetchall()


def best_of(fn, *args, runs=20):
    times = []
    for _ in range(runs):
        start = time.perf_counter()
        rows = fn(*args)
        times.append(time.perf_counter() - start)
    return rows, min(times) * 1000


rows, ms = best_of(newest_page)
print(f"newest 50:                seq {rows[0][0]}..{rows[-1][0]}  {ms:.3f} ms")
rows, ms = best_of(offset_page, 18_000)
print(f"OFFSET 900,000:           seq {rows[0][0]}..{rows[-1][0]}  {ms:.3f} ms")
rows, ms = best_of(keyset_page, 100_001)
print(f"keyset, seq < 100,001:    seq {rows[0][0]}..{rows[-1][0]}  {ms:.3f} ms")

# newest 50:                seq 1000000..999951  0.012 ms
# OFFSET 900,000:           seq 100000..99951  22.170 ms
# keyset, seq < 100,001:    seq 100000..99951  0.012 ms

Same 50 rows. The offset query took 22 ms (22.6 ms on a second run); the keyset query took 0.012 ms, the same as reading the newest page. That’s a factor of about 1,800, and it grows with the depth of the scroll.

Keyset pagination has a second advantage in chat specifically: new messages don’t shift the pages. With offsets, ten messages arriving while you scroll push everything down by ten, and your next “page” repeats ten messages you’ve already seen. A cursor names a position in the data, not a count from the top.

If you order by timestamp instead of seq, the cursor needs a tie-breaker, because two messages can share a millisecond. That’s the first exercise below. (Merging several sorted conversations into one time-ordered inbox is the same idea with two cursors, which the two pointers article covers.)

Reconnecting and catching up

Phones disconnect constantly: tunnels, lifts, switching from Wi-Fi to mobile data, the operating system freezing a background app. Each time, messages keep being sent while the socket is down. The sequence number makes catching up simple:

server_history = {"c1": []}   # what the server has stored, in seq order


def store(conversation, text):
    message = {"seq": len(server_history[conversation]) + 1, "text": text}
    server_history[conversation].append(message)
    return message


def fetch_after(conversation, after_seq):
    """GET /conversations/c1/messages?after=N: the source of truth, by sequence number."""
    return [m for m in server_history[conversation] if m["seq"] > after_seq]


class Client:
    def __init__(self):
        self.last_seq = {}      # conversation -> highest seq shown, saved on the device
        self.shown = []

    def show(self, conversation, message):
        if message["seq"] <= self.last_seq.get(conversation, 0):
            return              # already have it: a duplicate push, or an overlap with a fetch
        self.shown.append(message["seq"])
        self.last_seq[conversation] = message["seq"]

    def on_push(self, conversation, message):
        if message["seq"] > self.last_seq.get(conversation, 0) + 1:   # a gap: something was missed
            for missed in fetch_after(conversation, self.last_seq.get(conversation, 0)):
                self.show(conversation, missed)
        self.show(conversation, message)

    def on_reconnect(self):
        for conversation, seq in list(self.last_seq.items()):        # "everything after what I've got"
            for missed in fetch_after(conversation, seq):
                self.show(conversation, missed)


phone = Client()
for k in range(3):
    phone.on_push("c1", store("c1", f"hi {k}"))      # 1, 2, 3 arrive live
for k in range(4):
    store("c1", f"while offline {k}")                # 4 to 7 are sent while the phone is in a tunnel
phone.on_reconnect()
print("after reconnect:", phone.shown)
phone.on_push("c1", server_history["c1"][6])         # a late duplicate of 7
store("c1", "eight")
store("c1", "nine")                                  # 8 and 9 are stored, but their pushes are lost
phone.on_push("c1", store("c1", "ten"))              # 10 arrives live
print("after a gap:    ", phone.shown)

# after reconnect: [1, 2, 3, 4, 5, 6, 7]
# after a gap:     [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Three things happened there, and each is a rule worth keeping:

• Reconnect means catch up. The phone didn’t trust the socket to replay what it missed; it asked the database for everything after seq 3.
• A gap means catch up. When 10 arrived and the phone expected 8, it fetched the missing range before showing 10, so the conversation never displayed out of order.
• Duplicates are free. The late copy of 7 was dropped by one comparison, because seq <= last_seq means “already shown”.

This is the most important design decision in a chat system, so I’ll say it plainly: the socket is a fast path, not the source of truth. Pushes are allowed to be lost, duplicated or delayed, because the database, read by sequence number, can always fix the picture. Every hard failure in the Advanced section is survivable for exactly this reason.

m1  09:00:00.001
m2  09:00:00.002
m3  09:00:00.002
m4  09:00:00.003

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE messages (id TEXT PRIMARY KEY, created_at TEXT, body TEXT)")
db.executemany("INSERT INTO messages VALUES (?, ?, ?)", [
    ("m1", "2026-11-13T09:00:00.001", "one"),
    ("m2", "2026-11-13T09:00:00.002", "two"),
    ("m3", "2026-11-13T09:00:00.002", "three"),   # same millisecond as m2
    ("m4", "2026-11-13T09:00:00.003", "four"),
])


def pages(query, cursor_of, size=2):
    seen, cursor = [], None
    while True:
        rows = db.execute(query(cursor), (cursor or ()) + (size,)).fetchall()
        if not rows:
            return seen
        seen += [r[0] for r in rows]
        cursor = cursor_of(rows[-1])


by_time = pages(
    lambda c: "SELECT id, created_at FROM messages "
              + ("WHERE created_at < ? " if c else "")
              + "ORDER BY created_at DESC LIMIT ?",
    lambda last: (last[1],))
by_time_and_id = pages(
    lambda c: "SELECT id, created_at FROM messages "
              + ("WHERE created_at < ? OR (created_at = ? AND id < ?) " if c else "")
              + "ORDER BY created_at DESC, id DESC LIMIT ?",
    lambda last: (last[1], last[1], last[0]))
print("cursor on time only:   ", by_time)
print("cursor on (time, id):  ", by_time_and_id)

# cursor on time only:    ['m4', 'm2', 'm1']
# cursor on (time, id):   ['m4', 'm3', 'm2', 'm1']

conversations = {"c1": 120, "g7": 4_311, "c9": 15}            # conversation -> last_seq
read_upto = {("aisha", "c1"): 120, ("aisha", "g7"): 4_290}      # (user, conversation) -> cursor
own = {("aisha", "g7"): {4_301, 4_305}}                          # seqs she sent after her cursor


def unread(user):
    counts = {}
    for conversation, last_seq in conversations.items():
        cursor = read_upto.get((user, conversation), 0)
        mine = sum(1 for s in own.get((user, conversation), ()) if s > cursor)
        counts[conversation] = last_seq - cursor - mine
    return counts


print(unread("aisha"))

# {'c1': 0, 'g7': 19, 'c9': 15}

Advanced: many servers, slow readers and dead connections

Everything so far fits on one server. The day you add a second, the connected set from the Beginner section splits in two, and you discover which parts of your design assumed it wouldn’t.

Each server only holds its own sockets

A WebSocket is a TCP connection to one specific process. When Aisha’s phone connected, the load balancer picked a server, say gw-1, and her socket lives there for as long as it stays open. Ben’s socket might be on gw-2. When Ben sends a message, gw-2 receives it, and gw-2 cannot write to Aisha’s socket. It doesn’t have it.

So the servers need a way to pass messages to each other. The standard answer is a publish/subscribe (pub/sub) layer: a broker that servers subscribe to channels on, and that forwards anything published on a channel to every subscriber. Redis has one built in, and it’s what most chat backends start with.

Here’s that design running against a real Redis: two gateway servers, a registry of which server holds which user, and one channel per server.

import asyncio
import json

import redis.asyncio as redis

R = redis.Redis(port=6411)
members = {"g1": ["aisha", "ben", "chen", "dev"]}       # dev is offline


class Gateway:
    """One WebSocket server. It can only write to sockets it holds itself."""

    def __init__(self, name):
        self.name, self.sockets, self.pubsub = name, {}, R.pubsub()

    async def start(self):
        await self.pubsub.subscribe(f"gw:{self.name}")   # one channel per server
        asyncio.create_task(self.listen())

    async def connect(self, user):
        self.sockets[user] = []                          # stand-in for a live socket's send queue
        await R.sadd(f"conn:{user}", self.name)          # the registry: which servers hold this user
        await R.expire(f"conn:{user}", 60)               # refreshed by heartbeats; a crash ages out

    async def listen(self):
        async for msg in self.pubsub.listen():
            if msg["type"] != "message":
                continue
            frame = json.loads(msg["data"])
            for user in frame["to"]:
                if user in self.sockets:                 # it may have disconnected since
                    self.sockets[user].append(frame["text"])


async def send(conversation, sender, text):
    """Called on whichever server the sender is connected to, after the message is stored."""
    by_server = {}
    for user in members[conversation]:
        for server in await R.smembers(f"conn:{user}"):
            by_server.setdefault(server.decode(), []).append(user)
    for server, users in by_server.items():               # one publish per server, not per user
        await R.publish(f"gw:{server}", json.dumps({"to": users, "text": f"{sender}: {text}"}))
    return len(by_server)


async def main():
    await R.delete(*[f"conn:{u}" for u in members["g1"]])
    gw1, gw2 = Gateway("gw-1"), Gateway("gw-2")
    for gw in (gw1, gw2):
        await gw.start()
    await gw1.connect("aisha")
    await gw2.connect("ben")
    await gw2.connect("chen")
    await gw1.connect("chen")                            # chen also has a laptop open, on gw-1
    publishes = await send("g1", "aisha", "running 5 min late")
    await asyncio.sleep(0.05)
    print("publishes:", publishes)
    for gw in (gw1, gw2):
        print(gw.name, {u: q for u, q in gw.sockets.items()})
    print("dev is offline, registry:", await R.smembers("conn:dev"))


asyncio.run(main())

# publishes: 2
# gw-1 {'aisha': ['aisha: running 5 min late'], 'chen': ['aisha: running 5 min late']}
# gw-2 {'ben': ['aisha: running 5 min late'], 'chen': ['aisha: running 5 min late']}
# dev is offline, registry: set()

Two publishes reached four sockets on two servers, including both of Chen’s devices. Dev, who’s offline, isn’t in the registry, so nothing was sent for him; he’ll catch up by sequence number when he opens the app (and, in the Expert section, get a push notification). On my laptop, a Redis pub/sub message took a median of about 0.2 ms from PUBLISH to the subscriber (p99 about 0.5 ms, over 10,000 messages, Redis 7.2 on localhost). The hop is cheap; what matters is how many you make.

Redis pub/sub forgets everything

Read the Redis documentation’s description of pub/sub carefully: delivery is at most once. A message published to a channel with no subscribers at that instant is gone. Nothing is stored, nothing is retried:

import redis

r = redis.Redis(port=6411)
print("receivers:", r.publish("gw:gw-3", '{"to": ["ben"], "text": "hello?"}'))  # gw-3 is restarting
p = r.pubsub()
p.subscribe("gw:gw-3")                                    # it comes back a moment later
p.get_message(timeout=1)                                  # the subscribe confirmation
print("waiting for it:", p.get_message(timeout=1))        # the message is not stored anywhere

# receivers: 0
# waiting for it: None

PUBLISH returned 0: nobody heard it, and when gw-3 came back a moment later there was nothing to collect. That’s fine only because of the Intermediate section’s rule. The message is in the database with a sequence number, Ben’s phone reconnects when gw-3 goes down, and the reconnect fetches everything after his last seq. Pub/sub is allowed to be lossy because it isn’t the source of truth.

If you need the broker itself to keep messages, Redis Streams (with consumer groups), Kafka or NATS JetStream store them and let a consumer resume from an offset. I’d still keep the database as the truth and use a stream only where a server must not miss an event, such as a notification worker. A gateway that missed something can always ask the database.

Which channels should the servers subscribe to?

There are three layouts, and the choice decides how many publishes each message costs:

• A channel per user. Each gateway subscribes to user:{id} for every user it holds. A group of 50 costs 50 publishes, and all of a user’s devices get it, wherever they’re connected.
• A channel per conversation. One publish per message, however big the group. But a user in 400 groups costs 400 subscriptions on connect, and every server holding any member of a busy group receives all its traffic.
• A channel per server, plus a connection registry (the example above). Never more publishes than servers. The cost is an extra lookup per send, and registry entries that must expire when a server dies.

For 1:1 chats and small groups I’d start with a channel per user, the easiest to reason about. Move to per-server routing when groups get large; use per-conversation channels for big rooms where most members are watching.

One Redis Cluster detail: classic PUBLISH is broadcast to every node, so pub/sub doesn’t scale out by adding shards. Redis 7.0 added sharded pub/sub (SSUBSCRIBE, SPUBLISH), where a channel lives on the node that owns its hash slot. Past one Redis, plan on the sharded commands.

Do you need sticky sessions?

Not for WebSockets themselves. A WebSocket is one TCP connection, so every frame already goes to the server that accepted it; the load balancer decides once, at connect time. Sticky sessions matter when one logical session spans several HTTP requests, which is long polling. Socket.IO, for example, documents that its long-polling fallback across several servers needs sticky sessions.

What you do need is a plan for rebalancing. New servers start empty while old ones keep their sockets, and a restarted server’s thousands of clients reconnect at once. Drain servers gradually on deploy, and make clients reconnect with full jitter (a random delay across the whole backoff window), so a restart becomes a trickle instead of a stampede on your history endpoint. The production section has a client that does exactly that.

Fan-out on write or fan-out on read?

When a message goes to many people, somebody has to do work for each of them. Fan-out is the name for that work, and the question is when it happens.

• Fan-out on write: at send time, do the per-member work. Write an entry into each member’s inbox, push a frame to each online member. Reading is then cheap: each member’s inbox is ready.
• Fan-out on read: at send time, store the message once in the conversation and push it only to people looking at that conversation right now. Everyone else finds out when they open it, by reading from their cursor.

The same choice appears in social feeds (Twitter’s timelines are the textbook example), but chat has its own twist: in a 1:1 chat, the other person must hear about the message now, even if they’re on a different screen. Try it:

The model behind the demo, in Python, prints the same table:

SERVERS = 8


def servers_touched(users):
    """Users are spread evenly over the servers, so `users` of them sit on at most SERVERS servers."""
    return min(SERVERS, users)


def on_write(recipients, online, viewing):
    rows = 1 + recipients                      # the message, plus one inbox entry per member
    publishes = servers_touched(online)        # every online member's server is told
    frames = online                            # every online member gets the message now
    return rows, publishes, frames


def on_read(recipients, online, viewing):
    rows = 1                                   # the message, once, in the conversation
    publishes = servers_touched(viewing)       # only servers with someone looking at it now
    frames = viewing
    return rows, publishes, frames


chats = {                                      # recipients, online now, viewing it now
    "1:1": (1, 1, 0),
    "group of 12": (11, 7, 3),
    "room of 50,000": (49_999, 6_000, 400),
}
print(f"{'':16}{'strategy':10}{'rows':>8}{'publishes':>11}{'frames':>8}{'work':>8}")
for name, shape in chats.items():
    for label, strategy in (("on write", on_write), ("on read", on_read)):
        rows, publishes, frames = strategy(*shape)
        print(f"{name:16}{label:10}{rows:>8,}{publishes:>11}{frames:>8,}{rows + publishes + frames:>8,}")

#                 strategy      rows  publishes  frames    work
# 1:1             on write         2          1       1       4
# 1:1             on read          1          0       0       1
# group of 12     on write        12          7       7      26
# group of 12     on read          1          3       3       7
# room of 50,000  on write    50,000          8   6,000  56,008
# room of 50,000  on read          1          8     400     409

Read the table by row, not by column:

• 1:1, on read, does almost nothing, and that’s the bug. The recipient is online but on another screen, so nobody tells them. A 1:1 chat must push.
• A group of 12 is cheap either way. 26 units of work against 7. Pick whichever makes unread badges and notifications simpler, which is usually on write.
• A room of 50,000 is where on write breaks. Fifty thousand inbox rows for one message, and a busy room sends several messages a second. At five a second that’s a quarter of a million rows a second, for one room.

So real systems are hybrids, and I’d design it exactly that way: fan out on write for small conversations (push to every online member, push-notify the offline ones, keep per-member state), and fan out on read for large rooms (store once, push only to viewers, compute unread from cursors when someone opens the room). The threshold is a product decision, typically somewhere in the hundreds of members. Discord’s engineering blog describes the large-room version: users who aren’t actively looking at a big server get “passive” sessions that don’t receive every event.

Slow readers and backpressure

Go back to the Beginner server’s loop:

for other in list(connected):
    await other.send(frame)

send returns once the frame is in the connection’s buffer. If a phone on a bad network stops reading, its buffer fills, and send starts waiting. Every socket after it in the loop waits too. One slow reader has become a slow server.

I measured it: 21 sockets, one of which takes 200 ms per write (a phone on a terrible connection), and 100 messages. First with inline sends, then with one bounded queue and one writer task per socket:

import asyncio
import time


class Socket:
    """A connection whose writes take `delay` seconds: a phone on a bad network, or a frozen tab."""

    def __init__(self, delay):
        self.delay, self.lag = delay, []

    async def send(self, sent_at):
        await asyncio.sleep(self.delay)
        self.lag.append(time.perf_counter() - sent_at)


async def inline(sockets, messages):
    """The obvious loop: for each message, await the send to every socket in turn."""
    for _ in range(messages):
        sent_at = time.perf_counter()
        for s in sockets:
            await s.send(sent_at)           # one slow socket holds up everyone after it
        await asyncio.sleep(0.01)


async def queued(sockets, messages, limit=32):
    """One bounded queue and one writer per socket. The reader never waits on a socket."""
    queues = {s: asyncio.Queue(maxsize=limit) for s in sockets}
    dropped = {s: 0 for s in sockets}

    async def writer(s):
        while True:
            await s.send(await queues[s].get())

    writers = [asyncio.create_task(writer(s)) for s in sockets]
    for _ in range(messages):
        sent_at = time.perf_counter()
        for s in sockets:
            try:
                queues[s].put_nowait(sent_at)
            except asyncio.QueueFull:
                dropped[s] += 1             # this reader is too far behind: tell it to resync instead
        await asyncio.sleep(0.01)
    await asyncio.sleep(0.05)
    for w in writers:
        w.cancel()
    return dropped


async def main():
    for name, design in (("inline sends", inline), ("bounded queue per socket", queued)):
        slow = Socket(delay=0.2)
        fast = [Socket(delay=0.0005) for _ in range(20)]
        dropped = await design([slow] + fast, messages=100)
        lag = sorted(x for f in fast for x in f.lag)
        print(f"{name:26} fast readers: median {lag[len(lag) // 2] * 1000:6.1f} ms, "
              f"worst {lag[-1] * 1000:7.1f} ms"
              + (f", slow reader dropped {dropped[slow]}" if dropped else ""))


asyncio.run(main())

# inline sends               fast readers: median  208.7 ms, worst   216.4 ms
# bounded queue per socket   fast readers: median    0.7 ms, worst     1.2 ms, slow reader dropped 62

With inline sends, twenty healthy readers inherited the slow one’s 200 ms on every message. With a queue per socket they got theirs in under a millisecond, and the slow reader’s queue filled and started refusing frames.

That last part is the decision you must make on purpose: what happens when a reader’s queue is full? There are three honest answers:

1. Drop the oldest frames. Fine for presence and typing, where only the latest state matters.
2. Disconnect the reader. It reconnects, and the reconnect catches up by sequence number. Blunt, and hard to get wrong.
3. Drop, and tell it to resync. Send one special frame that means “you’ve missed things, refetch”. The client does what it would do on a reconnect, without the reconnect.

The wrong answer is the default one: an unbounded buffer. In Python that’s a queue with no maxsize, and memory grows until the process dies. In Redis it’s worse, and subtler: if your subscriber reads from Redis more slowly than messages arrive, Redis buffers them on its side, and when that client’s output buffer passes the limit (by default, client-output-buffer-limit pubsub 32mb 8mb 60), Redis disconnects the subscriber. Your server loses its subscription, and if nothing notices, it keeps serving sockets that will never get another message. The production section has exactly this failure, measured.

Heartbeats: is anyone still there?

TCP is bad at noticing that the other side has vanished. A phone in a tunnel sends no goodbye, and a home router that drops an idle connection tells nobody, so the server’s socket stays “open” while writes pile into a buffer that will never drain. Meanwhile, load balancers close connections they think are idle; on AWS, an Application Load Balancer’s idle timeout defaults to 60 seconds.

Both problems have the same fix: send something regularly, and expect an answer. WebSockets have control frames for it, ping and pong, and most libraries do this for you. Python’s websockets defaults to a ping every 20 seconds and closes the connection if no pong arrives within 20 seconds. The logic is small enough to write out:

PING_EVERY, DEAD_AFTER = 20, 40         # seconds; the websockets library defaults to 20 and 20


def sweep(connections, now):
    """Run every PING_EVERY seconds: ping everyone, close whoever hasn't answered in time."""
    alive, closed = {}, []
    for user, last_pong in connections.items():
        if now - last_pong > DEAD_AFTER:
            closed.append(user)          # the TCP socket may still look open; the person is gone
        else:
            alive[user] = last_pong      # (a ping goes out here; the pong updates last_pong)
    return alive, closed


connections = {"aisha": 0, "ben": 0}
for now in range(20, 101, 20):
    connections["aisha"] = now           # aisha's phone answers every ping
    if now <= 20:
        connections["ben"] = now         # ben's answers once, then his train enters a tunnel
    connections, closed = sweep(connections, now)
    if closed:
        print(f"t={now}s closed {closed}; still open: {sorted(connections)}")

# t=80s closed ['ben']; still open: ['aisha']

Ben’s last pong was at 20 seconds and his connection was closed at 80. The server stopped holding memory for a dead socket, and his presence entry will expire on its own. Two details to get right: the heartbeat must be shorter than every idle timeout between client and server, and heartbeats must be cheap, because at scale they’re most of your traffic. The interview section counts them.

Browsers answer WebSocket pings automatically but can’t send them from JavaScript, so if the client wants to detect a dead server, the server should send application-level pings it can see (a {"type": "ping"} frame every 20 seconds), and the client should reconnect when they stop.

Authentication on the socket

The browser’s WebSocket API can’t set an Authorization header on the handshake. That surprises people, and it leaves four options:

• A cookie. Sent automatically with the handshake. But “automatically” means a malicious page can open a socket to your server with your user’s cookie attached. This is cross-site WebSocket hijacking, and the defence is to check the Origin header on every handshake.
• A token in the URL (wss://…/ws?token=…). Simple, and the token ends up in access logs, proxy logs and browser history. Avoid it for long-lived tokens.
• A token in the first frame. Accept the connection, wait briefly for {"type": "auth", "token": …}, verify it, and close the socket if it doesn’t come. The token never touches a URL or a log line. ezychat.io does this.
• A short-lived ticket. The client calls an authenticated HTTP endpoint, gets a single-use ticket valid for seconds, and passes it in the URL. A leaked ticket is worthless almost immediately.

Whichever you choose, two things are easy to forget. Re-check expiry while the socket is open. A request ends; a socket can outlive the token that opened it by days. And authorize every subscription, as well as the connection. The server decides which channels a connection hears, from the user’s memberships; a client never gets to name one. In a multi-tenant system every channel name includes the tenant, so a bug in one place can’t stream another customer’s messages.

Expert: offline devices, huge rooms and the interview

Push notifications for offline users

A phone in a pocket isn’t holding a WebSocket. Mobile operating systems suspend background apps to save battery, and a suspended app’s sockets die. The only way to reach it is the platform’s push service: Apple Push Notification service (APNs) for iOS, Firebase Cloud Messaging (FCM) for Android. Your server sends a notification to the push service; the push service, which holds the one connection the operating system keeps open, wakes the device.

The design question is when to use it. “The user has no socket” is too simple, because a socket can exist and be dead for a while (the heartbeat section’s tunnel). A better rule: push a frame to every socket the user has, wait a few seconds for the device to acknowledge it, and fall back to a push notification for any device that didn’t. Then collapse: if three messages arrive before Ben looks at his phone, he should see one notification that says “3 new messages”, not three buzzes.

ACK_WAIT = 5      # seconds a device has to acknowledge a frame before we fall back to a push


class Notifier:
    def __init__(self):
        self.pending = {}         # (device, conversation) -> unseen count, collapsed into one alert
        self.sent = []

    def deliver(self, device, conversation, seq, socket_acked):
        """socket_acked: None (no socket), or how many seconds the device took to ack the frame."""
        if socket_acked is not None and socket_acked <= ACK_WAIT:
            return "on screen"                              # it arrived live; a push would be noise
        key = (device, conversation)
        self.pending[key] = self.pending.get(key, 0) + 1
        count = self.pending[key]
        # One alert per conversation, replaced in place: APNs apns-collapse-id, FCM collapse_key.
        self.sent.append({"device": device, "collapse_id": conversation,
                          "text": "New message" if count == 1 else f"{count} new messages"})
        return "push"

    def opened(self, device, conversation):
        self.pending.pop((device, conversation), None)      # read: the next alert starts from one


n = Notifier()
print(n.deliver("ben-laptop", "c1", 41, socket_acked=0.2))
print(n.deliver("ben-phone", "c1", 41, socket_acked=None))    # the app is suspended in his pocket
print(n.deliver("ben-phone", "c1", 42, socket_acked=None))
print(n.deliver("ben-phone", "c1", 43, socket_acked=9.0))     # a socket, but it went quiet
print([s["text"] for s in n.sent])
n.opened("ben-phone", "c1")
n.deliver("ben-phone", "c1", 44, socket_acked=None)
print(n.sent[-1]["text"])

# on screen
# push
# push
# push
# ['New message', '2 new messages', '3 new messages']
# New message

The decision is per device: the laptop showed the message live and the phone still got a notification, because it’s a different screen. (Whether to buzz a phone while the laptop is active is a product choice.) The collapse identifier lets a new notification replace the old one, so the lock screen keeps one line per conversation.

And a push notification is not a delivery. It’s a hint that wakes the app, which then connects and catches up by seq like any reconnect. Push services rate-limit and drop, so never put anything in one the app can’t fetch again.

Multi-device sync

Once a user has a phone, a laptop and a tablet, “the user” stops being one recipient. Each device is its own:

• Delivery target, with its own socket, its own push token and its own last_seq per conversation. Ben’s phone might be at seq 38 while his laptop is at 42.
• Source of sends, with its own client ids. Prefix them with a device id, or use UUIDs, so two devices can never mint the same one.
• Echo target. A message Ben sends from his laptop must also appear on his phone. Treat the sender as a member like any other: the server fans the stored message out to all of Ben’s devices, and the laptop recognizes its own by client_msg_id.

Read state is per user: reading on the laptop should clear the badge on the phone. So read_upto lives on the server, keyed by user, and a change to it is an event fanned out to the user’s other devices.

WhatsApp’s multi-device design (Meta’s engineering blog, 2021) shows how far this goes with end-to-end encryption. Each device has its own identity key, and the sending client encrypts and transmits a message once per recipient device (“client-fanout”). The server can’t fan out copies it can’t read, so the sender’s phone does it.

Ordering across devices and people

The per-conversation sequence number gives one total order that every device agrees on, which is most of what users need. Three situations still look like ordering bugs to users, and a reviewer will ask about them:

• Your own message jumps. Ben’s phone shows his message at the bottom immediately. When the ack arrives with seq 42, a message from Aisha with seq 41 lands just above it. The phone must re-sort by seq, and should do it without a visible jump if it can.
• Two people send at once. Both phones show their own message last. The server’s order decides, and both phones converge on it. There’s no “right” order to argue about; there’s only the server’s.
• A reply arrives before what it quotes. When you ingest from somewhere else (a WhatsApp bridge, a federated server), message B quoting A can arrive first. Resolve the quote by id when you display it, not when you store it.

Across regions, a counter per conversation means every send for that conversation goes to the region that owns it. Systems that want writes in many regions at once reach for hybrid logical clocks, timestamps that combine physical time with a counter so cause always sorts before effect, at the price of the cheap gap detection. I’d keep a home region per conversation for as long as the latency allows.

Delivery guarantees, end to end

“Exactly once” isn’t available over a network, as the idempotency article explains, but “each message shows exactly once on each screen” is, and it comes from making every hop either at-least-once or recoverable, then deduping at the end:

The weakest hop is highlighted: pub/sub to the gateway is best-effort, and that’s deliberate. It’s the only hop on the hot path of every message, so it gets to be fast and lossy, and catch-up by sequence number covers it. If someone in a review says the pub/sub layer “guarantees delivery”, ask them what happens to a message published while a gateway is restarting. The honest answer is “it’s lost, and the client recovers it”, which is a perfectly good design as long as everyone knows it’s the design.

The other hop people get wrong is database to pub/sub. Publishing straight after the commit is a dual write: crash between the two and the message is stored but never pushed. Usually the next catch-up hides it. A transactional outbox (write the event row in the same transaction as the message, and let a relay publish it) closes the gap properly, and it’s what the production system below does.

Very large rooms

A room with 100,000 members breaks things that were fine at 50:

• Presence becomes a firehose. If every member’s online and offline changes go to every other member, a room of N people generates on the order of N² updates. Show a count (“2,314 online”), refreshed every few seconds, and look up individual presence only for the handful of members on screen.
• Typing indicators go off, or become “several people are typing”.
• The member list is lazy. Send the first page (online first, then by role) and load more on scroll, exactly like history.
• Slow mode. A per-member posting limit keeps a hot room readable, and bounds the fan-out cost too.
• Fan-out moves off the hot path. Discord’s 2017 post on scaling Elixir describes publishing an event from a large guild taking up to 2.1 seconds, and a library (Manifold) that groups recipients by node so a sender makes at most one send per remote node. Their 2023 post about a server with over a million online users reports that around 90% of user-guild connections in large servers were passive, making fan-out work 90% cheaper, and that a guild’s sessions were split across “relays” of up to 15,000 sessions each. The same post notes that the work to run a server grows quadratically with its size.

That last point is the one to carry into an interview. A room is quadratic in its size if everyone tells everyone everything. Every large-room technique is a way to stop telling everyone everything.

What Slack, Discord and WhatsApp have published

The posts are worth reading in full. Here’s what they documented, and nothing more:

• Slack (“Real-time Messaging”, April 2023) runs four core Java services. Gateway servers hold WebSocket connections and subscriptions, in several regions so clients connect to the nearest. Channel servers are stateful, in memory, hold some history per channel, and are mapped to channels by consistent hashing, about 16 million channels per host at peak. Admin servers sit between the web app and the channel servers, and presence servers track who’s online. A message goes from the client to the web app, an admin server, the channel’s channel server, and from there to every subscribed gateway. The post cites tens of millions of connected clients and 500 ms delivery worldwide.
• Discord partitions messages by channel and a “bucket” (a static time window), with time-sortable Snowflake ids. “How Discord Stores Trillions of Messages” (2023) describes moving from 177 Cassandra nodes to 72 ScyllaDB nodes, behind data services that coalesce requests, so many users asking for the same row cause one query. Its fan-out work for large servers is in the previous section.
• WhatsApp wrote in early 2012 that it had pushed a single server past two million simultaneous TCP connections, on FreeBSD with Erlang. And its 2021 multi-device design, above, moves fan-out to the sending client.

Notice the shared shape in the first two: a layer that holds connections (Slack’s gateway servers, Discord’s session processes), separated from a layer that owns each conversation and its order (Slack’s channel servers, Discord’s guild processes). That separation is the design to draw in an interview.

The interview: design WhatsApp (or Slack, or Messenger)

“Design a chat system” is a classic because it touches everything in this article. Here’s a structure that works, with the numbers worked out.

1. Requirements. Functional: 1:1 and group chats (groups up to, say, 500), text and media, delivery and read receipts, presence, typing, history on any device, push notifications when offline. Non-functional: a message appears within about a second for online recipients, messages are never lost once acknowledged, never duplicated on screen, and in the same order for everyone. Say out loud that “never lost” is about storage and “real-time” is about the socket, because the design keeps them separate.

2. Estimates. State the assumptions, then compute:

DAU = 50_000_000
SENT_PER_USER_PER_DAY = 40
PEAK_FACTOR = 3                    # the busiest hour versus the daily average
CONNECTED_AT_PEAK = 0.20           # share of daily users holding a socket at the peak
BYTES_PER_MESSAGE = 360            # measured: PostgreSQL 14, three indexes (see below)
ONLINE_RECIPIENTS = 1.5            # per message, mostly 1:1 chats with some groups
CONNECTIONS_PER_SERVER = 100_000   # a cautious plan, well under what memory alone allows
HEARTBEAT_SECONDS = 20

per_day = DAU * SENT_PER_USER_PER_DAY
avg = per_day / 86_400
peak = avg * PEAK_FACTOR
connections = DAU * CONNECTED_AT_PEAK
print(f"messages a day:          {per_day:,.0f}")
print(f"messages a second:       {avg:,.0f} average, {peak:,.0f} at peak")
print(f"frames pushed at peak:   {peak * ONLINE_RECIPIENTS:,.0f} a second")
print(f"storage:                 {per_day * BYTES_PER_MESSAGE / 1e9:,.0f} GB a day, "
      f"{per_day * BYTES_PER_MESSAGE * 365 / 1e12:,.0f} TB a year, before replicas")
print(f"open sockets at peak:    {connections:,.0f}")
print(f"gateway servers:         {connections / CONNECTIONS_PER_SERVER:,.0f}")
print(f"heartbeats:              {connections / HEARTBEAT_SECONDS:,.0f} a second, "
      f"{CONNECTIONS_PER_SERVER / HEARTBEAT_SECONDS:,.0f} per server")

# messages a day:          2,000,000,000
# messages a second:       23,148 average, 69,444 at peak
# frames pushed at peak:   104,167 a second
# storage:                 720 GB a day, 263 TB a year, before replicas
# open sockets at peak:    10,000,000
# gateway servers:         100
# heartbeats:              500,000 a second, 5,000 per server

Two inputs are measurements, and it helps to say where yours came from:

• 360 bytes per message. I loaded 200,000 messages shaped like the Intermediate table (uuid ids, a 63-character average body, the two unique indexes and the primary key) into PostgreSQL 14 on my laptop: 178 bytes per row of table and 182 of indexes.
• Connections per server. Idle connections to a Python websockets server on my laptop grew its resident memory by 15 to 28 KB each, across three runs of 5,000 and 10,000 connections (kernel buffers not included). At 28 KB, 100,000 connections is under 3 GB, so 100,000 per server is cautious. WhatsApp’s two million per box shows how far a tuned Erlang server goes.

The numbers say where the hard parts are. Storage is large but boring: shard messages by conversation id. The 500,000 heartbeats a second outnumber the messages, so heartbeats must be tiny and never leave the gateways. And the pub/sub layer sees about 104,000 frames a second at peak, so it needs sharding too.

3. API. On the socket, client to server: send {conversation_id, client_msg_id, body}, delivered, read and typing. Server to client: ack {client_msg_id, id, seq}, message, receipt, presence, typing, resync. On HTTP: history (?before=N, ?after=N), the conversation list and media upload, because history is cacheable, retryable and shouldn’t compete with live frames.

4. High-level design. The Advanced section’s figure: gateway servers that hold sockets and nothing else, a message service that assigns seqs and stores messages sharded by conversation, an outbox relay into sharded pub/sub, presence on Redis with expiring keys, and a notification service in front of APNs and FCM.

5. Deep dives the interviewer is likely to pick:

• Ordering and duplicates: a per-conversation seq under the conversation’s lock, client message ids under a unique constraint, and devices that drop seqs they already have.
• Offline: catch-up by seq on reconnect, collapsed push notifications, read state per user.
• Groups: fan-out on write below a member threshold, on read above it, presence counts in big rooms.
• A gateway dies or a client is slow: jittered reconnects with catch-up; a bounded queue and a resync frame.
• Hot conversation: one seq counter serializes a conversation’s sends. It’s the one part that doesn’t scale out, and it doesn’t need to: a conversation is bounded by how fast people type.

6. What breaks first. At these numbers, the pub/sub layer and the heartbeat load on the gateways, not the database. Say what you’d measure (frames per second per gateway, pub/sub output buffer sizes, reconnects per minute) and what you’d do (shard pub/sub, keep gateways stateless apart from sockets, and make every client reconnect with full jitter).

In production: the live inbox in ezychat.io

ezychat.io is a multi-tenant customer messaging product. Customers write to a business on WhatsApp, Instagram, Messenger or the web chat widget, and the business’s team (and its AI) answers from one inbox. The inbox is real-time: a customer’s message appears in an agent’s list without a reload, and so do replies, read state, AI handovers and lead updates. Here’s how, from the code.

From a webhook to an event

The real-time path starts with a message that isn’t ours: Meta delivers it by webhook. The handler verifies the signature over the raw bytes, redacts credentials, writes the payload to webhook_logs and hands off to a Celery task. The task stores the message through one function, deduped on the provider’s own id, scoped to the organization:

def insert_inbound_message_dedup(session, *, org_id, conversation_id, content_type,
                                 content, raw_payload, channel_msg_id) -> uuid.UUID | None:
    """Insert an inbound Message, short-circuiting cleanly on a duplicate
    `(org_id, channel_msg_id)` (a provider retry). Returns the new message
    id, or None if this delivery was already recorded."""

A duplicate delivery returns None and the ingest returns straight away. A new message carries on, and at the end of the same transaction, the ingest code writes the event the inbox will hear about:

emit_event(
    session,
    org_id=org_id,
    event_type="message.received",
    payload={
        "message_id": str(message_id),
        "conversation_id": str(conversation.id),
        "org_id": str(org_id),
        "channel": kind,
    },
)

emit_event writes a row to a domain_events table on the caller’s transaction: the outbox from the Expert section. A post-commit hook nudges a relay task, and the relay publishes that organization’s undispatched events to Redis and stamps them dispatched in the same transaction. Its failure branch states the delivery guarantee better than I could:

except redis.RedisError:
    # The rows published before this point stay published, and the
    # transaction rolls back, so they will be published again by
    # whoever picks the batch up next. Duplicate frames are the
    # correct trade against losing one: the client applies an event
    # it already has, where a lost event is invisible forever.

Delivery status webhooks take the same road: a WhatsApp “delivered” or “read” status updates the stored message and emits message.delivered or message.read, which is how an agent sees the ticks on their own reply change. A status that arrives before the message it describes is logged and dropped, because, as the docstring puts it, “a status can in principle race ahead of its content message”.

One channel per organization, with an allowlist

The handbook’s message-broker page describes the channel layout: “One channel per org, not per-resource: org:{org_id}:events.” Every API process subscribes once per open socket to the channel of that socket’s organization, and the tenant id in the channel name is the isolation boundary.

An org-wide channel carries events not every socket should see, so conversations/ws.py forwards only an allowlist of event types, each with a comment saying why it’s there. One entry is notable for being absent:

# `comment.received` is deliberately absent (plan Stage C, C1).
# Ingestion emits no event for it, so this entry would be dead — but
# it is also a guard: comment arrival is bounded by reach rather than
# by MAC, and one storm sharing this channel measured a 34x delay on
# conversation delivery for every org on the platform. If something
# starts emitting it again, this allowlist is where it must be
# argued for rather than where it quietly slips through.

That’s the large-room lesson in a different shape. Conversations are bounded by the customers a business actually talks to. Comments on a viral post are bounded by nothing. Putting both on one bus let the unbounded one slow the bounded one for everyone, by a measured factor of 34. So comments left the bus, and the inbox’s engagement feed polls instead, for a reason its comment states in one line: “A push design scales with the storm; this scales with the number of people looking at it.”

Some events are addressed to one person. Notifications ride the same org-wide channel, so the socket filters them per connection, and the comment is explicit about why it’s enforced at that point:

# Per-member events ride the same org-wide channel as everything else, so
# the addressee is enforced here: a notification is only ever framed to the
# member it belongs to, never to their colleagues' open tabs.
if event_type in _PER_MEMBER_EVENT_TYPES:
    if not isinstance(payload, dict) or payload.get("membership_id") != str(membership_id):
        return None

The web chat widget makes the opposite choice for the same reason: each visitor’s socket subscribes to exactly one channel, webchat:visitor:{visitor_id}, keyed by an opaque id. The module docstring: “one visitor’s socket can never receive another’s messages.”

Frames say what changed; HTTP says what it is

Look at the payload of message.received again: ids, nothing else. No message text. ezychat’s frames don’t carry content; they tell the browser that something changed, and the browser fetches it over the same HTTP endpoints it used to load the page:

onReady: (needsCatchUp) => {
  if (!needsCatchUp) return
  void refetchListHead()
  const current = activeItemRef.current
  if (current?.type === 'conversation') void refetchThread(current.id)
  // ...
},

On a message.received or message.sent frame, the inbox page refetches the head of the conversation list, and if that conversation is open, refetches its newest messages, merging them into what’s on screen by message id and sorting by created_at. That design makes the transport’s failures cheap. A duplicate frame causes a redundant refetch. A lost frame is repaired by the next one, or by the resync below. The frame never has to be correct, only timely.

It’s a different trade from the sequence-number catch-up taught in the Intermediate section, and I think it’s the right one here: an agent’s screen shows a list head and one open thread, both small, both one indexed query, so “go and look” is cheap and always correct. A consumer chat app with thousands of cached conversations on a phone would want sequence numbers instead, because refetching everything isn’t an option.

Authentication in the first frame

The module docstring of ws.py lays out the browser constraint and the decision:

Browsers cannot set custom headers on a WebSocket handshake, so the access token cannot ride the usual Authorization bearer header. It also must not ride the URL as a ?token= query parameter — query strings are routinely captured in server access logs and by intermediate proxies, which would leak live JWTs into logging infrastructure.

So the server accepts the connection, waits for a {"type": "auth", "token": …} frame, verifies the token, checks that the user’s membership in the organization is active, and closes with code 4401 otherwise. The token’s expiry is enforced while the socket lives, too: each heartbeat cycle checks it, and the comment explains the point: “Without this the connection (and its presence heartbeat) would outlive the token indefinitely.” The browser client treats a 4401 close as “refresh the session once, then reconnect immediately”.

The slow reader that lost frames silently

This is the Advanced section’s slow-reader problem, as it happened. The first version of the socket handler awaited each send inside the loop that read from Redis. The reader’s docstring records what that did:

async def _relay_reader(pubsub, channel_name, membership_id, queue, state) -> None:
    """Read the relay channel forever, handing frames to the writer.

    Never awaits the socket. That is the whole point: the send used to happen
    inline in this loop, so a client that stopped reading stopped this loop,
    Redis filled its output buffer for the subscriber and dropped it — silently
    on both sides, which is how a storm delivered 16,926 of 19,992 frames with
    no error anywhere and nothing undispatched in the database.
    """

Every layer reported success. The outbox had dispatched everything, Redis had dropped a subscriber that couldn’t keep up, as configured, and about 15% of frames never reached a browser.

The fix is the bounded queue from the Advanced section, with the third policy: drop and resync. Frames go into a queue of 256 per connection, a single writer task drains it to the socket, and when the queue is full the connection stops trying to describe what changed:

# Frames a connection may have outstanding before it is resynced instead.
# Sized to hold an ordinary burst — an agent bulk-resolving a queue, a deal
# room filling — without ever becoming a place a backlog can accumulate: past
# this many frames behind, refetching is both cheaper and more correct than
# replaying, because the client has no way to tell how far behind it is.
_SEND_QUEUE_MAX_FRAMES = 256

When it overflows, the queue is emptied and one {"type": "resync"} frame goes in its place. The client treats it exactly like a reconnect: refetch, don’t replay. And then there’s the detail I’d never have thought of without measuring it, a cooldown:

# The shortest gap between two resync frames on one connection. A resync costs
# the client a full refetch, so telling a still-stalled client again and again
# turns one slow tab into a burst of requests against `/inbox` — measured at 35
# for a single overload before this existed.
_RESYNC_COOLDOWN_SECONDS = 5.0

Writing the resync frame doesn’t mean the client read it; send_json returns once the frame is buffered. Without a cooldown, the queue refills behind the resync and overflows again. The handbook records that a probe holding a socket deaf drew 35 resync frames from a single overload, and one with the cooldown: 35 refetches from one slow tab, aimed at the endpoint every recovery depends on.

Two more repairs close the loop. If the Redis subscription itself dies, the reader resubscribes after a second and queues a resync, because “a subscription that dies takes the gap with it”. And if forwarding stops for any other reason, the endpoint closes the socket with 1011 rather than keep sending pings over a connection that delivers nothing. The comment: “A visible close is strictly better than a silent one: the client reconnects, and its reconnect already carries a catch-up refetch.” The handbook sums up the real-time bus’s guarantee as “Best-effort, never silent”, and that’s the phrase I’d want on every pub/sub layer.

Heartbeats and presence

The socket sends {"type": "ping"} after 20 seconds without a frame from the browser, and every cycle refreshes the member’s presence key, which lives for 60 seconds. Closing a tab sends nothing; the key just expires, and a second open tab keeps it alive. The refresh detects the offline-to-online edge in the same command:

async def heartbeat_presence(org_id: uuid.UUID, membership_id: uuid.UUID) -> bool:
    """Refresh the member's presence TTL. Returns True on an offline→online
    transition (the caller uses that to re-evaluate team queues). Fail-open:
    Redis trouble reports no transition and never raises."""
    try:
        redis = get_redis()
        previous = await redis.set(
            presence_key(org_id, membership_id), "1", ex=PRESENCE_TTL_SECONDS, get=True
        )
        return previous is None
    except Exception:
        logger.warning("presence heartbeat skipped (Redis unavailable) for org %s", org_id)
        return False

SET ... GET writes the key and returns what was there before, so “there was nothing” means “they just came online”, and conversations waiting in that agent’s team queue get routed to them at once.

Presence in ezychat is two facts, and the presence module’s docstring separates them. Connected comes from the heartbeat. Active comes from the browser sending an activity frame at most once a minute while the person is actually using the tab; ten minutes without one reads as Away. The server ignores activity frames that arrive faster than that, so “a misbehaving client cannot turn frames into writes”. Idle only changes the dot, never routing: “a team that steps away from the keyboard is never stranded”.

Typing, and what a receipt claims

The typing endpoint is quoted in the rate limiter article, and it’s the Intermediate section’s rules in a few lines:

# One call per conversation per five seconds. A composer fires these on
# keystrokes, so a wedged tab is the normal way this turns into a bill;
# being rationed is already an ordinary answer here, not an error.
verdict = await ratelimit.consume("conversation_typing", str(conversation_id), 1, 5)
if not verdict.allowed:
    return {"sent": False}

The bill is real because this typing indicator goes to the customer, through the messaging channel’s own API, not to a colleague’s browser. And on WhatsApp, as the service module notes, “its typing indicator and its mark-as-read are the same API call”. That ties typing to read receipts, and the module that owns both opens with a rule I’d copy into any chat system:

# **A read receipt means a person looked at this.** Exactly two callers may
# make that claim: the router, when a human opens a thread, and the AI
# pipeline, immediately before it replies.

The code enforces it: both functions take an emitter argument that must be "human" or "ai", and a test reads the source to assert that no third call site exists. A webhook processor, a list refresh or a background sweep can’t mark a customer’s message as seen, because none of them saw it.

Reconnecting with full jitter

The browser client, lib/ws.ts, reconnects on any unexpected close with exponential backoff from 1 second up to 30, and its comment explains why the jitter spans the whole window:

// Full jitter: uniform across the whole window rather than the window's
// end plus a 30% smear, which put every open tab in a 300ms band. A server
// restart wakes all of them at once, and each one's `onReady(true)` is a
// refetch of the very endpoint the recovery depends on — so the spread is
// the protection, and a narrow spread is barely any.
const ceiling = Math.min(BASE_BACKOFF_MS * 2 ** this.attempt, MAX_BACKOFF_MS)
this.attempt += 1
const delay = MIN_BACKOFF_MS + Math.random() * (ceiling - MIN_BACKOFF_MS)

Every reconnect is a catch-up, so every reconnect is a burst of reads. Without jitter, a deploy is a thundering herd aimed at the history endpoint at the exact moment the servers are coldest.

History: keyset on time and id

Message history is cursor-paginated, newest first, with the time-plus-id tie-breaker from the Intermediate exercise:

async def list_messages(session, *, org_id, conversation_id, limit, cursor):
    """Cursor-paginated message history for one conversation, newest first
    (matches the `ix_messages_convo_created_id` index)."""
    stmt = select(Message).where(
        Message.org_id == org_id, Message.conversation_id == conversation_id
    )
    if cursor:
        ts, cursor_id = parse_datetime_cursor(cursor)
        stmt = stmt.where(
            or_(
                Message.created_at < ts,
                and_(Message.created_at == ts, Message.id < cursor_id),
            )
        )
    stmt = stmt.order_by(Message.created_at.desc(), Message.id.desc()).limit(limit + 1)

The cursor is the last row’s (created_at, id), base64-encoded so clients treat it as opaque. Fetching limit + 1 rows answers “is there more?” without a count: if the extra row came back, there’s another page. The index matches the sort exactly, which is what keeps page 500 as fast as page 1.

The other platform: a WebSocket hub at 5,000 messages an hour

I also architected the backend of a WhatsApp operations platform for construction companies, where an AI agent lives in project WhatsApp groups. Its real-time layer was a different stack with the same shape: a Laravel REST API, a Swoole WebSocket hub holding the connections, Redis Pub/Sub for real-time delivery between them, and a WhatsApp gateway, at about 5,000 messages an hour. The same separation as Slack’s and Discord’s, at a much smaller size: one layer holds sockets, another owns the messages, and pub/sub carries notifications between them.

What I’d copy

• Keep the database as the truth and the socket as a hint. Frames can be lost, duplicated or late; a refetch, or a catch-up by sequence number, makes the screen right again.
• Put the event in the same transaction as the change, and accept duplicate frames as the price of never losing one.
• Bound every connection’s queue, and make overflow loud and specific: one resync, a cooldown, a log line with the count.
• Never let a broken stream look alive. Close the socket when forwarding stops; a client with no reason to reconnect never recovers.
• Decide what each bus is for, and keep unbounded traffic off the bus that carries bounded traffic.
• Reconnect with full jitter, because every reconnect is a read storm in waiting.

Cheat sheet