Cache Invalidation at Scale: Redis Streams, SSE, and Varnish PURGE

The problem: two hard things in computer science

The old joke is that there are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors. Cache invalidation earns its spot because caching is easy in isolation — the hard part is telling every layer that's holding a copy of the data that it's now stale, without either missing a copy (stale reads) or purging so aggressively you lose the point of caching in the first place.

That gets harder as soon as more than one service is involved. If the service that writes content is also the one that has to know about every Varnish node, every SSE listener, and every downstream cache, it ends up doing infrastructure work that has nothing to do with its actual job. The fix is to split it into two services with one job each:

• Caching service — owns the data, serves cached reads, and knows the instant something changes
• Invalidation service — owns nothing but the fan-out: turning one "this changed" signal into every purge and notification that needs to happen because of it

The two are connected by a signal, not a function call — so the caching service doesn't need to know or care who's listening.

Architecture at a glance

Why Redis Streams, not plain Pub/Sub

Redis Pub/Sub is tempting because it's simple — PUBLISH and SUBSCRIBE, done. But it's fire-and-forget: if the invalidation service is mid-deploy or briefly disconnected when the event fires, that event is gone forever, and whatever it should have purged stays stale with no way to know. Streams fix exactly that gap.

The signal: what actually gets published

The caching service publishes one small, self-describing event — nothing about who needs to act on it:

XADD cache:invalidations * \
  entity   "article" \
  id       "12489" \
  urls     "/news/12489,/news/12489/amp,/api/articles/12489" \
  tags     "section:news,author:42" \
  reason   "content-updated"

The urls field drives the Varnish purge; the tags field lets the invalidation service do broader sweeps later (e.g. "purge everything by author 42") without the caching service needing to enumerate every affected URL up front.

Fan-out target 1: Varnish PURGE and BAN

Varnish gives you two invalidation primitives, and the invalidation service picks whichever fits the event:

• PURGE — removes one exact object from cache immediately. Cheap, precise, the default choice when the event lists specific URLs.
• BAN — marks anything matching a pattern as stale; a background lurker thread reclaims it lazily. Used for tag-based sweeps where enumerating every URL isn't practical.

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge_acl) {
            return (synth(405, "Not allowed"));
        }
        return (purge);
    }
}

# Invalidation service, per URL in the event:
curl -X PURGE https://edge-01.internal/news/12489
curl -X PURGE https://edge-02.internal/news/12489

# Tag-based sweep via BAN:
varnishadm ban "obj.http.x-tags ~ section:news"

Fan-out target 2: SSE for the live broadcast

Varnish handles the HTTP cache layer, but plenty of things hold their own copies too — an admin dashboard showing "last purged," an edge worker keeping a local LRU cache, another service instance with an in-process cache. Polling Redis for changes wastes cycles and adds latency; a raw WebSocket is more than the job needs, since nothing ever needs to talk back up the connection. Server-Sent Events fits the actual shape of the problem: one-directional, plain HTTP, automatic reconnect built into every browser and most HTTP clients.

// Invalidation service — SSE endpoint
app.get('/events/invalidations', (req, res) => {
  res.set({
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    Connection: 'keep-alive',
  });

  const send = (id, data) => {
    res.write(`id: ${id}\n`);
    res.write(`data: ${JSON.stringify(data)}\n\n`);
  };

  const lastId = req.headers['last-event-id'] || '$';
  subscribeFromStreamId(lastId, (id, event) => send(id, event));

  req.on('close', () => unsubscribe());
});

The client's Last-Event-ID header is the piece that matters most: if a browser tab or edge worker drops its connection for ten seconds, reconnecting sends that header automatically, and the invalidation service resumes the Redis Stream read from exactly that ID — no gap, no missed purge, no client-side bookkeeping required.

Putting it together

Handling the failure modes

• Invalidation service restarts mid-purge — the stream entry stays unacknowledged and is reclaimed by the consumer group on restart; the purge simply re-fires.
• An SSE client disconnects briefly — it reconnects with Last-Event-ID and the invalidation service replays everything it missed straight from the Redis Stream.
• One Varnish node is down — its purge call fails independently of the others; a dead node serving stale content until it recovers is an acceptable trade-off against blocking every other node's purge on one failure.

Key takeaways