Varnish cache warming: keep your hit rate up after restarts, bans and deploys

Varnish has one architectural honesty that most caches hide: it forgets everything when it restarts. The default malloc storage lives in RAM — a restart, a VCL reload gone wrong, or a server reboot means your hit rate drops to zero and your backend eats the full load until the cache refills organically. Add the bans and purges that every deploy fires, and "Varnish makes my site fast" quietly becomes "Varnish makes my site fast for pages somebody recently visited."

When Varnish runs cold

• Restarts — malloc storage is gone entirely; even file storage starts logically empty (the pointers don't survive).
• Bans — ban req.url ~ . after a deploy invalidates everything matching; the objects die on next lookup.
• Purges — targeted, but a CMS that purges generously (many do) empties whole sections.
• TTL expiry — long-tail pages with little traffic expire and stay cold until the next unlucky visitor.

The Varnish-side toolbox

1. Grace mode — soften the cliff

Grace is Varnish's stale-while-revalidate: serve the expired object now, fetch a fresh one in the background. It's the single best-value line of VCL for hit rates:

sub vcl_backend_response {
  set beresp.grace = 6h;   # serve stale up to 6 h while revalidating
}

But grace only helps objects that are in the cache. After a restart or a broad ban there's nothing to serve stale — grace does nothing for a truly cold cache.

2. A sitemap crawl — the actual warm-up

The classic approach is a script that reads your sitemap and requests every URL through Varnish (not around it — the request must arrive on the frontend port so the hash matches real traffic):

curl -s https://example.com/sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | xargs -P 4 -n 1 curl -s -o /dev/null -H 'User-Agent: warmer'

Three details decide whether this works or silently lies to you:

• Match the Vary. If your VCL or backend varies on Accept-Encoding, device type or a language header, warm each variant you care about — a gzip-warmed object doesn't help a brotli client on older Varnish setups, and a desktop-warmed page misses a mobile-varied cache.
• Don't trigger hit-for-miss. If the backend answers a warming request with Set-Cookie or Cache-Control: private (say, your warmer hit a page that starts a session), Varnish stores a hit-for-miss marker instead of the object — and now real traffic bypasses the cache for that URL for up to the marker's TTL. Strip cookies for anonymous traffic in vcl_backend_response.
• Pace it. A warm-up that hammers the backend with 50 parallel requests defeats its own purpose. 2–4 concurrent requests is plenty.

3. hash_always_miss — refresh without purging

For planned content updates, there's a more elegant move than purge-then-warm: a request with hash_always_miss fetches a fresh copy and replaces the cached object, with zero cold window:

sub vcl_recv {
  if (req.http.X-Warm == "${secret}") {
    set req.hash_always_miss = true;
  }
}

Point your warmer at the sitemap with that header and you get an atomic refresh: visitors are served the old object until the new one lands. (warmup.rocks projects can send a custom secret header for exactly this pattern.)

The layer Varnish can't reach

Here's the part most Varnish guides skip: Varnish runs in one place — your data center. If you put a CDN in front (Cloudflare, Fastly, CloudFront — and most Varnish sites do), you now have two cache layers, and warming Varnish fills only the bottom one. The CDN still caches per edge location: a visitor in Singapore is served by the Singapore edge, which has its own cold cache regardless of how warm your Varnish is.

The two layers compose nicely though: a warming request that reaches the Singapore edge and MISSes there travels to your origin — where warm Varnish answers it in milliseconds instead of seconds. The edge stores that fast response, and both layers are now hot. That's why a multi-location warmer and Varnish are complements, not alternatives: warmup.rocks requests your sitemap from 90+ edge locations, filling every CDN colo and keeping Varnish itself hot as a side effect. You can see both layers in one look with the cache checker — x-cache/Age from Varnish, cf-cache-status or x-served-by from the CDN.

Checklist

1. Set beresp.grace generously (hours, not seconds).
2. Strip cookies for anonymous pages so warming (and browsing) never creates hit-for-miss markers.
3. Warm from the sitemap after every restart, deploy and broad ban — paced, variant-aware.
4. Use hash_always_miss with a secret header for zero-downtime refreshes.
5. If a CDN sits in front: warm it per location — that's the layer your far-away visitors actually hit first.