Benchmarks measure a CDN’s best behaviour; trials measure its dashboard; neither measures how it handles your traffic — your weird query strings, your token scheme, your long-tail object set, your 3 a.m. crawler storms. Shadow testing does: the candidate platform serves copies of real traffic while users depend only on the incumbent, and every difference surfaces with nobody watching. Three setups, in rising fidelity and effort — most evaluations should climb all three.
Why shadow beats benchmarks and trials
What kills CDN migrations is never headline latency — it is semantics: a platform that normalizes query strings differently and fragments your cache, treats a Vary header your app emits literally and craters hit ratio, handles range requests on your large objects with different edge behaviour, or 400s the one malformed-but-tolerated header your oldest mobile app still sends. None of that appears in a benchmark (which fetches clean URLs, fairly, per the method) or a trial checklist. It appears when real request diversity meets the candidate’s real behaviour — which is exactly what shadow setups manufacture, safely. Shadow is also the proof phase every larger decision leans on: phase 1 of a migration, the readiness gate before a platform joins a live split, and the honest core of any bake-off.
Setup 1: log-driven synthetic replay
The entry setup needs no production changes: configure the candidate with a test hostname pointing at your origin (or better, a staging origin — see the etiquette in section five), then replay request samples drawn from your real CDN logs — the URL, method, and safe headers of each request re-issued against the candidate’s hostname by a replay runner. Sample deliberately rather than firehosing: the top-N popular paths (cache behaviour), a random tail slice (the objects that expose miss-path performance), every path class in your routing rules, and a curated “awkward file” of the weird-but-real requests your logs contain — odd encodings, huge query strings, ranges, legacy user agents. Replays only GETs and other safe methods, never mutating traffic. What it yields: cache behaviour per class (are the right things caching, with the right TTLs?), miss-path latency, and a first sweep of outright errors — the coarse filter that eliminates unsuitable platforms in days, cheaply.
Setup 2: response diffing, the semantic net
The middle setup catches what replay alone cannot: for each sampled request, fetch through both platforms — incumbent and candidate — and diff the responses systematically: status, body (hashed, with dynamic regions masked), and the headers that matter (Content-Type, Cache-Control as served, Vary, compression, CORS, your security header set), plus the cache-status headers read per platform’s dialect. Route the diffs into a report ranked by frequency, and drive the count toward explained-or-fixed: every persistent diff is either a candidate misconfiguration (fix it now, while it is a config ticket rather than an incident), an incumbent behaviour you were unknowingly depending on (the truly valuable finds), or an acceptable cosmetic difference (document it). This is the same cross-edge comparison machinery the config-sync suite runs forever after — building it during evaluation means the estate inherits its drift detector for free.
Setup 3: the dark-launch slice
The highest-fidelity setup puts real users on the candidate — a small, revocable slice: a low DNS weight on a non-critical hostname (static assets are the classic canary class), or a specific low-stakes hostname assigned outright. Now you measure what only real clients generate: RUM cut by platform (the comparison populations arrive automatically once the platform dimension is in your beacon, per the RUM setup), the full diversity of real networks and devices, connection reuse and protocol behaviour under genuine browsing patterns, and cache performance under true popularity distributions rather than sampled ones. Prerequisites are exactly the go-live checklist in miniature — valid certificates, token parity, origin allowlisting for the candidate’s ranges — which is the point: the dark launch is a dress rehearsal of onboarding, and its friction list is your real integration estimate. Hold the slice for at least two weekly cycles, and keep the revert one weight-change away throughout.
Etiquette, costs and reading the results
Three pieces of etiquette keep shadow testing clean. Origin: shadow doubles fetches for whatever misses, so point replay at a staging origin where possible, keep production-origin shadow load capped and shielded, and remember the candidate needs allowlisting like any CDN per origin protection — then remove it if the evaluation ends in a no. Provider: tell the candidate you are shadow testing — trials have terms, replay at volume can trip abuse heuristics, and a named contact converts anomalies into answers instead of mysteries (their engagement quality during evaluation is itself a data point about support later). Money: shadow traffic is billable traffic on usage terms; cap it and watch it. Reading results: score against the criteria you wrote before testing — the requirements sheet from your one-pager — not against the most recent interesting graph; shadow produces so much data that unanchored evaluations drift toward whatever the candidate happens to be good at. The output worth having is one page: parity findings fixed and outstanding, performance deltas by region and class, integration frictions, and a recommendation your future self can audit.