Every provider wins its own benchmark. The only test that matters runs on your traffic, your regions and your percentiles, and it is easier to run honestly than most teams expect.
Measure users, not data centers
Synthetic probes from cloud regions measure the distance between data centers, which nobody sells tickets to. Real-user measurement, timing actual visitor requests, is the ground truth. Free RUM tooling and resource-timing APIs make this a configuration task, not a project. The bar is lower than it looks: you do not need research-grade instrumentation, only the discipline to measure both candidates identically.
Compare percentiles, per region
Averages hide exactly the users a CDN exists to help. Compare p75 and p95 by region, because the median user in Frankfurt was always fine; the decision is about the slow tail in Jakarta. Weight regions by revenue, not by request count. Percentiles also change conversations internally. An average improves and nobody feels it; a p95 improves and support tickets quietly thin out, which is the difference between a chart and an outcome.
One design decision deserves special care: trial length versus traffic cycles. Two weeks is the minimum because it contains two of everything weekly, two peak days, two quiet Sundays, two Monday morning surges. Shorter trials sample a single cycle and inherit its accidents: one provider’s trial week contained your marketing launch, the other’s contained a holiday. If your business has monthly rhythms, payday spikes, end-of-month reporting traffic, consider stretching to cover one full month. The principle is boring and absolute: benchmark across the cycles your traffic actually has, or the calendar will pick your CDN for you.
Hold everything else still
Same origin, same cache rules, same time window, traffic split randomly rather than by geography. Two weeks of clean data beats two months of confounded data. Write down the success criteria before you look at results, the oldest trick for staying honest. Random assignment is the detail most bake-offs fumble. Splitting by geography or by content type builds the bias into the experiment, and the winner becomes whoever received the friendlier slice.
In practice
Write a one-page protocol before contacting any vendor: metrics, regions, weights, duration, success thresholds. Share it with both candidates so nobody can dispute the rules after losing. This document takes an hour and converts the entire evaluation from a sales process you attend into an experiment you run. The psychological difference is the point.
This is the methodology behind our index. The assessment runs it for you, instrumented and umpired.