Software updates are the delivery workload nobody celebrates and everybody depends on: a background service that must reach every machine, survive every network, and never, ever ship the wrong bytes. The engineering is unglamorous — immutable paths, range resumes, staged percentages, a fallback URL — and precisely because updates carry executable code to your entire install base, the unglamorous parts are load-bearing.
The update workload, honestly described
Update traffic has a distinctive shape: near-zero baseline, then a release spike that decays over days as the fleet checks in, repeated forever — plus a long tail of old versions updating from years back. Clients are unattended background processes on every network quality that exists, including metered, flaky and censored ones. And the stakes are asymmetric: a slow update is invisible; a corrupted, truncated or attacker-substituted one is an incident with your product’s name on it. Those three facts drive everything: the spike wants CDN economics and staging, the client diversity wants resumability and patience, and the stakes want cryptographic verification end to end — delivery is the transport for a signature scheme, never the security itself.
Artifact layout: immutable, signed, delta-friendly
Publish every artifact under an immutable versioned path — /releases/4.2.1/app-linux-x64.tar.zst — with infinite TTLs and the immutable directive; the only mutable objects in the estate are the small metadata files clients poll (“what is current for my channel and platform?”), which get short TTLs measured in minutes. That split — heavy immutable payloads, tiny mutable pointers — is the whole caching design, the same discipline as asset versioning. Sign artifacts and verify in the client (a checksum fetched over the same channel proves integrity, not authenticity — you need a signature against a key the client already holds); serve deltas from recent versions with the full artifact as fallback, since deltas are routinely an order of magnitude smaller and the fleet mostly sits one version behind; and never delete old versions on a whim — the long tail and your own rollback both depend on them.
The client contract: ranges, resume, backoff
The updater is half the delivery system, and its manners decide your incident rate. It must download with range requests and resume from the last verified byte — on big artifacts over real-world networks, restart-from-zero multiplies delivered bytes and support tickets together (edge range behaviour on large objects: see the range-requests piece). It must back off exponentially with jitter on failure and honour 429 and Retry-After, because a synchronized fleet retrying in lockstep is a self-inflicted DDoS. It should jitter its scheduled check-ins so daily polls do not arrive as a global pulse, respect metered connections, and verify the signature before replacing anything, atomically, with the previous version kept for rollback. None of this involves the CDN directly — and all of it determines what the CDN sees.
Staged rollout and the kill switch
Never point the whole fleet at a new version at once. The metadata service is your rollout dial: release to one percent, watch crash reports and update-success telemetry, widen to ten, fifty, everyone — with cohorting stable per machine so a device does not flap between versions. This staging is simultaneously a delivery control (it converts the release wall into a managed ramp, shrinking the peak the CDN must absorb) and a safety control (the blast radius of a bad build is the current percentage). Pair it with a kill switch that actually stops the bleed: flipping current-version back must propagate through those short-TTL metadata files in minutes, which is exactly why the pointers get short TTLs while payloads get infinite ones. Rehearse the reverse path once — a rollback that has never been exercised is a hope, not a control.
Mirrors, fallbacks and measuring health
Updates must work on the day the primary path does not, so build the fallback in: a second hostname on a different CDN (or a direct origin/mirror path) listed in the client’s configuration, tried automatically after the primary fails N times — the poor operator’s multi-CDN, and often sufficient for update traffic; the full version is proper failover. Keep origin load honest with a shield and collapsing during release spikes, per reducing origin load. Then measure the funnel, not the server: check-ins, downloads started, completed, verified, installed — segmented by version, platform and network — because update health is a fleet property, and the gap between “downloads completed” and “installed” is where corrupted-byte and disk-full realities live. An update pipeline with staged rollout, signature verification, resume and a fallback path is boring by design; in this workload, boring is the deliverable.