your deploy strategy matters less than you think
Table of contents
- Recreate — Kill everything. Start over.
- Rolling — Change the tire while the car is moving.
- Blue/Green — The mirrored building.
- the myth of the rollback
- Canary — Three tables in the back to see if anyone throws up.
- the math nobody wants to do
- Shadow — Dirty work in the dark.
- the blast radius
- the database nightmare
- tactical checklist
Friday, 3:12 PM. Routine deploy. We changed the user_data.sh on the instance — nothing exotic. Rolling update, like we always did.
The Auto Scaling Group’s warmup time was set to 5 minutes. But with the new dependencies we added, the instance took 9 minutes to boot. The ASG, impatient, marked the instance as unhealthy. Killed it. Spun up another. Which also didn’t boot in time. Which was also marked unhealthy.
In just a few minutes, a cascading failure. 2 instances became 20. None healthy. Latency exploded and the screen was painted with 5xx errors everywhere.
We had an elegant Rolling deploy on paper, but a catastrophic Recreate in practice.
The root cause wasn’t the strategy. It was an ignored warmup. A secondary script with no load testing. A default value we accepted blindly.
Teams spend weeks debating whiteboard architectures as if they were magic shields against production outages. They aren’t.
The strategy is just the envelope. What dictates whether your system soars or sinks are the silent parameters: the readiness delay, the warmup time, the connection draining, the batch size. Having a perfect Canary rollout with a generic health check is like jumping out of a plane with a state-of-the-art parachute tied with a square knot. The design is flawless, but the execution will be fatal.
Think of your infrastructure as a busy restaurant being renovated. Three questions make the strategies easier to compare:
- Does the service stay up? The dining room keeps operating while the kitchen changes.
- How much does it cost to keep running? Do you need to duplicate space, staff, or equipment?
- If it fails, how many people feel it? Does the new kitchen start with a few orders or serve everyone at once?
The three corners answer those questions. The closer the shape gets to a corner, the better the strategy tends to perform there.
quick comparison
what each strategy prioritizes
closer to a corner = better at that goal
Rolling
Replaces a few instances at a time.
- strong point
- keeps the service running without cloning the whole fleet
- in exchange
- v1 and v2 must work correctly at the same time
To prove this, each strategy below has an interactive simulation you can (and should) break. The point isn’t to pick the “winner” — it’s to understand why none of them save you on their own.
About the model: Timings are compressed to fit the page, while failures, latency, and response divergence use illustrative probabilities. Each run can vary. Compare cause and effect — do not treat the numbers as production benchmarks.
Recreate — Kill everything. Start over.
The old version (v1) is completely shut down. The environment hits zero. The new version (v2) boots from scratch. No overlaps. No coexistence.
In Practice: You nail a sign to the door: “Closed for renovations”. You turn off the instances. For 30 days, nobody is served. When it reopens, the operation is 100% new and cohesive. No mixing old state with new schemas.
How it works on the Server:
- Kill all v1 traffic.
- Wait for full shutdown.
- Start the v2 instances.
- Turn traffic back on.
Where it makes sense: When your environment rejects concurrency. A structural DB change that fatally breaks the previous version.
The Cost: Absolute downtime. No crying.
| Pros | Cons |
|---|---|
| No version coexistence | Inevitable downtime |
| No version collision | Users completely locked out |
| Lowest complexity | Rollback requires immediate re-deploy |
deploy config
Break the simulation: Increase the cluster to 4 instances and watch draining time, cost, and the amount of recycled capacity grow. Because the new instances boot in parallel, cluster size alone does not automatically extend downtime; the window is dominated by how long the first v2 takes to become ready for traffic.
Rolling — Change the tire while the car is moving.
Version B replaces version A gradually. One out, one in. Piece by piece until the environment is 100% updated.
In Practice: With minActive, readiness, and draining configured correctly, the service remains available. You drain one machine from the pool, boot its v2 replacement, and repeat. During the transition, some traffic reaches the old version and some reaches the new one — the ratio changes with every batch.
The Catch: The database must serve v1 and v2 at the same time. If both versions are not compatible with the same schema, reads and writes can fail throughout the coexistence window.
The Cost: Capacity, rollout speed, or temporary surge. Your operational limits decide which bill arrives first.
| Pros | Cons |
|---|---|
| Continuous availability with correct safeguards | Can be slow |
| Familiar and widely supported pattern | Requires strict backward compatibility |
| Low aggregate cost | Drain errors are fatal |
deploy config
strategy parameters
Break the simulation: Set
minActiveto zero and max outbatchSize. Boom. You allowed one batch to drain all capacity, turning your refined Rolling deploy into a disguised Recreate. Then raisebootTimebeyond the readiness delay and health-check tolerance: premature instances enter the traffic path, fail, and trigger replacements.
Blue/Green — The mirrored building.
High cost. Reduced risk. Version B (Green) rises in a parallel, equivalent environment and is tested outside the primary traffic path. Once it crosses the safety line, routing moves to the new environment.
In Practice: You preserve the active environment while provisioning and testing Green. All clear? A Load Balancer can switch destinations almost immediately; a DNS change still depends on TTLs and caches to converge. If readiness, capacity, and dependencies are correct, clients do not notice the cutover.
The Collateral Limit: Green can start with cold caches, unwarmed connections, and configuration drift. Sharing the same database also preserves state-related risk between both environments.
The Cost: Application capacity can approach 2x during the overlap. That does not necessarily mean doubling the entire monthly cloud bill.
| Pros | Cons |
|---|---|
| No-downtime cutover when properly prepared | Near-double capacity during the overlap |
| Fast traffic rollback | Does not undo state already changed |
| Robust pre-production smoke tests | Cold caches can saturate dependencies |
deploy config
strategy parameters
Break the simulation: Open the strategy parameters and set Cache pre-warming to zero. At cutover, watch the Thundering Herd: Green passes a shallow health check, but the first requests contend for cache and database resources. The gateway starts returning 504s because its upstreams cannot respond in time. Healthy code; unavailable service.
the myth of the rollback
Blue/Green is celebrated for the dream: I hit Undo in five seconds. That can be true for traffic, but not for state. You can roll back code. You cannot automatically roll back everything it already changed.
- The Event: Kafka posted the notification. 12 APIs consumed it. No Ctrl+Z.
- Side-Effects: The machine ran webhooks. Email notified the users. The payment gateway pulled funds from credit cards.
- The DB Altered: Did you drop that dead table in the DB while asynchronous jobs still queried
user_id_legacy? The bleeding has started.
Distributed systems don’t accept “Ctrl+Z”. Limiting exposure in small steps (for example, with Canary) is safer than relying on an emergency rollback. Infrastructure can move back while the data wound remains open. Assume some effects are irreversible and design for smaller steps and fix-forward recovery.
Canary — Three tables in the back to see if anyone throws up.
If Blue/Green proved that you cannot undo state, Canary asks a sharper question: what if you limit the damage before you need to go back?
Canary is one of the strongest tools for limiting blast radius. It sends a small fraction of traffic to version B, observes it for a defined window, and only then promotes it to larger percentages.
In Practice: You begin with a small group of users or requests. If the signals degrade, you stop promotion while most of the operation continues on v1.
It is a disciplined way to find regressions before they reach the entire user base.
The Cost: Time, observability, and statistical decision criteria defined before deployment.
| Pros | Cons |
|---|---|
| Controlled blast radius | Demands reliable automation |
| Real production signals | Samples can be insufficient at low volume |
| Signal-driven promotion | Slow by design |
deploy config
strategy parameters
Do not confuse A/B testing with Canary: An A/B test compares product outcomes between variants. A Canary rollout limits operational exposure while validating whether a new version is safe. They can use similar routing machinery, but they answer different questions.
the math nobody wants to do
Teams buy the Canary hype with a token “15 minutes at 5%” pause before the blind jump to 100%.
If an independent failure affects 0.1% of requests, approximately 3,000 Canary requests provide only a 95% chance of observing at least one occurrence: 1 - (1 - 0.001)^n ≥ 0.95. At a 5% Canary allocation, that requires about 60,000 total requests. Detecting a degradation against a baseline with confidence usually requires an even larger sample and an explicit statistical test.
Letting 5% of low-volume traffic trickle for 15 minutes is not technical rigor; it is security theater. The observation window must follow from event frequency, traffic volume, and the signal that will actually drive the decision.
Shadow — Dirty work in the dark.
In this simulation, every request is copied to the Shadow environment. In production, you can also mirror only a sample. Version A keeps responding to the user while version B processes the copy — measuring latency, CPU, errors, and response content — without returning its result to the client.
In Practice: You run the new version under real production traffic without placing it in the response path. To avoid adding latency or load to production, mirroring must be asynchronous, isolated, and bounded.
The Side-Effect Trap: Without isolation, the Shadow version sends emails, charges cards, and writes to the database a second time. Shared resources can also saturate and affect production even when the Shadow response is discarded.
The Cost: With full mirroring and an equally sized fleet, compute can approach 2x during the validation window. Sampling and a smaller Shadow fleet reduce that bill.
| Pros | Cons |
|---|---|
| New response stays outside the user path | Extra cost proportional to mirroring |
| Performance and divergence under real traffic | Side-effect isolation is difficult |
deploy config
Break the simulation: Watch the yellow counter: in this configuration, the Shadow fleet duplicates all production capacity without serving external responses. The cost buys comparable evidence, not automatic safety. If the response diverges or mirroring fails, users stay on v1 — but your validation is incomplete.
the blast radius
You’ve seen five strategies, each answering the same three questions differently. But strategy is a what. The more dangerous question is where.
Deploys don’t just happen at the server level. The scope of the change — your blast radius — scales from a single CPU thread to intercontinental DNS routing:
Simple
Lower cost / Higher risk
Complex
Higher cost / Lower risk
Recreate
Total downtime
Rolling
Gradual update
Blue/Green
Exact copy, doubles cost
Canary
Percentage routing
Shadow
Mirrored traffic
| Layer | Control Boundary | Common Technologies |
|---|---|---|
| OS Processes | Ports / IPC | Gunicorn, PM2 |
| Pods / Containers | Ingress / Service Mesh | Kubernetes, Envoy |
| Instances / VMs | Load Balancers (ALB/NLB) | AWS EC2, Azure VMs |
| Global Traffic | Edge Routing / DNS | Cloudflare, Route53 |
Saying you “do Canary” in a standup is vague. Are you isolating 2% of requests in a local load balancer, or routing 10% of European traffic at the edge? Without naming the exact blast radius, you do not know the scale of a possible failure.
the database nightmare
Blast radius tells you how far the damage spreads. The database tells you where it stays.
If there is a brutal force capable of crushing your perfect Blue/Green or Rolling architecture, it’s the relational database.
The old version may not understand changes introduced by the new one. If a deploy removes a column that v1 instances still read, those requests begin to fail. The database is the shared-state cemetery where elegant deployment strategies go to die.
To survive complex schema changes, use the Expand & Contract pattern (Parallel Change):
- Expansion: Add the new columns or tables. Update the code to write both old and new representations while reads remain compatible with the old schema.
- Migration: Move historical data to the new representation in the background, with checkpoints and verification.
- Contraction: Weeks or months later, after telemetry and code confirm that nothing reads the old schema, remove the legacy field. An apparently simple change can span several releases before it is safely complete.
tactical checklist
The Friday that started this post — we had the right strategy and the wrong parameter. Every simulation you broke above proved the same thing from a different angle.
Before applying the next infrastructure change, validate:
- Parameters dictate operational behavior. A short
timeout, premature readiness, or an aggressive batch size can turn a gradual rollout into an outage. - State never rolls back. Your database has no emergency undo button. Design flows assuming failures will leak into persistence and prioritize continuous correction (fix-forward).
- Statistical samples are unforgiving. Running a Canary with 5% traffic for 10 minutes under low volume is just generating metric theater. If the volume doesn’t reach statistical significance, the alert will be blind.
At the end of the day, the strategy expresses intent. Production behavior comes from the values someone placed in the configuration.
results from this page’s simulations
Request Cemetery
Failures accumulate here while you experiment with the strategies.
The cemetery is empty
Run a simulation above. Results from this session will appear here.