Safe Deployments: Blue/Green, Canary and Feature Flags
ReliabilityShip a change to a small slice first, watch error rate and latency against the old version, and be able to undo it in seconds. Blue/green swaps traffic between two full environments, a canary exposes a few percent of users, and feature flags separate shipping code from turning it on.
Testing a new recipe at a restaurant: serve it to one table first, watch whether they send it back, and keep the old recipe ready on the stove. You only change the whole menu once that table is happy.
Why deploys are risky. Most outages follow a change, and the change that breaks things usually looks fine in testing. A safe release process assumes some changes will be bad and limits the damage: expose few users first, measure, and roll back fast.
Rolling update. Replace servers a few at a time. It needs no extra capacity, but during the rollout both versions serve traffic, so they must be compatible, and rolling back means another rollout. It is the default, and it is rarely enough alone.
Blue/green. Run the current version (blue) and the new version (green) side by side. Test green with real settings, then switch the load balancer from blue to green in one step. Rollback is switching back, which takes seconds. The cost is doubled capacity for the switch and the need to handle shared state, like the database, carefully.
Canary. Send a small share of real traffic, such as 1%, to the new version, and compare its error rate and latency with the old version's at the same time. If the numbers hold, widen to 5%, 25%, then 100%. A real user hits the new code, but only a few of them, and the old version is the live control group.
def canary_ok(baseline, canary, max_error_ratio=1.5, max_latency_ratio=1.2, min_requests=1000):
if canary['requests'] < min_requests:
return 'wait' # too little data to judge
err_base = max(baseline['errors'] / baseline['requests'], 0.0005) # floor avoids dividing by zero
err_canary = canary['errors'] / canary['requests']
if err_canary > err_base * max_error_ratio:
return 'rollback'
if canary['p99_ms'] > baseline['p99_ms'] * max_latency_ratio:
return 'rollback'
return 'promote'
baseline = dict(requests=99_000, errors=99, p99_ms=180)
print(canary_ok(baseline, dict(requests=400, errors=0, p99_ms=170))) # wait
print(canary_ok(baseline, dict(requests=1_000, errors=1, p99_ms=185))) # promote
print(canary_ok(baseline, dict(requests=1_000, errors=9, p99_ms=190))) # rollbackJudge by numbers, not feelings. Decide the checks before the rollout: error rate, tail latency (p99), saturation, and one business metric such as checkout success. Use a minimum sample size and a bake time, because a bug that needs a cache to warm or a nightly job will not show in five minutes. Automate the rollback so it does not wait for a person to notice.
Feature flags. A flag wraps new behavior so code can be deployed turned off, then enabled for staff, then 1% of users, then everyone, with no new deploy. Turning it off is the fastest rollback there is. Keep flags short-lived (delete them after the rollout) and give each user a stable bucket, such as a hash of the user id, so one person does not flip between versions.
Config is code. Many large outages came from a configuration push, not a code release. Treat config changes the same way: validate them, roll them out in stages, watch the same metrics, and keep the last good version ready to restore.
Database changes. The schema must work with both the old and the new code during a rollout. Use expand and contract: first add the new column or table (expand), deploy code that writes both, backfill, switch reads, and only later remove the old one (contract). Never ship a change that the previous version cannot survive.
- Limit blast radius: expose few users first, compare with the old version, and be able to undo in seconds.
- Blue/green swaps two full environments; rollback is switching back.
- A canary compares real traffic on the new and old versions using preset thresholds, a minimum sample and a bake time.
- Feature flags separate deploying code from enabling it; keep them short-lived and bucket users stably.
- Roll out config like code, and migrate schemas with expand and contract.
Curated Curation & Deep Insights
ScaleDojo CertifiedOur system architects are vetting high-quality, authorized video guides for Safe Deployments: Blue/Green, Canary and Feature Flags with zero third-party platform links.
We are preparing premium, zero-competitor deep-dives for Safe Deployments: Blue/Green, Canary and Feature Flags. Authorized reading references will appear automatically.