Deploying is not releasing

TL;DR: Deployment installs an artifact; release exposes behaviour to a chosen audience. A feature flag can separate those decisions and stop future exposure without a rebuild. It cannot undo external effects or guarantee that inactive code has no runtime impact.

The symptom this explains

A feature is turned off at 09:40 after a bad launch, and the errors continue until 09:41. Somebody concludes the flag did not work, and starts looking for a bug in the flag system.

The flag worked. Three separate things were still true: clients were holding a cached rule for up to a minute, requests already in flight were still executing the old decision, and orders written during the bad window had already been created. A flag controls future exposure, and each of those three is something else. Knowing which of them you are looking at is most of what this page is for.

Two controls with different evidence

Deploying a candidate puts its code and dependencies into the running environment. Even with a feature disabled, module initialisation, memory use or a background task can change. Verify the deployed artifact's basic operation before opening the feature to users.

Releasing the feature changes who executes a path. Stable tenant or user cohorts can make exposure predictable. Request-level allocation can suit some stateless experiments, but it may produce inconsistent experiences within a session. Choose the cohort unit from the behaviour being tested.

Worked example: a pricing path behind a flag

In a fictional checkout service, the old and new pricing implementations ship in one artifact. The new path is initially disabled. A shadow comparison computes both prices from the same input but allows only the established path to create the order. The shadow path must suppress payment, email and inventory effects, and its extra computation must fit the service's capacity.

An internal cohort then uses the candidate price, followed by a small customer cohort under agreed acceptance conditions. Compare price correctness and checkout completion alongside latency. A small audience limits direct exposure, but a shared database or cache can spread impact beyond that cohort.

If the result is wrong, stop new exposure through the flag. Suppose clients cache rules for sixty seconds: some can continue using the old rule during propagation. Test that delay, define behaviour when the flag service is unavailable and verify the affected instances actually adopted the change. Already-created orders require a separate reconciliation decision.

Read the lifecycle

Retire temporary controls deliberately

Every temporary rollout flag needs an owner and a removal condition. After the rollout is accepted, test the retained path directly and remove obsolete branches. Check older application versions before deleting a flag definition that their SDK still evaluates. An operational kill switch may remain permanently, but its fallback and both states still need periodic exercises.

Several interacting flags create combinations that individual flag tests miss. Identify the combinations that affect the same request or shared state and exercise them explicitly. An evaluation count is useful cleanup evidence, but an infrequent recovery switch may be intentionally dormant.

Changes that need another recovery plan

A flag cannot reconstruct a dropped column, unsend an email or make an old consumer understand a new message envelope. Those changes need compatible sequencing, repair or compensation. Keep artifact promotion and schema recovery plans connected to the flag lifecycle.

Continuous deployment can also ship small compatible changes directly with tests and progressive routing. Flags are useful where independent exposure control earns its maintenance cost. Required approvals and deployment windows can still apply to either operation.

Self-check

A new checkout path is released to 5 percent of customers at 09:00. At 09:38 the error rate on that cohort is clearly elevated and the flag is disabled globally at 09:40. Errors continue until 09:41, and two customers report a duplicate charge. The rollout also ran a shadow comparison beforehand, which called the payment provider and discarded the HTTP response.

Say whether the flag failed, account for the 60 seconds of continued errors, decide whether disabling the flag resolves the incident, and say what the shadow comparison did. Work it through before reading on.

The flag did not fail, and the evidence is that the errors stopped without anything else being done. What produced the 60 seconds is three distinct mechanisms, and separating them matters because they have different fixes. Clients cache the flag rule, so on a 60-second TTL some instances continued evaluating the old value until their cache expired; that is propagation and it is a tunable. Requests already in flight at 09:40 had made their decision and completed under the old path; that is not propagation and no TTL change affects it. And the flag state has to actually reach every instance, so a proportion of them should be confirmed as having adopted it rather than assumed to have.

Disabling the flag does not resolve the incident, which is the part most often got wrong. It stops new exposure, and it does nothing about effects already written: the orders created during those 38 minutes exist, priced by the path you have just decided was wrong. Those need a separate reconciliation decision, and it is a product decision rather than an engineering one, since the options are to honour them, reprice them or refund. A flag is a control over future behaviour and never a rollback of completed work.

The duplicate charges come from the shadow comparison, and this is the trap in the question. Discarding the HTTP response does not undo a charge; the request was sent, the provider processed it, and the response you threw away was the record of that. A shadow path must suppress the side effect at its source rather than at its result, which means not making the call at all, or routing it to a provider sandbox, and the general rule is that the safe thing to duplicate is the input and the computation, never the effect. It is also a reminder that the shadow's extra work has to fit inside the service's capacity, since it doubled the pricing computation on live traffic.

So the incident has three separate remaining tasks and only one of them was the flag: reprice or refund the affected orders, fix the shadow path so it cannot transact, and decide whether a 60-second flag TTL is acceptable given that it sets a floor on how fast you can stop exposure.

Next: mitigate before you diagnose covers the order these actions belong in, and delivery metrics counts what the repair deployments do to the numbers.

Mobile delivery adds signing and store distribution to the release decision, including the recovery work required for clients already installed on users’ devices.