How to stop relying on stale deployment checks

The release input stayed the same, but the service’s access rule changed between the check and the request. You can reproduce this sequence with Python’s standard library on a local machine without needing a cloud account.

Make a passing check go stale#

The demonstration starts an HTTP service on 127.0.0.1 with two endpoints: /policy returns the current access rule, and /ledger applies that rule to incoming requests.

Initially, the rule allows a caller named checkout:

{"allowed_callers": ["checkout"]}

The checker fetches /policy and saves an evidence receipt with the collection start and finish timestamps, the target address, and a SHA-256 hash of the proposed request (GET /ledger with the test header X-Caller: checkout). Evaluating that saved evidence returns PASS, and an actual request to /ledger returns HTTP 200.

Next, the script updates the service’s access rule:

{"allowed_callers": []}

When the script evaluates the saved evidence again and repeats the HTTP request, the cached check still returns PASS, while /ledger now returns 403. Because the checker has no TTL or invalidation mechanism, it treats a valid historical answer as permission to act now.

Run the example#

Download the script and run it with Python 3.9 or newer:

python3 reproduce.py --output demo-output

The script starts and stops its own local server, uses no cloud credentials, and installs no dependencies. The server is for this experiment only; X-Caller is a test label, not authentication.

Here is the output from the run used for this article. The first line identifies the fixed request specification:

change_sha256=3de7abcb81077964d33625db73bb0407ca583e471efeb97b476d7dd8fd482456
baseline: check=PASS request_http=200
policy_changed: cached_check=PASS request_http=403
fresh_check: check=BLOCK change_unchanged=true policy_changed=true
collector_failed: check=HOLD collector_http=503 request_http=200
assertions=passed

This run used Python 3.12.14 on macOS. The script writes evidence.json with the observation timestamps, policy hashes, and target address. The recorded evidence and transcript accompany the script; your timestamps and local port will differ when you run it.

This lab is an application access-control experiment. It does not run a Kubernetes cluster, perform a deployment, measure an outage rate, or exercise Proofline. The PASS, BLOCK, and HOLD labels belong to the demonstration, not a Proofline product run.

Refresh the evidence before using the result#

In the fresh_check step, the checker fetches /policy again before evaluating the request. The request hash matches the original, while the policy hash has changed. Evaluating the refreshed rule returns BLOCK.

That is the practical adjustment to the release workflow: keep the historical review record, but refresh the dependencies behind its conclusion before relying on it.

For a small system, re-reading the required inputs at release time is often easier to operate than maintaining an invalidation graph of everything that could alter a result. Record when collection starts and finishes, and keep those observation timestamps separate from when the check evaluation completed. A check that finished just now might still be evaluating yesterday’s data.

A maximum evidence age is useful, but a TTL window cannot detect every relevant change. The demonstration runs its operations back to back with no artificial wait, and its timestamps simply record when observations occurred rather than acting as a timing benchmark. An environment change can happen inside whichever freshness window you choose; there is no five-minute threshold that solves that problem for every release.

Refresh the dependencies that determine the answer#

The local service has one rule, so identifying its dependencies is straightforward. Production environments require tracking every input that determines the answer.

For a Kubernetes connection, the relevant inputs can include ingress and egress network policies along with the pod and namespace labels their selectors match. Because policies combine additively, a newly created policy or a label change can alter connectivity even when every policy in the saved snapshot remains unchanged. The Kubernetes network-policy documentation explains how selector matching and isolation work.

Collecting those configuration inputs still does not prove that DNS, TLS, or the application works. A conclusion about policy configuration has a narrower scope than a live request.

You also need to check which consistency properties the APIs you read provide. Separately listing policies, pods, and namespaces does not produce an atomic snapshot of the whole cluster. Kubernetes documents its list and resource-version semantics, including why a resource version is not an elapsed-time measurement.

Continuous invalidation can reduce unnecessary refreshes, but it must handle newly created matching objects, selector changes, and missed updates. Include that operational work when comparing continuous tracking with a simpler release-time read.

Keep unavailable evidence separate from a blocked request#

In the collector_failed step, the script restores access on /ledger but makes /policy return HTTP 503.

The live request to /ledger succeeds with HTTP 200, while the check returns HOLD because it cannot obtain the required policy evidence. Falling back to the original PASS would hide that collection failure. Returning BLOCK would also misdescribe the state, because the checker has not established that the current rule rejects the request.

A required release gate needs a policy for unavailable evidence. Decide before a hotfix is waiting whether the release waits and who can override a HOLD, and preserve both the reason and the identity behind any override. An advisory review check can instead notify the person performing the release that the evidence needs refreshing. Without that upfront policy, the first collector failure becomes a debate about whether an old green result is probably good enough.

Even after a fresh read succeeds, an interval remains between reading the evidence and performing the release action, and a dependency can change during that window. If that race between checking and acting is unacceptable, coordinate the relevant changes or enforce the constraint directly at the point of use. Refreshing the evidence narrows the interval, but it does not make the race disappear.