---
title: "How to stop relying on stale deployment checks | Proofline"
description: "A deployment check can stay green after the environment it verified has changed. In a small local experiment, a cached check returned PASS while the same HTTP request that succeeded moments earlier failed with 403 Forbidden. Refreshing the evidence updated the check result to BLOCK."
url: https://proofline.sh/learn/production-evidence-freshness/
---

[Learning center](/learn/)

# How to stop relying on stale deployment checks

A deployment check can stay green after the environment it verified has changed. In a small local experiment, a cached check returned PASS while the same HTTP request that succeeded moments earlier failed with 403 Forbidden. Refreshing the evidence updated the check result to BLOCK.

Deployment failures The Proofline team November 4, 2026 4 min read

The release input stayed the same, but the service’s access rule changed between the check and the request. You can reproduce this sequence with Python’s standard library on a local machine without needing a cloud account.

## Make a passing check go stale

The demonstration starts an HTTP service on `127.0.0.1` with two endpoints: `/policy` returns the current access rule, and `/ledger` applies that rule to incoming requests.

Initially, the rule allows a caller named `checkout`:

```
{"allowed_callers": ["checkout"]}
```

The checker fetches `/policy` and saves an evidence receipt with the collection start and finish timestamps, the target address, and a SHA-256 hash of the proposed request (`GET /ledger` with the test header `X-Caller: checkout`). Evaluating that saved evidence returns `PASS`, and an actual request to `/ledger` returns HTTP `200`.

Next, the script updates the service’s access rule:

```
{"allowed_callers": []}
```

When the script evaluates the saved evidence again and repeats the HTTP request, the cached check still returns `PASS`, while `/ledger` now returns `403`. Because the checker has no TTL or invalidation mechanism, it treats a valid historical answer as permission to act now.

## Run the example

Download [the script](/downloads/stale-deployment-checks/reproduce.py) and run it with Python 3.9 or newer:

```
python3 reproduce.py --output demo-output
```

The script starts and stops its own local server, uses no cloud credentials, and installs no dependencies. The server is for this experiment only; `X-Caller` is a test label, not authentication.

Here is the output from the run used for this article. The first line identifies the fixed request specification:

```
change_sha256=3de7abcb81077964d33625db73bb0407ca583e471efeb97b476d7dd8fd482456
baseline: check=PASS request_http=200
policy_changed: cached_check=PASS request_http=403
fresh_check: check=BLOCK change_unchanged=true policy_changed=true
collector_failed: check=HOLD collector_http=503 request_http=200
assertions=passed
```

This run used Python 3.12.14 on macOS. The script writes `evidence.json` with the observation timestamps, policy hashes, and target address. The [recorded evidence](/downloads/stale-deployment-checks/evidence.json) and [transcript](/downloads/stale-deployment-checks/transcript.txt) accompany the script; your timestamps and local port will differ when you run it.

This lab is an application access-control experiment. It does not run a Kubernetes cluster, perform a deployment, measure an outage rate, or exercise Proofline. The `PASS`, `BLOCK`, and `HOLD` labels belong to the demonstration, not a Proofline product run.

## Refresh the evidence before using the result

In the `fresh_check` step, the checker fetches `/policy` again before evaluating the request. The request hash matches the original, while the policy hash has changed. Evaluating the refreshed rule returns `BLOCK`.

That is the practical adjustment to the release workflow: keep the historical review record, but refresh the dependencies behind its conclusion before relying on it.

For a small system, re-reading the required inputs at release time is often easier to operate than maintaining an invalidation graph of everything that could alter a result. Record when collection starts and finishes, and keep those observation timestamps separate from when the check evaluation completed. A check that finished just now might still be evaluating yesterday’s data.

A maximum evidence age is useful, but a TTL window cannot detect every relevant change. The demonstration runs its operations back to back with no artificial wait, and its timestamps simply record when observations occurred rather than acting as a timing benchmark. An environment change can happen inside whichever freshness window you choose; there is no five-minute threshold that solves that problem for every release.

## Refresh the dependencies that determine the answer

The local service has one rule, so identifying its dependencies is straightforward. Production environments require tracking every input that determines the answer.

For a Kubernetes connection, the relevant inputs can include ingress and egress network policies along with the pod and namespace labels their selectors match. Because policies combine additively, a newly created policy or a label change can alter connectivity even when every policy in the saved snapshot remains unchanged. The [Kubernetes network-policy documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) explains how selector matching and isolation work.

Collecting those configuration inputs still does not prove that DNS, TLS, or the application works. A conclusion about policy configuration has a narrower scope than a live request.

You also need to check which consistency properties the APIs you read provide. Separately listing policies, pods, and namespaces does not produce an atomic snapshot of the whole cluster. Kubernetes documents its [list and resource-version semantics](https://kubernetes.io/docs/reference/using-api/api-concepts/), including why a resource version is not an elapsed-time measurement.

Continuous invalidation can reduce unnecessary refreshes, but it must handle newly created matching objects, selector changes, and missed updates. Include that operational work when comparing continuous tracking with a simpler release-time read.

## Keep unavailable evidence separate from a blocked request

In the `collector_failed` step, the script restores access on `/ledger` but makes `/policy` return HTTP `503`.

The live request to `/ledger` succeeds with HTTP `200`, while the check returns `HOLD` because it cannot obtain the required policy evidence. Falling back to the original `PASS` would hide that collection failure. Returning `BLOCK` would also misdescribe the state, because the checker has not established that the current rule rejects the request.

A required release gate needs a policy for unavailable evidence. Decide before a hotfix is waiting whether the release waits and who can override a `HOLD`, and preserve both the reason and the identity behind any override. An advisory review check can instead notify the person performing the release that the evidence needs refreshing. Without that upfront policy, the first collector failure becomes a debate about whether an old green result is probably good enough.

Even after a fresh read succeeds, an interval remains between reading the evidence and performing the release action, and a dependency can change during that window. If that race between checking and acting is unacceptable, coordinate the relevant changes or enforce the constraint directly at the point of use. Refreshing the evidence narrows the interval, but it does not make the race disappear.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free150)

## Related reading

-   ### [Lower code review costs and catch deployment risks with Proofline](/learn/reduce-ai-code-review-costs/)
    
    Deployment failures The Proofline team November 16, 2026 4 min read
    
-   ### [Reviewing AI-generated code](/blog/review-ai-generated-code/)
    
    Deployment failures The Proofline team October 7, 2026 2 min read
