Check a migration against the previous application
When a pull request changes a database migration, Proofline can run the previous version of your application's tests against the migrated schema in a sandbox. A test that passes on the base's schema and fails on the migrated one becomes a finding on the pull request. During a rolling deploy, the previous version keeps running against the new schema, and after a rollback that keeps the migration it runs there again.
When the check runs#
Proofline runs the check beside a pull request's review when all of these hold:
- Verification in a sandbox is on for the repository, or for the organization when the repository follows the organization's choice. It is off by default, and the console offers no setting for it.
- The pull request changes a migration: a
.sqlfile undermigrations/, a file named likeV2__add_orders.sqlor0042_orders.up.sql, a file underdb/migrate/oralembic/versions/, or a Python, TypeScript or JavaScript file undermigrations/. - The repository's sandbox recipe has a PostgreSQL database. Proofline records the database once a verification run has applied your migrations to it.
Review depth does not start the check. A standard review and an in-depth review run it under the same conditions.
The review publishes without waiting for the check. A finding from the check arrives on the pull request when the check finishes, usually after the review's own comments.
Which previous version runs#
The previous application is the version you deploy, which is often older than the pull request's base. When the review read an environment, and Kian records which commits that environment's workloads serve, the check runs each of those commits that is the base or an earlier commit of it, up to three. Both sides of a traffic split run.
Otherwise the check runs the pull request's base, and the finding says that the deployed revision was not established. A deployed commit that is newer than the base, such as a later build of the default branch, does not run, because the migrated schema starts from the base's migrations.
How the check decides#
The check builds the base's schema with your repository's migrate command, applies the pull request's migrations to a copy, and compares the two. It looks for a dropped or renamed table or column, a new NOT NULL column without a default or a column made NOT NULL, and a changed column type.
For each such change, it runs the previous version's tests that name the changed table or column:
- on the base's schema, where the test must pass;
- on two fresh copies of the migrated schema;
- with the pull request's own code on the migrated schema, which is shown beside the result and never makes a finding.
A test becomes a finding only when it passed on the base's schema, failed on both migrated copies, and both failures drew a PostgreSQL error that names the changed table or column. A test that fails under several previous versions is one finding.
Read a finding#
The finding is titled The old application's tests fail on this migration's schema, at warning severity, on one of the pull request's changed migrations. It says what the migrations did to the table or column, and names the failing test, the command the check ran and the PostgreSQL error code. It names the previous version it ran: each revision Kian records as serving that reproduced the failure, or the pull request's base with the deployed revision not established. Its evidence cites the test file at each of those revisions.
The finding claims only that this test of the previous version failed against this schema in the sandbox. It does not say that production will fail, or that it will be fine if no finding appears. The sandbox run shows what the code did against a schema built from your migrations. It does not show your production schema or its drift, data volume, lock time, long-lived connections, other applications that share the database, code paths your tests do not reach, down migrations, or any database other than PostgreSQL.
When a later push makes the same failure, the finding joins the existing thread and is not posted again. Reply on the thread to record a decision, as for any finding.
After a later push#
On each later push that still changes a migration, the check runs the finding's test again on the previous versions that hold it, before any other test:
- Verified resolved. The migrations still make the change, and the test passed under every previous version that ran it. Proofline adds a line to the thread's first comment saying the old application's test ran again on a later version's migrated schema and passed, and resolves the thread.
- Still present. The test failed again. Proofline replies once on the newest thread that the test fails again on this version's migrated schema.
- Fix not verified. The check could not settle the test, for example because the test failed on the base's schema, passed once and failed once, or did not reach a changed table, or because the sandbox stopped. The finding stays open, and Proofline posts nothing on the pull request.
When the migrations no longer make the change the finding names, or the pull request no longer changes a migration, the finding resolves as any finding does when a push removes its cause. That is not a verified fix: no test ran to show it.
The finding's page in Proofline lists each re-verification under Re-verification, with the commit, the outcome, the test it ran and, when the fix was not verified, the reason. Read what Proofline writes on a pull request explains the replies on other findings.
When the check finds nothing#
A check that cannot judge records why and makes no finding. Common reasons are a sandbox recipe without a database, a migrate command that fails on an empty database, a schema change the check does not cover, or no test of the previous version that names a changed table.
A test runner that rebuilds its own schema for each run tests that schema instead of the migrated one, and the check ignores the result. Add the runner's flag that reuses the existing database, such as --keepdb, to the recipe's test command.
No finding means that no test of the previous version failed against the migrated schema in a way the check could attribute to the change. It does not mean the migration is safe to deploy.