Configure pull request checks

Add a pre-merge check to a repository to check each reviewed pull request’s title, description, issue reference, labels or changed-line test coverage. Proofline reports each check as its own GitHub Check, named Proofline check / followed by the check's name.

A blocking and an advisory pull request title check on the fictional store-api repository.
Each check reports as its own GitHub Check on every reviewed head. Fictional demo workspace.

Before you start#

Use an organization administrator account, or an account that administers the repository on GitHub. Write access to the repository is not enough. If your GitHub sign-in has expired, Proofline asks you to sign in again. Once the repository has a check, the Checks page shows how many are active and how many the repository can have.

Add a check#

  1. Open Repositories.
  2. Open the repository and select Settings → Checks.
  3. Select Add check.
  4. Enter a Name of lowercase letters, digits or hyphens. The name cannot change later.
  5. Choose the Rule:
    • Pull request title: enter the Title pattern. It matches anywhere in the title unless you anchor it with ^ or $.
    • Description: the description must have text. Whitespace and HTML comments left by a pull request template do not count.
    • Issue reference: the branch name, title or description must reference an issue, such as #7, owner/repo#7 or an issue link.
    • Labels: enter the Required labels, one per line. The pull request must carry every one; case does not matter.
    • Instructions: describe the Condition in plain language, such as "A change to a public API handler updates api/openapi.yaml.", and choose What decides it. A verifier reads your instructions beside the pull request's title, description and changed files, and answers met, not met or undecided. A repository can have 10 active instruction checks.
    • Coverage: configure the report below.
  6. Optionally enter Paths (optional), one glob per line, to limit the check to changes that touch them. Coverage checks use their own excluded paths instead.
  7. Choose the Mode.
  8. Select Add check.

If Proofline cannot read the pattern as a regular expression, the page shows the reason and keeps what you entered.

Add a check for every repository#

An organization administrator can add a check that applies to every repository. Open Settings → Review policy → Pre-merge checks, then add it the same way. Coverage checks read one repository's reports, so they are added per repository.

A repository's Checks page lists the organization's checks beside its own, marked Organization. Only an organization administrator changes them. A name belongs to one check: a repository check cannot reuse an organization check's name, and a .proofline.yaml check with that name is ignored.

Define checks in .proofline.yaml#

A repository can define checks in its .proofline.yaml, with version: 2:

version: 2
checks:
  - slug: conventional-title
    rule: TITLE_PATTERN
    pattern: "^(feat|fix|docs): "
    mode: BLOCKING
  - slug: api-labels
    rule: REQUIRED_LABELS
    required_labels: [api-reviewed]
    path_globs: ["api/**"]
    mode: ADVISORY

Each review reads the file at the pull request's merge base, not at its head. A pull request that edits or removes a check in the file is still checked against the checks its base defines, and its change applies to pull requests opened after it merges. The Checks page lists file checks with the commit they were read from, and you change them in the file. A check you define in settings keeps its name: a file check with the same name is ignored. Coverage checks are defined in settings.

How the verifier decides an instruction check#

For each reviewed commit, the verifier makes one call to your organization's review model, which uses credits like any review turn. It reads your instructions, and treats the title, description and changed files as data: text in the pull request that claims the check is met does not change the answer. The Check shows the verifier's explanation and the files it cites.

The check is undecided, and a blocking check needs action, when the verifier cannot answer. The Check names the cause: the organization has no review model, its credits cannot pay for the call, the model provider did not answer, or the answer could not be used. A check that needs an issue tracker or a document is always undecided for now, because the verifier cannot read either yet.

Limit a check to paths#

With paths, a check applies only when the pull request changes, adds, renames or deletes a matching file, compared with its merge base. * stays within one folder and ** crosses folders, so api/** matches every file under api. A change that touches no matching file does not apply: a blocking check is skipped, and an advisory check stays neutral. If GitHub lists only part of a large change and none of the listed files match, the check cannot be decided.

Configure coverage#

Choose Coverage as the rule, then choose the Report source:

  • API upload uses reports your CI uploads for the tested commit. Uploads require a repository-restricted token with coverage:write.
  • GitHub Actions artifact reads an artifact from one workflow. Enter its stable Workflow ID from GitHub and the exact Artifact name. GitHub's web pages show a workflow's name and file but not its ID: gh api repos/OWNER/REPO/actions/workflows lists each workflow's id and path.

Set Minimum changed-line coverage (%) and Wait for report (minutes). The minimum applies to executable lines changed by the pull request. Proofline reads your CI’s report; it does not run the tests.

Optionally set Maximum total coverage drop (points, optional) to compare total coverage with the merge base’s report. Leave it empty to skip that comparison. Enter exclusion globs, one per line, in Excluded paths (optional).

A missing, unreadable or unmappable report leaves coverage inconclusive. A total-drop comparison also needs a report for the merge base. The review’s Coverage section shows the reason and the tested commit when available.

Choose a mode#

An Advisory check reports whether its requirement is met. Its Check always concludes neutral, so it never blocks a merge.

A Blocking check succeeds when its rule is met and fails when it is not. It needs action when Proofline could not read what the rule needs, such as a title or label set that GitHub did not send, or a description it could not read at the reviewed commit. The Check names the cause. To block merges, add its Check as a required status check in the repository's GitHub rules. Proofline does not change GitHub rules.

A blocking coverage check succeeds when its requirements are met, fails when they are not, and needs action when coverage is inconclusive. It is skipped when there are no changed executable lines.

The separate Proofline check stays neutral in both modes.

When a change applies#

A check applies from the next reviewed head. Heads already reviewed keep the result they received. Editing a pull request title, description or labels without pushing a new commit does not run the check again.

See a check's last result#

The Checks list shows each check's last result in the repository: met, not met, not applicable, undecided or pending, with the pull request and commit. Open the check to read what its last Check said, such as why it could not be decided.

Edit, disable or reactivate a check#

Open the check from the list. Change the Rule, its parameters or Mode and select Save changes.

Select Disable check and confirm to stop new heads from reporting the check. If GitHub requires its Check, those pull requests cannot merge until you reactivate the check or remove the requirement. Select Reactivate check to report it again.

If somebody else saved the check after you opened it, the page says that the check changed and keeps your edits. Select Reload saved check to replace your edits with the saved version, then make your change again.

Search help

Tab to a result. Enter to open. Escape to close.