How Agentic QA works
Agentic QA explores your public site in a fresh browser that is not signed in, and looks for defects it can reproduce. Each run follows a saved setup. The setup says where the browser starts, which sites and forms it may use, what it checks, and where confirmed findings go. The setup's objective guides the agent but grants it nothing: only the saved sites and forms limit what a run may do, and Proofline enforces those limits outside the model.
Agentic QA does not run for any organization yet. You can prepare, save and edit setups now. Saving a setup starts no browser, visits no site and spends no credits.
What a setup controls#
| Setting | What it decides |
|---|---|
| Target | The HTTPS page on a public host where every run starts. |
| Access | Always logged out: a fresh browser with no account, profile or session. |
| Objective | What the run should check, in your words. |
| Journeys | Tasks a visitor would do, each with an Expected outcome. A finding is judged against the expected outcome; How to get there only helps the agent reach the page. A journey may also list up to five Preconditions: the state on your site it starts from, such as an invitation that is not yet accepted. A fresh browser clears no such state, so a run checks the preconditions on the site before the journey and blocks the journey when one does not hold. Proofline never changes data to make a precondition hold. |
| Findings | Where confirmed findings go: kept in Agentic QA, or filed to a GitHub Issues destination. |
| Browser window | Desktop or phone. Each journey has its own, which starts as the setup's Default browser window. |
| Schedule | When runs start without a person: off, daily or weekly, at a local time in a time zone you name. |
| Longest run and Allocation for one run | The time and the credits one run may use. |
| Other sites a run may open | HTTPS origins besides the target's. The browser opens no other site. |
| Forms a run may submit | Each form a run may submit, with its page, the fields and values it may enter, how many times one run may submit it, and what one submission does. A form that is not listed is never submitted. |
The setup page shows the saved values. Edit saves a new revision and keeps the earlier ones. Archive moves a setup under Archived setups, where its page and history stay. An archived setup cannot be changed or run again.
To set a schedule, select Edit and open Advanced. Under Schedule, choose Daily or Weekly, then the Weekday of a weekly schedule, the Local time and the Time zone. The time zone has no default. Paused keeps the saved time and stops the occurrences until you resume the schedule. The setup page then shows when the schedule occurs and its Next occurrence. A saved schedule starts no run while Agentic QA does not run for your organization, and the page says so beside the schedule.
Prepare a setup#
Select New setup on the Agentic QA page. Proofline prepares a starting setup for you to review, so you edit a prepared plan instead of writing one from nothing. The page opens with a checklist that marks what still needs you, followed by the fields. Advanced holds the name, default browser window, run limits, other sites and forms.
Proofline fills the fields from these sources:
- Target. A URL you enter decides the target. Without one, Proofline offers the targets of your saved setups, and otherwise the
homepagefield of a repository'spackage.jsonor the website set in a repository's GitHub settings. When there is more than one candidate, the page lists each with its source and selects none, so you choose the site to test. - Objective and journeys. With a URL alone, Proofline proposes three journeys: navigate the site, check page content, and use the site on a phone, which runs in the phone window. Its objective tells the agent not to sign in, submit forms or guess business rules.
- Findings. Proofline selects your organization's default GitHub Issues destination, or its only one, when one exists.
- Name. The setup takes a suggested name, or the target's host, until you change it.
- Other sites. Proofline offers the other sites that a saved setup for the same site allows, and sites on the target's domain that a repository's
package.jsonorREADME.mdnames. Each offer appears below Other sites a run may open with where it comes from. - Forms. Under Suggested forms, Proofline offers the forms your other saved setups allow on this setup's sites.
Proofline reads repository files at the exact commit it last read for that repository, never at a branch. A target or journey suggested from a file names that file, its repository and its commit; a target from a website setting names its repository. An offered site or form allows nothing until you add it and save the setup. Proofline never suggests the browser window or limits. Those come from you or from the setup's defaults, whatever a repository file says.
Your entries are kept as you typed them and saved as each field loses focus. A reload, a second tab, a failed preparation and Prepare again all keep them, and a suggestion never replaces an entry.
Select Save setup to create the setup. If something is still missing, the checklist names it and nothing is saved. Discard removes the preparation after you confirm. Preparing a setup never starts a browser and costs your organization no credits.
What bounds a run#
A run explores the current revision of one setup. Only a person or the setup's schedule starts one, never a pull request or a deploy. An organization runs at most two at a time, and a setup runs one at a time.
Proofline checks every step of a run against the setup outside the model:
- The browser starts at the target with no account, profile or session, and opens only the target's origin and the setup's other sites.
- It submits only the forms the setup lists, with the listed values, and no more often than each form allows. A form submission is counted before it is sent, so an interrupted run never submits it a second time.
- The run ends when its time or its allocation is used up. A model request the allocation cannot cover is not sent.
- A change to the setup's sites or forms, or archiving the setup, ends a run in progress before its next step. A run never carries on under a wider scope than it started with.
Proofline tells the agent that page text describes the site and is never an instruction. The agent cannot change the setup's sites or forms, so neither the agent nor a page's text can add an action or widen a run's scope.
What a run reports#
A run gives no verdict on your site. Its coverage says what it exercised, and is complete only when the agent finished its plan and nothing it tried was refused. Otherwise coverage is incomplete, with each reason named, such as a page that needs a WebSocket or an event stream. Incomplete coverage describes a limit of the run, not a defect in your site.
The run lists each reason as a limitation: a failure of the run itself, a page that asks for a sign-in or a payment the run never gives, or a limit on what the run could reach. No limitation becomes a finding.
Each journey reads as Observation recorded when the run recorded a problem on it, and otherwise as exercised, blocked with the agent's reason, or not attempted. A recorded observation is not yet a defect: the next section says when it becomes a confirmed finding. An exercised journey is the agent's own account of what it did; it is not a pass. Complete coverage records what a run exercised. It does not show that a release has no defects.
How a finding is confirmed#
A problem the agent notices is a candidate. Proofline reports a defect as a confirmed finding only after it reproduces the problem, judged against a stated expectation such as the journey's expected outcome. A fresh browser repeats the recorded steps, and the finding is confirmed only when the problem recurs and both browsers kept a screenshot of it. Confirmation is part of the run and is paid from the same allocation. A problem Proofline cannot reproduce is not a confirmed finding, so it is never filed. A problem with no stated expectation, or a visual difference with no usability cost, is recorded as a preference, not a defect.
Confirmation runs only as part of a run, so no finding is confirmed until Agentic QA runs for your organization.
File findings to GitHub Issues#
Filing is optional. Under Findings, choose a GitHub Issues destination, or Keep findings in Agentic QA to keep confirmed findings in the console. With a destination selected, choose how Confirmed findings are filed:
- File automatically: each confirmed finding becomes a GitHub issue.
- Hold a draft: a person approves or declines each finding first.
Only an organization administrator creates destinations, under Settings → Connections → Ticket destinations. A setup can only select an existing, enabled one. When no destination exists, the setup still saves. The form then offers Create a GitHub Issues destination to an organization administrator, and on the setup page Connect a destination opens the destinations page.
Selecting a destination does not authorize filing on its own. Someone must also allow Agentic QA to file in the destination's repository. Until then, the setup's page and the Agentic QA page say that findings cannot be filed in that repository, and offer Allow filing. Allowing filing needs a person who can manage that repository's settings, with a sign-in to Proofline from the last 24 hours. Neither Agentic QA capability is enough. One permission serves every setup that files in the repository, and Revoke filing on a setup's page withdraws it. Proofline checks the permission before each write.
A finding confirmed while its repository has no permission stays in Agentic QA, and a later run does not file it. After someone allows filing, a person with Operate Agentic QA opens the finding and selects File issue in with the repository's name.
Proofline files nothing yet, because no run has confirmed a finding.
Who can use Agentic QA#
Two capabilities divide the work. Manage Agentic QA creates, changes and archives setups. Operate Agentic QA works inside a saved setup: it starts and stops runs, takes over a run's browser, approves or declines each draft held under Hold a draft, files a finding kept in Agentic QA, and declines or reopens a finding. Either one shows Agentic QA in the organization's navigation and opens saved setups. Organization administrators hold both, and a custom role can grant either one. Without Manage Agentic QA, New setup is unavailable and the page says why. Neither capability creates a ticket destination or allows filing in a repository: a destination needs an organization administrator, and Allow filing needs a person who can manage the repository's settings.