Analytics exports

View as Markdown

Export reviews, findings, finding outcomes, validations and spend with proofline export, one page at a time, and follow the cursor to get a complete window.

Export one page#

Set PROOFLINE_TOKEN to a token with analytics:read and PROOFLINE_ORGANIZATION_ID to your organization's ID, then name a dataset and a window:

proofline export reviews --from 2026-10-01T00:00:00Z --to 2026-10-25T00:00:00Z --format csv
OptionValue
DATASETOne of the datasets below
--from TIMEThe inclusive RFC3339 start of the window
--to TIMEThe exclusive RFC3339 end of the window, at most 31 days after --from
--formatndjson, the default, or csv
--group-byThe grouping of finding_outcomes, and of no other dataset
--cursorThe continuation the previous page printed

The command writes one page to standard output. NDJSON holds one JSON object per line. CSV repeats its column header on every page. A page holds at most 8 MiB and 5,000 rows, or 1,000 groups for finding_outcomes.

Read every page#

One invocation reads one page. The command prints the settled horizon and, when another page follows, the next cursor to standard error. Run the command again with --cursor and the same dataset, window, format and grouping, until a page prints no cursor:

proofline export reviews --from 2026-10-01T00:00:00Z --to 2026-10-25T00:00:00Z --format csv --cursor "$CURSOR"

Treat the cursor as opaque. It belongs to one export, and reusing it with a different window or grouping is refused.

A failed page exits 1. Output collected from several invocations is complete only when the last page printed no cursor; partial output is never a complete export.

The first page fixes the settled horizon five minutes before it reads, and later pages exclude records newer than it. Pages are not a snapshot: a status or completion time can change between pages. Start another export to include later changes.

Datasets#

DatasetVersionWindow applies to
reviews1Review creation time
findings2Finding creation time
finding_validations2Validation creation time
finding_outcomes1First publication time
spend1Debit time

Each row names its dataset's version, and so does the export's file name.

finding_outcomes groups by repository, the default, environment, risk_domain, severity or subject_kind.

spend holds daily debit totals in microcredits. It needs your current administrator membership in the organization on GitHub, and a token restricted to repositories cannot read it.

A token restricted to repositories exports only the records of those repositories. Exporting findings also needs read access on GitHub to each repository the selected findings come from.

HTTP request#

The command sends GET /api/v1/organizations/{org_id}/exports/{dataset} with from, to and the optional format, group_by and cursor as query parameters. Export reviews in the API reference describes it.

A page is an export page only when the status is 200 and the response carries X-Proofline-Export-Version. Any other status returns a problem document, such as a 403 for a token that lacks access or a 429 with Retry-After: 60, and that body is not CSV. X-Proofline-Export-Settled-Until holds the settled horizon. X-Proofline-Export-Next-Cursor holds the cursor for the next page; read it from the headers of the same 200 response, and send it with the same dataset, window, format and grouping. The last page omits it.

A page can hold no rows. In CSV it is the column header alone, still a 200 with its version and horizon. The row count never ends an export: it is complete when a 200 page carries no cursor, and an empty first page that carries none is a complete export with no records.

This script reads every page of one window into reviews.csv. It keeps each page in a temporary directory beside the output, joins the pages without their repeated headers, and replaces reviews.csv only after the page without a cursor arrives. A refused page, a dropped connection or a truncated body stops it with a nonzero status, prints the problem document to standard error and leaves any earlier reviews.csv as it was:

#!/bin/sh
set -eu
dataset=reviews
from=2026-10-01T00:00:00Z
to=2026-10-25T00:00:00Z
out="$dataset.csv"
url="${PROOFLINE_BASE_URL:-https://proofline.sh}/api/v1/organizations/$PROOFLINE_ORGANIZATION_ID/exports/$dataset"

work=$(mktemp -d "./.$dataset-export.XXXXXX")
trap 'rm -rf "$work"' EXIT

header() {
  awk -v name="$1" '{ sub(/\r$/, ""); i = index($0, ":") }
    i && tolower(substr($0, 1, i - 1)) == tolower(name) { v = substr($0, i + 1); sub(/^[ \t]+/, "", v) }
    END { print v }' "$work/headers"
}

page=1
cursor=
while :; do
  set -- --data-urlencode "from=$from" --data-urlencode "to=$to" --data-urlencode format=csv
  if [ -n "$cursor" ]; then set -- "$@" --data-urlencode "cursor=$cursor"; fi
  status=$(curl -sS -G "$url" "$@" -H "Authorization: Bearer $PROOFLINE_TOKEN" \
    -D "$work/headers" -o "$work/page" -w '%{http_code}') || {
    echo "page $page: the transfer failed" >&2
    exit 1
  }
  if [ "$status" != 200 ] || [ -z "$(header X-Proofline-Export-Version)" ]; then
    echo "page $page: HTTP $status" >&2
    cat "$work/page" >&2
    echo >&2
    exit 1
  fi
  if [ "$page" = 1 ]; then
    echo "settled until $(header X-Proofline-Export-Settled-Until)" >&2
    cp "$work/page" "$work/export"
  else
    tail -n +2 "$work/page" >> "$work/export"
  fi
  cursor=$(header X-Proofline-Export-Next-Cursor)
  [ -n "$cursor" ] || break
  page=$((page + 1))
done
mv -f "$work/export" "$out"
echo "$out: complete, $page page(s)" >&2

Change dataset, from and to for another export, and add --data-urlencode group_by=environment to the first set -- line to group finding_outcomes. The cursor belongs to that one export, so running the script again starts a new export with a new horizon. Set PROOFLINE_BASE_URL for a console other than proofline.sh.

SDKs#

The SDKs read the same pages. TypeScript's exportReviews and Python's export_reviews read one page and return it with these fields:

FieldValue
rowsThe page's records as objects. A CSV page's header becomes their keys and is not a row, and CSV values stay strings.
textThe page as the server sent it. Every CSV page starts with the column header.
versionThe dataset version from X-Proofline-Export-Version.
settledUntil, settled_until in PythonThe settled horizon, which the first page fixes for the export.
nextCursor, next_cursor in PythonThe cursor for the next page, or null on the last page.

Join rows across pages, or keep only the first page's header when you join CSV text. A page can hold no rows and still carry a cursor, so keep reading until nextCursor or next_cursor is null.

These examples read every page of the window that the script above reads, and write reviews.ndjson only after the last page arrives. Each page uses the same dataset, window, format and grouping, and only the cursor changes.

TypeScript reads one page per call, so the loop follows the cursor itself. It stops at 100 pages and refuses a cursor that repeats:

import { writeFileSync } from "node:fs";
import { ProoflineClient } from "@proofline/sdk";

const client = new ProoflineClient({
  baseUrl: "https://proofline.sh",
  token: process.env.PROOFLINE_TOKEN!,
});
const path = { org_id: process.env.PROOFLINE_ORGANIZATION_ID!, dataset: "reviews" as const };
const filters = { from: "2026-10-01T00:00:00Z", to: "2026-10-25T00:00:00Z", format: "ndjson" as const };
const rows: Record<string, unknown>[] = [];
const seen = new Set([] as string[]);
let cursor: string | undefined;
let complete = false;
for (let page = 1; page <= 100 && !complete; page += 1) {
  const result = await client.exportReviews({ path, query: { ...filters, cursor } });
  if (page === 1) console.error(`version ${result.version}, settled until ${result.settledUntil}`);
  rows.push(...result.rows);
  const next = result.nextCursor;
  if (next === null) complete = true;
  else if (!next || seen.has(next)) throw new Error(`page ${page} repeats a cursor`);
  else {
    seen.add(next);
    cursor = next;
  }
}
if (!complete) throw new Error("the export has more than 100 pages");
writeFileSync("reviews.ndjson", rows.map((row) => `${JSON.stringify(row)}\n`).join(""));

Python's iter_export_pages follows the cursors and yields each page. It reads at most max_pages pages, 100 unless you set another bound up to 1,000. It raises ProoflineError after yielding earlier pages when a page is refused, a cursor repeats, or the bound runs out while a cursor remains. So collect the rows first and write the file after the loop ends:

import json
import os
import sys
from uuid import UUID
from proofline import ProoflineClient

rows = []
with ProoflineClient(os.environ["PROOFLINE_TOKEN"], base_url="https://proofline.sh") as client:
    pages = client.iter_export_pages(
        UUID(os.environ["PROOFLINE_ORGANIZATION_ID"]),
        "reviews",
        from_time="2026-10-01T00:00:00Z",
        to_time="2026-10-25T00:00:00Z",
        format="ndjson",
        max_pages=100,
    )
    for number, page in enumerate(pages, start=1):
        if number == 1:
            print(f"version {page.version}, settled until {page.settled_until}", file=sys.stderr)
        rows.extend(page.rows)
with open("reviews.ndjson", "w", encoding="utf-8") as out:
    out.writelines(json.dumps(row) + "\n" for row in rows)

Both SDKs raise ProoflineError for a refused page or a failed transfer, with the problem's status, type and detail. Python's iterator also raises it for a repeated cursor, with the type urn:proofline:sdk:invalid-pagination, and when its bound runs out, with urn:proofline:sdk:pagination-limit; the TypeScript loop throws its own errors for those two. In every case the example stops before it writes the file, and the rows it read are not a complete export. Running it again starts a new export with a new horizon.