All posts

From 18 minutes to 8: two days of making our CI fast and cheap

In two days, the median time to check a merge group on Proofline's main branch fell from 18.4 minutes to 8.0, while the queue handled two to four times the load. The machines running those checks became less than half as expensive per hour. Discarded computation, which consumed half the merge queue's job time in the worst hour we measured, is mostly gone too. Those are three separate gains: shorter checks, cheaper machine-hours and less wasted work.

Why CI speed sets our pace#

Most of Proofline's code is written by coding agents, and all of it reaches main through GitHub's merge queue. The queue runs around the clock. Between midnight and 06:00 UTC on 16 November, 319 pull requests merged, more than fifty an hour. At that rate, every minute a merge group spends in CI is a minute that dozens of changes wait behind it.

Each change passes two checks. A pull request runs a Fast check: formatting, lints, Clippy (Rust's linting tool) and type checks. A merge group is the queue's trial merge of the change onto main with everything ahead of it in the queue. That group runs the full gate: source checks, a build of the Rust test suite, the suite itself split across parallel partitions, and the web and marketing gates. Building the tests and running them are separate jobs.

All of this runs on short-lived Google Cloud Spot VMs, one VM per job. Our controller, a small service on Cloud Run, reads GitHub's queue, boots or hands out a VM for each queued job, and deletes it when the job ends. That gives us a shared control point for startup time, execution, idle capacity and machine prices.

Where the time went#

We measured before we changed anything, using GitHub's job timings and the controller's own logs. Here are the figures from 14 November alongside the same measures after the changes below. Some are medians, some are ranges, and the Fast-check baseline is a count above a threshold.

  • Merge group on main, median: 18.4 min before; 8.0 min after.
  • Ready for review to merged, median: 27.7 min before; 17.5 min after.
  • Test build job: 8.8 min before; 237 to 258 s after.
  • Source checks job: 598 to 790 s before; 232 to 236 s after.
  • A test partition: 182 to 198 s each across three partitions before; 116 to 139 s each across five after.
  • Pull request Fast check with Rust: over 600 s in 37 of 100 checks before; a 295 s median after.
  • Check VM, created to ready: fast VMs took 113 s before and 47 s after; remote VMs took 136 to 155 s before and 59 to 61 s after.
  • A Rust change, merged to reaching main: 340 min via the Rust lane before; now merges into main directly.

The ready-to-merged figure is from 15 November; every other "after" figure is from 16 November.

Two days, in one line#

Median merge group time on main per three-hour block, from 19.4 minutes on the morning of November 14 to 8.0 minutes from November 16 onward, with the 75th percentile falling from 25 minutes to under 9.

Passed merge groups on main, from GitHub Actions run times, 06:00 UTC on November 14 to 18:00 UTC on November 16. Markers: 1, check VMs move to Belgium; 2, compile cache for our crates; 3, five partitions and 16 Cargo jobs; 4, the seed restored in a minute.

The median held near 19 minutes through 14 November and fell in steps on the 15th and 16th as each change landed. Since 06:00 UTC on 16 November it has stayed at 8.0 to 8.1 minutes, while the queue handled 92 to 117 groups every three hours, two to four times the load of the 14th. The slowest quarter of groups now finishes in under 9 minutes, compared with 19 to 25 before.

Machines that are ready in a minute#

Every check VM runs its jobs inside our CI image, which we call the seed. A new VM used to pull that image from Artifact Registry before it could take a job. The pull dominated startup: a fast VM was ready 113 s after Compute created it, and a remote VM took 136 to 155 s.

Now a build writes each seed once as a compressed ext4 file system containing the Docker store exactly as a pull leaves it. A booting VM streams that file onto its local SSD at the disk's speed. It mounts the file only if its bytes hash to the SHA-256 carried by its instance template; any failure falls back to the old pull. The fast seed's file is 1.4 GB.

  • Fast VMs, since 06:47 UTC on 16 November: ready in 47 s instead of 113 s.
  • Remote VMs, the pool used by merge groups, since 07:52 UTC: ready in 59 to 61 s instead of 136 to 155 s. All 160 remote VMs in the first ten minutes restored the seed, and none fell back to the pull.

Faster boots also help when the queue rebuilds. When a group fails, GitHub rebuilds every group behind it at once, and each new group asks for workers. Across 13 such rebuilds that morning, median waits for the first jobs ranged from 17 to 172 s. In the first two rebuilds after the change, they waited 19 and 27 s.

Quicker startup also let us keep fewer VMs idling in reserve. We cut these warm pools from 12 fast and 10 remote VMs to 8 of each. In the hour after, merge-group times were much the same, a median of 8.3 against 8.1 minutes, while the queue handled half again as many groups.

Compiling less#

Once a worker was ready, the longest stage of every merge group was building the test suite. The control plane is 1.1 million lines of Rust in 47 crates, Rust's compilation units, plus 414,000 lines of integration tests. Third-party dependencies already came from a prebuilt snapshot, so nearly all of the remaining build time went to our own crates, compiled one after another along a chain of large ones.

A cache for our own crates. Each check VM starts from an empty build directory, so every merge group compiled all of our crates again, even ones its change never touched. Now merge-group test builds write the libraries they compile to a shared store, keyed on every input. That includes the query files our SQL macros read behind the compiler's back, which we hash into the key. Later builds reuse any library whose key matches.

Only check VMs can read the store; nothing that gates a release ever does. Compilation in the test build fell from about 300 s to 223 s at the median, and to 162 to 215 s when every library was a hit.

More parallel work. The test build now runs 16 jobs through Cargo, Rust's build tool, on a 16-processor VM. The suite runs in five partitions instead of three, and each finishes in 116 to 139 s instead of 182 to 198 s. Pull requests' Fast checks moved to 16 processors as well. The built tests now travel to the partitions as about 40 content-addressed parts, uploaded and downloaded in parallel. A partition fetches its copy in about 4 s.

Clippy from the same cache. Clippy re-checked every crate on every pull request, making it the slowest part of a Fast check with Rust in it. It now reads the same store. In a check-VM example with a change to one crate, Clippy fell from 190 s to 100 s. In its first hour on real merge groups, Clippy's median fell from 241 s to 190 s; we are still measuring the gain on pull requests. Getting the cache working took one surprise: it recognises Clippy's compiler argument only when it is the bare word rustc, while Cargo passes a full path.

One thing we measured and left alone: tuning the throwaway PostgreSQL the tests run against. No setting saved more than 2.4% of a partition.

Waste we stopped paying for#

Shorter successful checks did not tell us how much computation the queue discarded. Between 08:27 and 09:27 UTC on 16 November, 41 of the 89 merge groups were cancelled before they finished. Their jobs had used 807 of the hour's 1,639 job-minutes, nearly half. Some loss is built in: when a change leaves the queue, GitHub rebuilds every group behind it. The rest was our controller's.

GitHub gives each rebuilt entry a new branch but can leave the old group's branch in place for minutes, sometimes days. Our controller treated a group as dropped only once its branch disappeared. Old groups kept running beside their replacements for a median of 306 s each.

Our first fix did nothing. It picked the newer of an entry's two groups by commit date, but GitHub gives a rebuilt group's commit the same date as the one it replaces. The second fix orders groups by when their runs were created. At the next rebuild that left old branches in place, the old groups stopped 49 to 53 s after their replacements started, against 306 to 330 s before. In the hours since, cancelled groups have used 5 to 9% of merge-group job time instead of 49%, though those hours also had fewer rebuilds.

Partitions that wait for their build. A merge group's test partitions can only start once its test build uploads the tests. The controller now learns how long each queue's builds usually take and boots a VM for each partition 2.25 minutes before the build is expected to end. Since 17:39 UTC on 16 November, it also registers the partitions' runners on those VMs 75 s before that expected finish, so each runner is already listening when GitHub queues its job. The wait after the build fell from a median of 30 s to 2 s. In the first hour, merge groups with a test build took a median of 429 s, against 484 s that morning.

A detour for Rust changes. Beyond job runtimes, Rust changes had a separate delivery delay. They passed through a separate merge lane whose promotion into main conflicted with main seven times on 15 November. A Rust change took about 340 minutes to reach main that way. The lane now takes no new changes, and Rust merges into main like everything else.

Cheaper hours#

Frankfurt to Belgium. Alongside reducing work and waiting, we lowered the listed price of the machines. Our check VMs ran in Frankfurt, where the same Spot machines list at about twice Belgium's price. A 16-core C2D VM with its SSD lists at €0.266 an hour against €0.131; a C3D at €0.487 against €0.237. We moved the fleet at 21:35 UTC on 15 November. Five minutes later, the running VMs priced at €16.1 an hour; the same VMs in Frankfurt would have cost €33.3.

Each region got its own registry and caches because a remote VM reads about 38 GiB an hour from storage, and reading across European regions costs €0.018 per GiB.

The cheapest family first. The controller tries machine families in order of what a job costs on them. Each family's local SSD quota caps how many VMs it can run. C2D, the cheapest, was capped at 33 VMs, so most remote VMs overflowed to C3D at twice the price. At 09:33 UTC on 16 November the fleet held 55 C3D VMs and 27 C2D. We asked Google for four times the C2D quota, and it was granted the same day. At 15:40 UTC all 54 remote VMs were C2D, at about €0.12 an hour each instead of €0.24.

Why we stayed in Europe. We priced a 16-core Spot VM with its SSD in every Google Cloud region from the billing catalog. Several regions list far lower prices: C3D in Berlin at €0.077 an hour, against Belgium's €0.244. But Berlin offers neither C3D nor C2D. Among regions that offer each machine in at least two zones, Belgium is the second-cheapest for both C2D and C3D. For C2D, which now runs all of our remote VMs, the one cheaper region is Delhi, at about 10% less. A price list is not a supply.

The fleet has caps on concurrent VMs per kind. We also have a €25,000 monthly budget with alerts.

What broke on the way#

Moving this fast on a queue that never stops had costs.

  • main was red for four and a half hours, from 23:00 to 03:34 UTC. A lockfile update moved a certificate-roots crate past its reviewed version, and only main's own CI checked that. We pinned it back, and the check now has to run before merge too.
  • Two builds raced on one upload. Splitting the test archive into parts let two builds upload the same part at once. 10 of 64 test builds failed, and the queue removed changes that had done nothing wrong. A fix that accepts an identical part already in place went to the front of the queue and landed at 02:37 UTC.
  • One SSD too few. Our first 16-processor template for Fast checks asked for one local SSD, and that machine type will not take just one. For 39 minutes, pull requests' checks ran on a dearer 8-processor fallback until we changed the template to ask for two.
  • A fix that fixed nothing. As described above, our first attempt at stopping dropped groups relied on commit dates that GitHub does not change.

What we learned#

  • Medians hide waste. The merge-group median looked healthy all morning on 16 November while half of the queue's job time went to groups that were thrown away. We found that by adding up job-minutes by how each group ended, rather than timing only the groups that passed.
  • Measure the hour after, every time. Each change went live with its own targeted apply, and we compared the hour after with the hour before. That caught the fix that fixed nothing: the next hour's numbers disproved its assumption about GitHub.
  • A price list is not a supply. The cheapest regions on paper were cheap for machines they do not offer. Prices only mean something alongside where the machines exist.
  • The bill arrives late. Google's billing export runs about ten hours behind. We steered by list prices and the live fleet's makeup. The bill still has to confirm the savings per check and per merged pull request.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading