From d2e940f4c1a24bc39485ba7dac1227dd9ecee5b0 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 12 Sep 2026 11:33:20 -0700 Subject: [PATCH] ci: run release Runner protocol and Rust checks in parallel (#13326) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud deploys verified images from merged source commits. > - Cloud readiness waits for every release verification check. > - Runner verification currently runs long TypeScript tests before Rust checks. > - These checks can run on independent runners with their own build directories. > - This PR runs them in parallel while preserving all checks and the shared dependency cache. ## Linked Issues or Issue Description Refs #13194. Related prior work: #13142 and #13259. A search found no duplicate parallel release-check change. **What existing behavior does this improve?** Time from merge to Cloud source verification and deployment readiness. **Current behavior** Recent successful runs take roughly 13 minutes from merge to deployable. In run 34705914878, Runner verification took 11m23s. Protocol tests finished before Rust tests and API authority checks started. **Proposed behavior** Run protocol and Rust verification in two matrix jobs. Cloud readiness still requires both jobs to pass. **Reason and benefit** Remove the serial dependency between independent checks. Expected improvement is about 2–3 minutes on a typical cached run, until the image build or server tests become the longest job. This is an estimate; post-merge timing will confirm it. **Breaking changes** Individual release Runner job names gain a lane suffix. Cloud source and readiness marker names stay the same. PR runner routing is unchanged. ## What Changed - Split release Runner checks into protocol and Rust lanes. Keep every constituent of `check:all` exactly once. - Restore the existing Rust dependency cache in both lanes. Allow only the Rust lane to save it after warming both build profiles. - Add coverage and cache authorization regressions. Document the parallel verification and single cache writer. ## Verification - Passed 477 workflow and source-verification tests with `node --test .github/scripts/tests/*.test.mjs scripts/cloud-source-verification.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Passed `actionlint`, `git diff --check`, and the private AWS routing regression suite. - Passed local `pnpm -r typecheck` and the standalone `check:runner && check:api-authority` lane, including all 1,671 API tests before the protocol lane had built TypeScript output. - The broad local protocol run under Node 25 had four failures. The two affected files passed under CI's Node 24.19.0: 67 passed, 6 platform skips. - Local `pnpm test:run` aborted when disk space ran out; local `pnpm build` could not run afterward. These are local verification limits. [Linux CI run 34710421424](https://github.com/paperclipai/paperclip/actions/runs/34710421424) passed full typecheck, all grouped tests, native verification, build, release dry run, and browser checks. Native protocol CI passed 1,986 tests, plus 1,671 API tests and the Rust suites. - Latest-head Greptile is 5/5 with no open findings. All 33 current-head checks are successful or intentionally skipped. ## Risks - Uses one additional short-lived verification runner per release verification. The existing AWS exact-master restriction remains in place. - The Rust lane warms debug dependencies so its cache save also serves protocol tests. Both lanes always rebuild workspace code. - A workflow regression could omit a check. The new coverage test compares the matrix checks directly with `check:all`; Cloud readiness depends on the complete reusable workflow. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../tests/release-runner-cache.test.mjs | 47 ++++++++++++++++--- .github/workflows/release-verify.yml | 31 ++++++++++-- doc/RELEASE-AUTOMATION-SETUP.md | 32 ++++++++----- .../release-verify-workflow.test.mjs | 10 ++-- 4 files changed, 92 insertions(+), 28 deletions(-) diff --git a/.github/scripts/tests/release-runner-cache.test.mjs b/.github/scripts/tests/release-runner-cache.test.mjs index 9e4b8fe715..7e35ec09c9 100644 --- a/.github/scripts/tests/release-runner-cache.test.mjs +++ b/.github/scripts/tests/release-runner-cache.test.mjs @@ -1,6 +1,7 @@ import test from "node:test"; import assert from "node:assert/strict"; import { readFileSync } from "node:fs"; +import { runInNewContext } from "node:vm"; const workflow = readFileSync(new URL("../../workflows/release-verify.yml", import.meta.url), "utf8"); const runner = workflow.split(" verify_paperclip_runner:")[1].split(" build:")[0]; @@ -22,16 +23,48 @@ test("the shared cache excludes workspace artifacts and only restores or saves t assert.match(runner, /cache-workspace-crates: false/); assert.match(runner, /cache-bin: false/); const saveIf = runner.match(/^\s*save-if: (.+)$/m)?.[1]; - assert.equal(saveIf, "${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}"); + assert.equal(saveIf, "${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}"); const cacheStep = runner.split(" - name: Cache Runner Rust dependencies")[1].split(" - name: Install dependencies")[0]; - assert.equal(cacheStep.match(/^\s*if: (.+)$/m)?.[1], saveIf); + assert.equal(cacheStep.match(/^\s*if: (.+)$/m)?.[1], saveIf.replace("matrix.lane == 'rust' && ", "")); assert.doesNotMatch(runner, /cache-on-failure: true|cache-all-crates: true/); }); -test("cache hits cannot bypass Runner verification", () => { - const verify = runner.split(" - name: Verify Paperclip Runner")[1]; - assert.match(verify, /run: pnpm --filter @paperclipai\/paperclip-runner check:all/); - assert.doesNotMatch(verify, /if:|continue-on-error:/); - assert.ok(runner.indexOf("Cache Runner Rust dependencies") < runner.indexOf(" - name: Verify Paperclip Runner\n")); +test("parallel lanes cover check:all exactly once and never bypass verification", () => { + const scripts = JSON.parse(readFileSync(new URL("../../../packages/paperclip-runner/package.json", import.meta.url))).scripts; + const checks = [...runner.matchAll(/^ checks: (.+)$/gm)].flatMap(([, value]) => value.split(" ")); + assert.deepEqual(checks, scripts["check:all"].split(" && ").map((command) => command.replace(/^pnpm run /, ""))); + assert.deepEqual([...runner.matchAll(/^ - lane: (.+)$/gm)].map(([, value]) => value), ["protocol", "rust"]); + assert.match(runner, /fail-fast: false/); + assert.doesNotMatch(runner, /max-parallel: 1|^ needs:|continue-on-error:/m); + const verify = runner.split(" - name: Verify Paperclip Runner\n")[1].split(" - name: Warm debug")[0]; + assert.match(verify, /RUNNER_CHECKS: \$\{\{ matrix.checks \}\}/); + assert.match(verify, /set -euo pipefail/); + assert.match(verify, /for check in \$RUNNER_CHECKS; do\s+pnpm --filter @paperclipai\/paperclip-runner "\$check"\s+done/); + assert.doesNotMatch(verify, /if:|cache-hit/); assert.doesNotMatch(runner, /id-token: write|packages: write|secrets: inherit/); }); + +test("only the trusted Rust lane writes, and warms both build profiles before saving", () => { + const cache = runner.split(" - name: Cache Runner Rust dependencies")[1].split(" - name: Install dependencies")[0]; + const warm = runner.split(" - name: Warm debug dependencies for the shared Runner cache")[1]; + const expr = (body, field) => body.match(new RegExp(`^ +${field}: \\$\\{\\{ (.+) \\}\\}$`, "m"))[1]; + assert.equal(expr(cache, "save-if"), expr(warm, "if")); + assert.match(warm, /run: pnpm --filter @paperclipai\/paperclip-runner build:rust/); + const sha = "a".repeat(40); + const base = { repository: "paperclipai/paperclip", event_name: "push", ref: "refs/heads/master", sha }; + for (const lane of ["protocol", "rust"]) { + for (const [overrides, ref, trusted] of [ + [{}, sha, true], + [{ event_name: "pull_request", ref: "refs/pull/1/merge" }, sha, false], + [{ event_name: "pull_request_target" }, sha, false], + [{ event_name: "workflow_dispatch" }, sha, false], + [{ repository: "someone/paperclip" }, sha, false], + [{ ref: "refs/heads/feature" }, sha, false], + [{}, "b".repeat(40), false], + ]) { + const context = { matrix: { lane }, github: { ...base, ...overrides }, inputs: { ref } }; + assert.equal(runInNewContext(expr(cache, "if"), context), trusted); + assert.equal(runInNewContext(expr(cache, "save-if"), context), trusted && lane === "rust"); + } + } +}); diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 442d198f20..505624a6b0 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -262,12 +262,21 @@ jobs: run: pnpm test:runner-workflow-evals verify_paperclip_runner: - name: Verify Paperclip Runner + name: Verify Paperclip Runner (${{ matrix.lane }}) runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }} timeout-minutes: 20 permissions: contents: read + strategy: + fail-fast: false + matrix: + include: + - lane: protocol + checks: check:eval-kernel check:protocol + - lane: rust + checks: check:runner check:api-authority + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 @@ -305,15 +314,27 @@ jobs: # dependencies; never restore installed executables from cargo/bin. cache-workspace-crates: false cache-bin: false - # The step guard also restricts restores. Save only after a successful - # master-push verification of that push's exact commit. - save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} + # Both lanes restore the existing dependency cache. Only the Rust + # lane saves it, after warming both release and debug dependencies. + save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} - name: Install dependencies run: pnpm install --no-frozen-lockfile - name: Verify Paperclip Runner - run: pnpm --filter @paperclipai/paperclip-runner check:all + env: + RUNNER_CHECKS: ${{ matrix.checks }} + run: | + set -euo pipefail + for check in $RUNNER_CHECKS; do + pnpm --filter @paperclipai/paperclip-runner "$check" + done + + - name: Warm debug dependencies for the shared Runner cache + # Protocol tests need debug binaries. Populate their dependencies in + # the sole cache writer, so a cold save also serves the protocol lane. + if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} + run: pnpm --filter @paperclipai/paperclip-runner build:rust build: name: Build diff --git a/doc/RELEASE-AUTOMATION-SETUP.md b/doc/RELEASE-AUTOMATION-SETUP.md index 43b6fee7bd..4817e5d7fc 100644 --- a/doc/RELEASE-AUTOMATION-SETUP.md +++ b/doc/RELEASE-AUTOMATION-SETUP.md @@ -338,19 +338,27 @@ Check: ## Runner verification dependency cache -`release-verify.yml` caches Cargo dependencies for its `Verify Paperclip Runner` -job using a pinned Rust Cache action. It selects the compiler from the Runner -package's `rust-toolchain.toml` before computing the cache key. Compiler and Cargo -metadata changes select a new cache; the `release-runner-v1` shared key lets -callers of this reusable verification workflow reuse the same dependency cache. +`release-verify.yml` runs `Verify Paperclip Runner` on two independent runners. +The protocol lane runs `check:eval-kernel` and `check:protocol`. The Rust lane +runs `check:runner` and `check:api-authority`. Together they retain every check +in `check:all`; both lanes must pass before Cloud source verification or +readiness can succeed. A failed lane does not cancel the other lane. -Workspace crates and installed Cargo binaries are excluded. Every run still -builds the Runner workspace and runs `check:all`, including the Rust and -TypeScript tests. Only an own-repository master-push run verifying that push's exact -SHA can restore the cache, and only a successful run saves it. PR, tag, and -manual candidate verification compile without this cache. A miss or eviction costs compilation time but does not change the checks. -To discard old dependency caches, increment the shared-key version and let the -next successful master verification warm it again. +Both lanes restore Cargo dependencies with the pinned Rust Cache action. The +compiler comes from the Runner package's `rust-toolchain.toml` before the action +computes its key. Compiler and Cargo metadata changes select a new cache. The +existing `release-runner-v1` shared key avoids separate copies for these lanes. +Only the Rust lane saves this cache. After verification it also runs `build:rust` +to warm the debug dependencies used by the protocol lane; its own tests already +warm release dependencies. The cache writer is shorter than the protocol lane. + +Workspace crates and installed Cargo binaries are excluded. Every run rebuilds +workspace code and runs all assigned checks, including on a cache hit. Only an +own-repository master-push run verifying that push's exact SHA can restore the +cache, and only a successful Rust lane saves it. PR, tag, and manual candidate +verification compile without this cache. A miss or eviction costs compilation +time but does not change the checks. To discard old dependency caches, increment +the shared-key version and let the next successful master verification warm it. The trust boundary is the protected master branch, not the cache-key text. GitHub does not let master restore caches created by a child branch, sibling diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 702edcab8f..d93b4bf5da 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -221,10 +221,12 @@ test("release verify workflow covers the same split test surface as stable PR ve ); assert.match(verifyWorkflow, /pnpm -r typecheck/); assert.match(verifyWorkflow, /pnpm build/); - assert.match( - verifyWorkflow, - /pnpm --filter @paperclipai\/paperclip-runner check:all/, - ); + const runnerScripts = JSON.parse(readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8")).scripts; + const runnerChecks = [...verifyWorkflow.matchAll(/^ checks: (.+)$/gm)] + .flatMap(([, checks]) => checks.split(" ")); + assert.deepEqual(runnerChecks, runnerScripts["check:all"].split(" && ") + .map((command) => command.replace(/^pnpm run /, ""))); + assert.match(verifyWorkflow, /pnpm --filter @paperclipai\/paperclip-runner "\$check"/); assert.match(verifyWorkflow, /runner_workflow_evals:/); assert.match(verifyWorkflow, /runner_chaos_evals:/); assert.match(