ci: run release Runner protocol and Rust checks in parallel (#13326)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud deploys verified images from merged source commits. > - Cloud readiness waits for every release verification check. > - Runner verification currently runs long TypeScript tests before Rust checks. > - These checks can run on independent runners with their own build directories. > - This PR runs them in parallel while preserving all checks and the shared dependency cache. ## Linked Issues or Issue Description Refs #13194. Related prior work: #13142 and #13259. A search found no duplicate parallel release-check change. **What existing behavior does this improve?** Time from merge to Cloud source verification and deployment readiness. **Current behavior** Recent successful runs take roughly 13 minutes from merge to deployable. In run 34705914878, Runner verification took 11m23s. Protocol tests finished before Rust tests and API authority checks started. **Proposed behavior** Run protocol and Rust verification in two matrix jobs. Cloud readiness still requires both jobs to pass. **Reason and benefit** Remove the serial dependency between independent checks. Expected improvement is about 2–3 minutes on a typical cached run, until the image build or server tests become the longest job. This is an estimate; post-merge timing will confirm it. **Breaking changes** Individual release Runner job names gain a lane suffix. Cloud source and readiness marker names stay the same. PR runner routing is unchanged. ## What Changed - Split release Runner checks into protocol and Rust lanes. Keep every constituent of `check:all` exactly once. - Restore the existing Rust dependency cache in both lanes. Allow only the Rust lane to save it after warming both build profiles. - Add coverage and cache authorization regressions. Document the parallel verification and single cache writer. ## Verification - Passed 477 workflow and source-verification tests with `node --test .github/scripts/tests/*.test.mjs scripts/cloud-source-verification.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Passed `actionlint`, `git diff --check`, and the private AWS routing regression suite. - Passed local `pnpm -r typecheck` and the standalone `check:runner && check:api-authority` lane, including all 1,671 API tests before the protocol lane had built TypeScript output. - The broad local protocol run under Node 25 had four failures. The two affected files passed under CI's Node 24.19.0: 67 passed, 6 platform skips. - Local `pnpm test:run` aborted when disk space ran out; local `pnpm build` could not run afterward. These are local verification limits. [Linux CI run 34710421424](https://github.com/paperclipai/paperclip/actions/runs/34710421424) passed full typecheck, all grouped tests, native verification, build, release dry run, and browser checks. Native protocol CI passed 1,986 tests, plus 1,671 API tests and the Rust suites. - Latest-head Greptile is 5/5 with no open findings. All 33 current-head checks are successful or intentionally skipped. ## Risks - Uses one additional short-lived verification runner per release verification. The existing AWS exact-master restriction remains in place. - The Rust lane warms debug dependencies so its cache save also serves protocol tests. Both lanes always rebuild workspace code. - A workflow regression could omit a check. The new coverage test compares the matrix checks directly with `check:all`; Cloud readiness depends on the complete reusable workflow. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
GitHub
parent
a59f5a8adc
commit
d2e940f4c1
|
|
@ -1,6 +1,7 @@
|
|||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { runInNewContext } from "node:vm";
|
||||
|
||||
const workflow = readFileSync(new URL("../../workflows/release-verify.yml", import.meta.url), "utf8");
|
||||
const runner = workflow.split(" verify_paperclip_runner:")[1].split(" build:")[0];
|
||||
|
|
@ -22,16 +23,48 @@ test("the shared cache excludes workspace artifacts and only restores or saves t
|
|||
assert.match(runner, /cache-workspace-crates: false/);
|
||||
assert.match(runner, /cache-bin: false/);
|
||||
const saveIf = runner.match(/^\s*save-if: (.+)$/m)?.[1];
|
||||
assert.equal(saveIf, "${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}");
|
||||
assert.equal(saveIf, "${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}");
|
||||
const cacheStep = runner.split(" - name: Cache Runner Rust dependencies")[1].split(" - name: Install dependencies")[0];
|
||||
assert.equal(cacheStep.match(/^\s*if: (.+)$/m)?.[1], saveIf);
|
||||
assert.equal(cacheStep.match(/^\s*if: (.+)$/m)?.[1], saveIf.replace("matrix.lane == 'rust' && ", ""));
|
||||
assert.doesNotMatch(runner, /cache-on-failure: true|cache-all-crates: true/);
|
||||
});
|
||||
|
||||
test("cache hits cannot bypass Runner verification", () => {
|
||||
const verify = runner.split(" - name: Verify Paperclip Runner")[1];
|
||||
assert.match(verify, /run: pnpm --filter @paperclipai\/paperclip-runner check:all/);
|
||||
assert.doesNotMatch(verify, /if:|continue-on-error:/);
|
||||
assert.ok(runner.indexOf("Cache Runner Rust dependencies") < runner.indexOf(" - name: Verify Paperclip Runner\n"));
|
||||
test("parallel lanes cover check:all exactly once and never bypass verification", () => {
|
||||
const scripts = JSON.parse(readFileSync(new URL("../../../packages/paperclip-runner/package.json", import.meta.url))).scripts;
|
||||
const checks = [...runner.matchAll(/^ checks: (.+)$/gm)].flatMap(([, value]) => value.split(" "));
|
||||
assert.deepEqual(checks, scripts["check:all"].split(" && ").map((command) => command.replace(/^pnpm run /, "")));
|
||||
assert.deepEqual([...runner.matchAll(/^ - lane: (.+)$/gm)].map(([, value]) => value), ["protocol", "rust"]);
|
||||
assert.match(runner, /fail-fast: false/);
|
||||
assert.doesNotMatch(runner, /max-parallel: 1|^ needs:|continue-on-error:/m);
|
||||
const verify = runner.split(" - name: Verify Paperclip Runner\n")[1].split(" - name: Warm debug")[0];
|
||||
assert.match(verify, /RUNNER_CHECKS: \$\{\{ matrix.checks \}\}/);
|
||||
assert.match(verify, /set -euo pipefail/);
|
||||
assert.match(verify, /for check in \$RUNNER_CHECKS; do\s+pnpm --filter @paperclipai\/paperclip-runner "\$check"\s+done/);
|
||||
assert.doesNotMatch(verify, /if:|cache-hit/);
|
||||
assert.doesNotMatch(runner, /id-token: write|packages: write|secrets: inherit/);
|
||||
});
|
||||
|
||||
test("only the trusted Rust lane writes, and warms both build profiles before saving", () => {
|
||||
const cache = runner.split(" - name: Cache Runner Rust dependencies")[1].split(" - name: Install dependencies")[0];
|
||||
const warm = runner.split(" - name: Warm debug dependencies for the shared Runner cache")[1];
|
||||
const expr = (body, field) => body.match(new RegExp(`^ +${field}: \\$\\{\\{ (.+) \\}\\}$`, "m"))[1];
|
||||
assert.equal(expr(cache, "save-if"), expr(warm, "if"));
|
||||
assert.match(warm, /run: pnpm --filter @paperclipai\/paperclip-runner build:rust/);
|
||||
const sha = "a".repeat(40);
|
||||
const base = { repository: "paperclipai/paperclip", event_name: "push", ref: "refs/heads/master", sha };
|
||||
for (const lane of ["protocol", "rust"]) {
|
||||
for (const [overrides, ref, trusted] of [
|
||||
[{}, sha, true],
|
||||
[{ event_name: "pull_request", ref: "refs/pull/1/merge" }, sha, false],
|
||||
[{ event_name: "pull_request_target" }, sha, false],
|
||||
[{ event_name: "workflow_dispatch" }, sha, false],
|
||||
[{ repository: "someone/paperclip" }, sha, false],
|
||||
[{ ref: "refs/heads/feature" }, sha, false],
|
||||
[{}, "b".repeat(40), false],
|
||||
]) {
|
||||
const context = { matrix: { lane }, github: { ...base, ...overrides }, inputs: { ref } };
|
||||
assert.equal(runInNewContext(expr(cache, "if"), context), trusted);
|
||||
assert.equal(runInNewContext(expr(cache, "save-if"), context), trusted && lane === "rust");
|
||||
}
|
||||
}
|
||||
});
|
||||
|
|
|
|||
|
|
@ -262,12 +262,21 @@ jobs:
|
|||
run: pnpm test:runner-workflow-evals
|
||||
|
||||
verify_paperclip_runner:
|
||||
name: Verify Paperclip Runner
|
||||
name: Verify Paperclip Runner (${{ matrix.lane }})
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- lane: protocol
|
||||
checks: check:eval-kernel check:protocol
|
||||
- lane: rust
|
||||
checks: check:runner check:api-authority
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
|
|
@ -305,15 +314,27 @@ jobs:
|
|||
# dependencies; never restore installed executables from cargo/bin.
|
||||
cache-workspace-crates: false
|
||||
cache-bin: false
|
||||
# The step guard also restricts restores. Save only after a successful
|
||||
# master-push verification of that push's exact commit.
|
||||
save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
||||
# Both lanes restore the existing dependency cache. Only the Rust
|
||||
# lane saves it, after warming both release and debug dependencies.
|
||||
save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Verify Paperclip Runner
|
||||
run: pnpm --filter @paperclipai/paperclip-runner check:all
|
||||
env:
|
||||
RUNNER_CHECKS: ${{ matrix.checks }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
for check in $RUNNER_CHECKS; do
|
||||
pnpm --filter @paperclipai/paperclip-runner "$check"
|
||||
done
|
||||
|
||||
- name: Warm debug dependencies for the shared Runner cache
|
||||
# Protocol tests need debug binaries. Populate their dependencies in
|
||||
# the sole cache writer, so a cold save also serves the protocol lane.
|
||||
if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}
|
||||
run: pnpm --filter @paperclipai/paperclip-runner build:rust
|
||||
|
||||
build:
|
||||
name: Build
|
||||
|
|
|
|||
|
|
@ -338,19 +338,27 @@ Check:
|
|||
|
||||
## Runner verification dependency cache
|
||||
|
||||
`release-verify.yml` caches Cargo dependencies for its `Verify Paperclip Runner`
|
||||
job using a pinned Rust Cache action. It selects the compiler from the Runner
|
||||
package's `rust-toolchain.toml` before computing the cache key. Compiler and Cargo
|
||||
metadata changes select a new cache; the `release-runner-v1` shared key lets
|
||||
callers of this reusable verification workflow reuse the same dependency cache.
|
||||
`release-verify.yml` runs `Verify Paperclip Runner` on two independent runners.
|
||||
The protocol lane runs `check:eval-kernel` and `check:protocol`. The Rust lane
|
||||
runs `check:runner` and `check:api-authority`. Together they retain every check
|
||||
in `check:all`; both lanes must pass before Cloud source verification or
|
||||
readiness can succeed. A failed lane does not cancel the other lane.
|
||||
|
||||
Workspace crates and installed Cargo binaries are excluded. Every run still
|
||||
builds the Runner workspace and runs `check:all`, including the Rust and
|
||||
TypeScript tests. Only an own-repository master-push run verifying that push's exact
|
||||
SHA can restore the cache, and only a successful run saves it. PR, tag, and
|
||||
manual candidate verification compile without this cache. A miss or eviction costs compilation time but does not change the checks.
|
||||
To discard old dependency caches, increment the shared-key version and let the
|
||||
next successful master verification warm it again.
|
||||
Both lanes restore Cargo dependencies with the pinned Rust Cache action. The
|
||||
compiler comes from the Runner package's `rust-toolchain.toml` before the action
|
||||
computes its key. Compiler and Cargo metadata changes select a new cache. The
|
||||
existing `release-runner-v1` shared key avoids separate copies for these lanes.
|
||||
Only the Rust lane saves this cache. After verification it also runs `build:rust`
|
||||
to warm the debug dependencies used by the protocol lane; its own tests already
|
||||
warm release dependencies. The cache writer is shorter than the protocol lane.
|
||||
|
||||
Workspace crates and installed Cargo binaries are excluded. Every run rebuilds
|
||||
workspace code and runs all assigned checks, including on a cache hit. Only an
|
||||
own-repository master-push run verifying that push's exact SHA can restore the
|
||||
cache, and only a successful Rust lane saves it. PR, tag, and manual candidate
|
||||
verification compile without this cache. A miss or eviction costs compilation
|
||||
time but does not change the checks. To discard old dependency caches, increment
|
||||
the shared-key version and let the next successful master verification warm it.
|
||||
|
||||
The trust boundary is the protected master branch, not the cache-key text.
|
||||
GitHub does not let master restore caches created by a child branch, sibling
|
||||
|
|
|
|||
|
|
@ -221,10 +221,12 @@ test("release verify workflow covers the same split test surface as stable PR ve
|
|||
);
|
||||
assert.match(verifyWorkflow, /pnpm -r typecheck/);
|
||||
assert.match(verifyWorkflow, /pnpm build/);
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
/pnpm --filter @paperclipai\/paperclip-runner check:all/,
|
||||
);
|
||||
const runnerScripts = JSON.parse(readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8")).scripts;
|
||||
const runnerChecks = [...verifyWorkflow.matchAll(/^ checks: (.+)$/gm)]
|
||||
.flatMap(([, checks]) => checks.split(" "));
|
||||
assert.deepEqual(runnerChecks, runnerScripts["check:all"].split(" && ")
|
||||
.map((command) => command.replace(/^pnpm run /, "")));
|
||||
assert.match(verifyWorkflow, /pnpm --filter @paperclipai\/paperclip-runner "\$check"/);
|
||||
assert.match(verifyWorkflow, /runner_workflow_evals:/);
|
||||
assert.match(verifyWorkflow, /runner_chaos_evals:/);
|
||||
assert.match(
|
||||
|
|
|
|||
Loading…
Reference in New Issue