From fc06f7f05f42c675be71ff0927b6334405d520ed Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 10 Sep 2026 23:59:44 -0700 Subject: [PATCH] fix(ci): isolate chaos verification by caller workflow (#13208) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud deployments require verified artifacts for the merged source commit. > - Cloud readiness and the npm release independently run the same source checks. > - Their shared chaos workflow used only the source ref as its concurrency key. > - One caller could cancel the other caller's required job for the same commit. > - This pull request scopes that key to the caller workflow and source ref. > - Both callers can finish their checks without blocking deployment readiness. ## Linked Issues or Issue Description Refs #13192 and #13205. Searched for related open issues and PRs; no duplicate fix was found. **What happened?** The master push for `398d304e15739d1ee6105633bd8a0e42c929d33f` started Cloud readiness and Release together. GitHub cancelled the Cloud readiness chaos job before it acquired a runner. Its annotation reported a higher-priority waiting request for the same concurrency group. The required readiness gate cannot pass after that cancellation. **Expected behavior** Cloud readiness and Release must each finish source verification for the same SHA. Standalone chaos evals must also have a separate group. **Steps to reproduce** Merge a commit to master while the npm release queue is empty. Both callers reach the reusable chaos workflow with the same source SHA. See [the cancelled job](https://github.com/paperclipai/paperclip/actions/runs/34569569760/job/103168603926). **Paperclip version or commit** `398d304e15739d1ee6105633bd8a0e42c929d33f`. **Deployment mode** GitHub Actions on master. ## What Changed - Add the caller workflow name to the chaos workflow concurrency group. Retain source isolation and cancellation of duplicate calls within the same workflow. - Add a regression test that evaluates the group for Cloud readiness, Release, and standalone evals at the same source SHA. - Document the concurrency boundary in the readiness runbook. ## Verification - `node --test scripts/preview-artifacts.test.mjs scripts/__tests__/release-verify-workflow.test.mjs` passed: 26 tests. - The new regression test fails against the previous concurrency key and passes with this fix. - `actionlint -shellcheck= -pyflakes= .github/workflows/runner-chaos-evals.yml .github/workflows/release-verify.yml .github/workflows/cloud-readiness.yml` passed. - `git diff --check` passed. - The full local typecheck passed for the same application source in #13205. Its macOS general-server test phase had 10,471 passes and 70 failures in seven unchanged application test files: missing Cargo/Runner test binaries, filesystem permissions, timeouts, a port conflict, and a load-test count mismatch. Linux CI test checks passed. The full local build passed with Cargo on PATH. This PR changes workflow configuration, its test, and documentation only. - All CI checks pass on the final head, including typecheck, tests, browser suites, build, and canary dry run. Greptile is 5/5 with no open findings. After merge, verify both callers' chaos jobs complete for the same master SHA and record the resulting readiness time. ## Risks - Two callers may now run chaos tests at the same time. This uses two existing GitHub runners, which is the intended cost of independent verification. - Renaming a caller changes its concurrency group. The fixed prefix keeps this child group separate from caller-level concurrency groups. - The readiness gate continues to require every verification prerequisite. No gate is bypassed. ## Model Used - OpenAI GPT-6 / Codex, with reasoning, repository editing, and command/API tools. Exact serving model ID and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (26 focused workflow/artifact tests) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .github/workflows/runner-chaos-evals.yml | 4 +++- doc/cloud-build-readiness.md | 4 ++++ .../release-verify-workflow.test.mjs | 21 +++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-) diff --git a/.github/workflows/runner-chaos-evals.yml b/.github/workflows/runner-chaos-evals.yml index 3131959522..42f8d6f307 100644 --- a/.github/workflows/runner-chaos-evals.yml +++ b/.github/workflows/runner-chaos-evals.yml @@ -12,7 +12,9 @@ on: type: string concurrency: - group: runner-chaos-evals-${{ inputs.ref || github.ref }} + # Reusable calls inherit the caller's workflow name. Cloud readiness and + # Release verify the same SHA independently and must not cancel each other. + group: runner-chaos-evals-${{ github.workflow }}-${{ inputs.ref || github.ref }} cancel-in-progress: true jobs: diff --git a/doc/cloud-build-readiness.md b/doc/cloud-build-readiness.md index 68f7827437..6522f03e05 100644 --- a/doc/cloud-build-readiness.md +++ b/doc/cloud-build-readiness.md @@ -50,6 +50,10 @@ successfully verified a real master commit. ## Timing and rollout +The reusable Runner chaos workflow scopes concurrency to the caller workflow +and source ref. Cloud readiness and the npm release can verify the same commit +at the same time. They must not cancel each other's required test job. + Measure the complete path from a master merge to a healthy target running that exact commit. Keep readiness and deployment as separate milestones: diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 3ff0d35b27..9def9e3800 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -14,6 +14,27 @@ function readWorkflow(name) { return readFileSync(path.join(repoRoot, ".github/workflows", name), "utf8"); } +test("chaos verification isolates callers that verify the same source commit", () => { + const chaosWorkflow = readWorkflow("runner-chaos-evals.yml"); + const group = chaosWorkflow.match(/^ group: (.+)$/m)?.[1]; + assert.ok(group, "chaos verification must define its concurrency group"); + + // GitHub supplies the top-level caller's workflow name to reusable calls. + const resolveGroup = (caller, ref) => group + .replaceAll("${{ github.workflow }}", readWorkflow(caller).match(/^name: (.+)$/m)[1]) + .replaceAll("${{ inputs.ref || github.ref }}", ref) + .toLowerCase(); + const sha = "a".repeat(40); + const callers = ["cloud-readiness.yml", "release.yml", "runner-chaos-evals.yml"]; + const groups = callers.map((caller) => resolveGroup(caller, sha)); + assert.equal(new Set(groups).size, callers.length, + "Cloud readiness, Release, and standalone evals must not cancel each other"); + assert.ok(groups.every((value) => !value.includes("${{")), "resolve every group input"); + assert.notEqual(resolveGroup("cloud-readiness.yml", sha), + resolveGroup("cloud-readiness.yml", "b".repeat(40)), "different sources remain independent"); + assert.match(chaosWorkflow, /cancel-in-progress: true/); +}); + test("release workflow delegates stable and canary verification to the reusable workflow", () => { const releaseWorkflow = readWorkflow("release.yml");