From 0ce7df2648ec0d7a0e55f4267a2aeb07b0618896 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 12 Sep 2026 12:32:46 -0700 Subject: [PATCH] ci: keep Cloud readiness markers out of the builder queue (#13330) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud consumes versioned readiness markers for each merged source commit. > - Those markers can be published only after the required checks and artifacts pass. > - The marker jobs currently wait for the same AWS runner capacity as builds and tests. > - A busy builder pool can delay readiness after all required work has finished. > - This PR moves the small readiness jobs to GitHub-hosted runners while retaining every dependency gate. ## Linked Issues or Issue Description Refs #13326 and #13328. **What existing behavior does this improve?** Time from completed Cloud verification to a deployable marker, and AWS capacity occupied by artifact polling. **Current behavior** In [Cloud readiness run 34711557083](https://github.com/paperclipai/paperclip/actions/runs/34711557083), all builds and tests finished at 18:43:05 UTC. The source marker did not start until 18:44:21, and the deployable marker did not start until 18:44:37. Merge-to-deployable was 11m33s, although the prerequisite work finished in 9m44s. **Proposed behavior** Run the artifact wait and both versioned marker jobs on `ubuntu-latest`. Keep the compute jobs on the approved post-merge AWS fleet. **Reason and benefit** Avoid builder-pool queue delays after verification finishes. This also removes the long artifact-wait job from AWS capacity. Expected savings depend on queue depth: the observed run had over 90 seconds of avoidable marker waiting. The marker commands themselves take only seconds. **Breaking changes** Runner placement changes for three bookkeeping jobs. Marker names, exact-source artifact checks, required verification, and image verification stay the same. **Additional context** Searched the related runner and Cloud readiness work. This addresses queue time observed after the parallel verification change. ## What Changed - Place the artifact wait, source-verification marker, and deployable marker on GitHub-hosted runners. - Keep all existing job dependencies, source guards, permissions, and commands. - Extend routing regressions to enforce this placement and retain fail-closed readiness gates. - Document why readiness bookkeeping uses separate runner capacity. ## Verification - Passed 433 workflow, routing, source-verification, and Cloud readiness tests with `node --test .github/scripts/tests/*.test.mjs scripts/cloud-source-verification.test.mjs scripts/cloud-readiness.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Passed `actionlint` and `git diff --check`. - Passed all latest-head CI gates in [run 34712624340, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/34712624340), including typecheck, build, browser, Runner, and all general/serialized tests. - Attempt 1 had one localhost readiness timeout in an unchanged test. A single targeted retry passed all 166 files (3,225 tests passed, one existing skip), including all seven tests in that file. No timeout, assertion, or application source was changed; the retry is documented in the PR comment. - Local full-suite verification is limited by local disk exhaustion; the focused checks above pass. - Fresh Greptile review is 5/5 with no unresolved findings. ## Risks - GitHub-hosted capacity can also queue, but these jobs no longer compete with AWS build/test demand. The change does not reserve instances or change box sizes. - Readiness must still fail if any prerequisite fails. The existing `needs` relationships and success-only execution are preserved and tested. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../tests/post-merge-runner-routing.test.mjs | 26 ++++++++++++++++++- .github/workflows/cloud-readiness.yml | 9 ++++--- doc/RELEASE-AUTOMATION-SETUP.md | 15 +++++++++++ 3 files changed, 46 insertions(+), 4 deletions(-) diff --git a/.github/scripts/tests/post-merge-runner-routing.test.mjs b/.github/scripts/tests/post-merge-runner-routing.test.mjs index 95c2a1c5ef..68a2cada62 100644 --- a/.github/scripts/tests/post-merge-runner-routing.test.mjs +++ b/.github/scripts/tests/post-merge-runner-routing.test.mjs @@ -10,7 +10,7 @@ const base = { ref: "refs/heads/master", event_name: "push", sha, }; const expectedJobs = { - "cloud-readiness.yml": ["artifacts", "source_verified", "ready"], + "cloud-readiness.yml": [], "cloud-artifacts.yml": ["dispatch_migrator"], "release-verify.yml": ["typecheck", "general_tests", "serialized_tests", "runner_workflow_evals", "verify_paperclip_runner", "build"], "runner-chaos-evals.yml": ["chaos_and_recovery"], @@ -81,3 +81,27 @@ for (const [file, expectedNames] of Object.entries(expectedJobs)) { }); } } + + +test("Cloud readiness bookkeeping never waits for the AWS verification fleet", () => { + const workflow = readFileSync(new URL("../../workflows/cloud-readiness.yml", import.meta.url), "utf8"); + const bodies = new Map(); + for (const [name, needs] of [ + ["artifacts", null], + ["source_verified", "[verify]"], + ["ready", "[verify, image, artifacts]"], + ]) { + const body = workflow.match(new RegExp(`^ ${name}:\\n([\\s\\S]*?)(?=^ [a-z_]+:|(?![\\s\\S]))`, "m"))?.[1]; + assert.ok(body, `missing ${name} job`); + bodies.set(name, body); + assert.match(body, /^ runs-on: ubuntu-latest$/m); + assert.doesNotMatch(body, /^ +continue-on-error:|^ +if:.*always\(\)/m); + assert.match(body, /^ if: github.repository == 'paperclipai\/paperclip' && github.ref == 'refs\/heads\/master'$/m); + assert.match(body, /^ +SOURCE_SHA: \$\{\{ github.sha \}\}$/m); + assert.equal(body.match(/^ needs: (.+)$/m)?.[1] ?? null, needs, `${name} prerequisites`); + } + assert.match(bodies.get("artifacts"), /^ run: node scripts\/cloud-readiness.mjs "\$SOURCE_SHA"$/m); + assert.match(bodies.get("source_verified"), /^ run: node --test scripts\/cloud-source-verification.test.mjs$/m); + assert.match(bodies.get("source_verified"), /echo "Cloud source verified v1: \$SOURCE_SHA"/); + assert.match(bodies.get("ready"), /echo "Cloud deployable v1: \$SOURCE_SHA"/); +}); diff --git a/.github/workflows/cloud-readiness.yml b/.github/workflows/cloud-readiness.yml index 002da29614..453cad2172 100644 --- a/.github/workflows/cloud-readiness.yml +++ b/.github/workflows/cloud-readiness.yml @@ -32,7 +32,8 @@ jobs: artifacts: if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master' name: Wait for exact-source cloud artifacts - runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }} + # Bookkeeping must not wait for the AWS builders it observes. + runs-on: ubuntu-latest timeout-minutes: 35 permissions: contents: read @@ -55,7 +56,8 @@ jobs: name: Cloud source verified v1 needs: [verify] if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master' - runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }} + # Bookkeeping must not wait for the AWS builders it observes. + runs-on: ubuntu-latest timeout-minutes: 5 permissions: contents: read @@ -81,7 +83,8 @@ jobs: name: Cloud deployable v1 needs: [verify, image, artifacts] if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master' - runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }} + # Bookkeeping must not wait for the AWS builders it observes. + runs-on: ubuntu-latest timeout-minutes: 5 steps: - name: Record cloud readiness diff --git a/doc/RELEASE-AUTOMATION-SETUP.md b/doc/RELEASE-AUTOMATION-SETUP.md index 4817e5d7fc..5e5afc481d 100644 --- a/doc/RELEASE-AUTOMATION-SETUP.md +++ b/doc/RELEASE-AUTOMATION-SETUP.md @@ -399,3 +399,18 @@ fixture setup; it does not make a single test faster. The file-duration manifest also records the native Codex Runner integration suite's measured import and execution cost, so the existing file balancer accounts for it in both ordinary PR and release verification. + + +## Cloud readiness runner placement + +When AWS routing is enabled, Cloud image builds use `paperclip-cloud-build-x64` +and source verification uses `paperclip-post-merge-x64`. The artifact wait and +the `Cloud source verified v1` and `Cloud deployable v1` marker jobs run on +GitHub-hosted runners. These small jobs must not hold or wait for capacity in +the source-verification fleet. During a merge +burst, even a completed build must wait for its marker before consumers can +recognize readiness. + +Runner placement does not change readiness requirements: exact-source artifacts, +all source checks, and the image verification must still pass. The versioned +markers and their dependency gates are unchanged.