ci: keep Cloud readiness markers out of the builder queue (#13330)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud consumes versioned readiness markers for each merged source commit. > - Those markers can be published only after the required checks and artifacts pass. > - The marker jobs currently wait for the same AWS runner capacity as builds and tests. > - A busy builder pool can delay readiness after all required work has finished. > - This PR moves the small readiness jobs to GitHub-hosted runners while retaining every dependency gate. ## Linked Issues or Issue Description Refs #13326 and #13328. **What existing behavior does this improve?** Time from completed Cloud verification to a deployable marker, and AWS capacity occupied by artifact polling. **Current behavior** In [Cloud readiness run 34711557083](https://github.com/paperclipai/paperclip/actions/runs/34711557083), all builds and tests finished at 18:43:05 UTC. The source marker did not start until 18:44:21, and the deployable marker did not start until 18:44:37. Merge-to-deployable was 11m33s, although the prerequisite work finished in 9m44s. **Proposed behavior** Run the artifact wait and both versioned marker jobs on `ubuntu-latest`. Keep the compute jobs on the approved post-merge AWS fleet. **Reason and benefit** Avoid builder-pool queue delays after verification finishes. This also removes the long artifact-wait job from AWS capacity. Expected savings depend on queue depth: the observed run had over 90 seconds of avoidable marker waiting. The marker commands themselves take only seconds. **Breaking changes** Runner placement changes for three bookkeeping jobs. Marker names, exact-source artifact checks, required verification, and image verification stay the same. **Additional context** Searched the related runner and Cloud readiness work. This addresses queue time observed after the parallel verification change. ## What Changed - Place the artifact wait, source-verification marker, and deployable marker on GitHub-hosted runners. - Keep all existing job dependencies, source guards, permissions, and commands. - Extend routing regressions to enforce this placement and retain fail-closed readiness gates. - Document why readiness bookkeeping uses separate runner capacity. ## Verification - Passed 433 workflow, routing, source-verification, and Cloud readiness tests with `node --test .github/scripts/tests/*.test.mjs scripts/cloud-source-verification.test.mjs scripts/cloud-readiness.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Passed `actionlint` and `git diff --check`. - Passed all latest-head CI gates in [run 34712624340, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/34712624340), including typecheck, build, browser, Runner, and all general/serialized tests. - Attempt 1 had one localhost readiness timeout in an unchanged test. A single targeted retry passed all 166 files (3,225 tests passed, one existing skip), including all seven tests in that file. No timeout, assertion, or application source was changed; the retry is documented in the PR comment. - Local full-suite verification is limited by local disk exhaustion; the focused checks above pass. - Fresh Greptile review is 5/5 with no unresolved findings. ## Risks - GitHub-hosted capacity can also queue, but these jobs no longer compete with AWS build/test demand. The change does not reserve instances or change box sizes. - Readiness must still fail if any prerequisite fails. The existing `needs` relationships and success-only execution are preserved and tested. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
7435b2ee9c
commit
0ce7df2648
|
|
@ -10,7 +10,7 @@ const base = {
|
|||
ref: "refs/heads/master", event_name: "push", sha,
|
||||
};
|
||||
const expectedJobs = {
|
||||
"cloud-readiness.yml": ["artifacts", "source_verified", "ready"],
|
||||
"cloud-readiness.yml": [],
|
||||
"cloud-artifacts.yml": ["dispatch_migrator"],
|
||||
"release-verify.yml": ["typecheck", "general_tests", "serialized_tests", "runner_workflow_evals", "verify_paperclip_runner", "build"],
|
||||
"runner-chaos-evals.yml": ["chaos_and_recovery"],
|
||||
|
|
@ -81,3 +81,27 @@ for (const [file, expectedNames] of Object.entries(expectedJobs)) {
|
|||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
test("Cloud readiness bookkeeping never waits for the AWS verification fleet", () => {
|
||||
const workflow = readFileSync(new URL("../../workflows/cloud-readiness.yml", import.meta.url), "utf8");
|
||||
const bodies = new Map();
|
||||
for (const [name, needs] of [
|
||||
["artifacts", null],
|
||||
["source_verified", "[verify]"],
|
||||
["ready", "[verify, image, artifacts]"],
|
||||
]) {
|
||||
const body = workflow.match(new RegExp(`^ ${name}:\\n([\\s\\S]*?)(?=^ [a-z_]+:|(?![\\s\\S]))`, "m"))?.[1];
|
||||
assert.ok(body, `missing ${name} job`);
|
||||
bodies.set(name, body);
|
||||
assert.match(body, /^ runs-on: ubuntu-latest$/m);
|
||||
assert.doesNotMatch(body, /^ +continue-on-error:|^ +if:.*always\(\)/m);
|
||||
assert.match(body, /^ if: github.repository == 'paperclipai\/paperclip' && github.ref == 'refs\/heads\/master'$/m);
|
||||
assert.match(body, /^ +SOURCE_SHA: \$\{\{ github.sha \}\}$/m);
|
||||
assert.equal(body.match(/^ needs: (.+)$/m)?.[1] ?? null, needs, `${name} prerequisites`);
|
||||
}
|
||||
assert.match(bodies.get("artifacts"), /^ run: node scripts\/cloud-readiness.mjs "\$SOURCE_SHA"$/m);
|
||||
assert.match(bodies.get("source_verified"), /^ run: node --test scripts\/cloud-source-verification.test.mjs$/m);
|
||||
assert.match(bodies.get("source_verified"), /echo "Cloud source verified v1: \$SOURCE_SHA"/);
|
||||
assert.match(bodies.get("ready"), /echo "Cloud deployable v1: \$SOURCE_SHA"/);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -32,7 +32,8 @@ jobs:
|
|||
artifacts:
|
||||
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
|
||||
name: Wait for exact-source cloud artifacts
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
# Bookkeeping must not wait for the AWS builders it observes.
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 35
|
||||
permissions:
|
||||
contents: read
|
||||
|
|
@ -55,7 +56,8 @@ jobs:
|
|||
name: Cloud source verified v1
|
||||
needs: [verify]
|
||||
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
# Bookkeeping must not wait for the AWS builders it observes.
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
|
|
@ -81,7 +83,8 @@ jobs:
|
|||
name: Cloud deployable v1
|
||||
needs: [verify, image, artifacts]
|
||||
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
|
||||
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
|
||||
# Bookkeeping must not wait for the AWS builders it observes.
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Record cloud readiness
|
||||
|
|
|
|||
|
|
@ -399,3 +399,18 @@ fixture setup; it does not make a single test faster.
|
|||
The file-duration manifest also records the native Codex Runner integration
|
||||
suite's measured import and execution cost, so the existing file balancer
|
||||
accounts for it in both ordinary PR and release verification.
|
||||
|
||||
|
||||
## Cloud readiness runner placement
|
||||
|
||||
When AWS routing is enabled, Cloud image builds use `paperclip-cloud-build-x64`
|
||||
and source verification uses `paperclip-post-merge-x64`. The artifact wait and
|
||||
the `Cloud source verified v1` and `Cloud deployable v1` marker jobs run on
|
||||
GitHub-hosted runners. These small jobs must not hold or wait for capacity in
|
||||
the source-verification fleet. During a merge
|
||||
burst, even a completed build must wait for its marker before consumers can
|
||||
recognize readiness.
|
||||
|
||||
Runner placement does not change readiness requirements: exact-source artifacts,
|
||||
all source checks, and the image verification must still pass. The versioned
|
||||
markers and their dependency gates are unchanged.
|
||||
|
|
|
|||
Loading…
Reference in New Issue