ci: keep Cloud readiness markers out of the builder queue (#13330)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud consumes versioned readiness markers for each merged
source commit.
> - Those markers can be published only after the required checks and
artifacts pass.
> - The marker jobs currently wait for the same AWS runner capacity as
builds and tests.
> - A busy builder pool can delay readiness after all required work has
finished.
> - This PR moves the small readiness jobs to GitHub-hosted runners
while retaining every dependency gate.

## Linked Issues or Issue Description

Refs #13326 and #13328.

**What existing behavior does this improve?**

Time from completed Cloud verification to a deployable marker, and AWS
capacity occupied by artifact polling.

**Current behavior**

In [Cloud readiness run
34711557083](https://github.com/paperclipai/paperclip/actions/runs/34711557083),
all builds and tests finished at 18:43:05 UTC. The source marker did not
start until 18:44:21, and the deployable marker did not start until
18:44:37. Merge-to-deployable was 11m33s, although the prerequisite work
finished in 9m44s.

**Proposed behavior**

Run the artifact wait and both versioned marker jobs on `ubuntu-latest`.
Keep the compute jobs on the approved post-merge AWS fleet.

**Reason and benefit**

Avoid builder-pool queue delays after verification finishes. This also
removes the long artifact-wait job from AWS capacity. Expected savings
depend on queue depth: the observed run had over 90 seconds of avoidable
marker waiting. The marker commands themselves take only seconds.

**Breaking changes**

Runner placement changes for three bookkeeping jobs. Marker names,
exact-source artifact checks, required verification, and image
verification stay the same.

**Additional context**

Searched the related runner and Cloud readiness work. This addresses
queue time observed after the parallel verification change.

## What Changed

- Place the artifact wait, source-verification marker, and deployable
marker on GitHub-hosted runners.
- Keep all existing job dependencies, source guards, permissions, and
commands.
- Extend routing regressions to enforce this placement and retain
fail-closed readiness gates.
- Document why readiness bookkeeping uses separate runner capacity.

## Verification

- Passed 433 workflow, routing, source-verification, and Cloud readiness
tests with `node --test .github/scripts/tests/*.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/cloud-readiness.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`.
- Passed `actionlint` and `git diff --check`.
- Passed all latest-head CI gates in [run 34712624340, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/34712624340),
including typecheck, build, browser, Runner, and all general/serialized
tests.
- Attempt 1 had one localhost readiness timeout in an unchanged test. A
single targeted retry passed all 166 files (3,225 tests passed, one
existing skip), including all seven tests in that file. No timeout,
assertion, or application source was changed; the retry is documented in
the PR comment.
- Local full-suite verification is limited by local disk exhaustion; the
focused checks above pass.
- Fresh Greptile review is 5/5 with no unresolved findings.

## Risks

- GitHub-hosted capacity can also queue, but these jobs no longer
compete with AWS build/test demand. The change does not reserve
instances or change box sizes.
- Readiness must still fail if any prerequisite fails. The existing
`needs` relationships and success-only execution are preserved and
tested.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley 2026-09-12 12:32:46 -07:00 committed by GitHub
parent 7435b2ee9c
commit 0ce7df2648
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 46 additions and 4 deletions

View File

@ -10,7 +10,7 @@ const base = {
ref: "refs/heads/master", event_name: "push", sha,
};
const expectedJobs = {
"cloud-readiness.yml": ["artifacts", "source_verified", "ready"],
"cloud-readiness.yml": [],
"cloud-artifacts.yml": ["dispatch_migrator"],
"release-verify.yml": ["typecheck", "general_tests", "serialized_tests", "runner_workflow_evals", "verify_paperclip_runner", "build"],
"runner-chaos-evals.yml": ["chaos_and_recovery"],
@ -81,3 +81,27 @@ for (const [file, expectedNames] of Object.entries(expectedJobs)) {
});
}
}
test("Cloud readiness bookkeeping never waits for the AWS verification fleet", () => {
const workflow = readFileSync(new URL("../../workflows/cloud-readiness.yml", import.meta.url), "utf8");
const bodies = new Map();
for (const [name, needs] of [
["artifacts", null],
["source_verified", "[verify]"],
["ready", "[verify, image, artifacts]"],
]) {
const body = workflow.match(new RegExp(`^ ${name}:\\n([\\s\\S]*?)(?=^ [a-z_]+:|(?![\\s\\S]))`, "m"))?.[1];
assert.ok(body, `missing ${name} job`);
bodies.set(name, body);
assert.match(body, /^ runs-on: ubuntu-latest$/m);
assert.doesNotMatch(body, /^ +continue-on-error:|^ +if:.*always\(\)/m);
assert.match(body, /^ if: github.repository == 'paperclipai\/paperclip' && github.ref == 'refs\/heads\/master'$/m);
assert.match(body, /^ +SOURCE_SHA: \$\{\{ github.sha \}\}$/m);
assert.equal(body.match(/^ needs: (.+)$/m)?.[1] ?? null, needs, `${name} prerequisites`);
}
assert.match(bodies.get("artifacts"), /^ run: node scripts\/cloud-readiness.mjs "\$SOURCE_SHA"$/m);
assert.match(bodies.get("source_verified"), /^ run: node --test scripts\/cloud-source-verification.test.mjs$/m);
assert.match(bodies.get("source_verified"), /echo "Cloud source verified v1: \$SOURCE_SHA"/);
assert.match(bodies.get("ready"), /echo "Cloud deployable v1: \$SOURCE_SHA"/);
});

View File

@ -32,7 +32,8 @@ jobs:
artifacts:
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
name: Wait for exact-source cloud artifacts
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
# Bookkeeping must not wait for the AWS builders it observes.
runs-on: ubuntu-latest
timeout-minutes: 35
permissions:
contents: read
@ -55,7 +56,8 @@ jobs:
name: Cloud source verified v1
needs: [verify]
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
# Bookkeeping must not wait for the AWS builders it observes.
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
@ -81,7 +83,8 @@ jobs:
name: Cloud deployable v1
needs: [verify, image, artifacts]
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }}
# Bookkeeping must not wait for the AWS builders it observes.
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Record cloud readiness

View File

@ -399,3 +399,18 @@ fixture setup; it does not make a single test faster.
The file-duration manifest also records the native Codex Runner integration
suite's measured import and execution cost, so the existing file balancer
accounts for it in both ordinary PR and release verification.
## Cloud readiness runner placement
When AWS routing is enabled, Cloud image builds use `paperclip-cloud-build-x64`
and source verification uses `paperclip-post-merge-x64`. The artifact wait and
the `Cloud source verified v1` and `Cloud deployable v1` marker jobs run on
GitHub-hosted runners. These small jobs must not hold or wait for capacity in
the source-verification fleet. During a merge
burst, even a completed build must wait for its marker before consumers can
recognize readiness.
Runner placement does not change readiness requirements: exact-source artifacts,
all source checks, and the image verification must still pass. The versioned
markers and their dependency gates are unchanged.