Commit Graph

4 Commits

Author SHA1 Message Date
Devin Foley 0ce7df2648
ci: keep Cloud readiness markers out of the builder queue (#13330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud consumes versioned readiness markers for each merged
source commit.
> - Those markers can be published only after the required checks and
artifacts pass.
> - The marker jobs currently wait for the same AWS runner capacity as
builds and tests.
> - A busy builder pool can delay readiness after all required work has
finished.
> - This PR moves the small readiness jobs to GitHub-hosted runners
while retaining every dependency gate.

## Linked Issues or Issue Description

Refs #13326 and #13328.

**What existing behavior does this improve?**

Time from completed Cloud verification to a deployable marker, and AWS
capacity occupied by artifact polling.

**Current behavior**

In [Cloud readiness run
34711557083](https://github.com/paperclipai/paperclip/actions/runs/34711557083),
all builds and tests finished at 18:43:05 UTC. The source marker did not
start until 18:44:21, and the deployable marker did not start until
18:44:37. Merge-to-deployable was 11m33s, although the prerequisite work
finished in 9m44s.

**Proposed behavior**

Run the artifact wait and both versioned marker jobs on `ubuntu-latest`.
Keep the compute jobs on the approved post-merge AWS fleet.

**Reason and benefit**

Avoid builder-pool queue delays after verification finishes. This also
removes the long artifact-wait job from AWS capacity. Expected savings
depend on queue depth: the observed run had over 90 seconds of avoidable
marker waiting. The marker commands themselves take only seconds.

**Breaking changes**

Runner placement changes for three bookkeeping jobs. Marker names,
exact-source artifact checks, required verification, and image
verification stay the same.

**Additional context**

Searched the related runner and Cloud readiness work. This addresses
queue time observed after the parallel verification change.

## What Changed

- Place the artifact wait, source-verification marker, and deployable
marker on GitHub-hosted runners.
- Keep all existing job dependencies, source guards, permissions, and
commands.
- Extend routing regressions to enforce this placement and retain
fail-closed readiness gates.
- Document why readiness bookkeeping uses separate runner capacity.

## Verification

- Passed 433 workflow, routing, source-verification, and Cloud readiness
tests with `node --test .github/scripts/tests/*.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/cloud-readiness.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`.
- Passed `actionlint` and `git diff --check`.
- Passed all latest-head CI gates in [run 34712624340, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/34712624340),
including typecheck, build, browser, Runner, and all general/serialized
tests.
- Attempt 1 had one localhost readiness timeout in an unchanged test. A
single targeted retry passed all 166 files (3,225 tests passed, one
existing skip), including all seven tests in that file. No timeout,
assertion, or application source was changed; the retry is documented in
the PR comment.
- Local full-suite verification is limited by local disk exhaustion; the
focused checks above pass.
- Fresh Greptile review is 5/5 with no unresolved findings.

## Risks

- GitHub-hosted capacity can also queue, but these jobs no longer
compete with AWS build/test demand. The change does not reserve
instances or change box sizes.
- Readiness must still fail if any prerequisite fails. The existing
`needs` relationships and success-only execution are preserved and
tested.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 12:32:46 -07:00
Devin Foley bc68312327
ci: use reserved AWS capacity for post-merge cloud verification (#13257)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployments consume a verified image and exact-source
migrator.
> - An image alone is not deployable until source checks and artifact
checks pass.
> - GitHub-hosted queues delayed those checks and the final readiness
signal.
> - This PR gives trusted master work a separate concurrency allowance
on existing AWS runners.
> - Community PRs and arbitrary source inputs keep the GitHub-hosted
fallback.

## Linked Issues or Issue Description

Refs #13243.

**What existing behavior does this improve?**
Time from a master merge to the Cloud deployable v1 signal.

**Current behavior**
For merge d0b7ba4, the image was available after 7m 42s, but readiness
took 16m 05s. Typecheck queued for 6m 40s and the final readiness job
queued for 1m 46s.

**Proposed behavior**
Allow up to 36 concurrent post-merge verification and migrator jobs on
the existing four-vCPU, 16-GiB AWS runners. Workers launch on demand and
terminate after their job; no always-on worker pool or AWS Reserved
Instance purchase is introduced. Keep the combined runner ceiling
unchanged. A separate operator switch enables this route only after the
restricted runner group and Fleet exist.

**Reason and benefit**
Remove GitHub-hosted queue delays from the cloud deployment path. The
gain depends on queue pressure and which remaining job finishes last;
the observed queues are not additive savings.

**Breaking changes**
None to source verification or readiness contracts. Paid routing is
limited to canonical master push/manual events, with exact source checks
on reusable and migrator jobs.

## What Changed

- Route source verification, artifact waiting, dispatch, and readiness
jobs to the separate post-merge Fleet when enabled.
- Require source inputs to match the event's master SHA. Preview inputs
and raced older migrator dispatches stay GitHub-hosted.
- Keep npm publication on GitHub-hosted runners for trusted publishing.
- Bound AWS job timeouts below the 45-minute instance lifetime.
- Document activation, capacity reservation, and rollback.
- Exercise each actual runner selector against allowed and rejected
event/source combinations.

## Verification

- 268 routing and timeout cases pass, including unapproved PR, fork,
branch/tag, arbitrary ref, and disabled-switch cases.
- All 461 focused workflow, preview, and readiness tests pass. The 284
routing/preview cases also pass after the review fixes.
- actionlint passes for changed workflows with the existing
SC2012/SC2016/SC2129 warnings excluded.
- Full local typecheck and build pass (167s and 206s). `pnpm test:run`
completed: 10,600 passed, 65 skipped, and 13 failed in the unchanged
company-skills-service/runtime-skill-cache suites with local filesystem
permission errors. Linux CI is the required test gate; this is not a
claim of a fully passing local suite. Current-head Linux CI is green,
Greptile is 5/5, and all findings are resolved. The final Build retry
passed on a verified 60 GiB AWS runner after correcting the earlier
disk-capacity failure.
- After activation, verify a master run selects the separate group and
all readiness prerequisites pass.

## Risks

- A missing or incorrectly restricted runner group can leave eligible
jobs queued. Enable the switch only after Fleet and group verification.
- PR bursts have 64 slots after reserving 36 for post-merge work. The
image Fleet retains eight, for the same 108-runner total.
- A migrator dispatch racing a newer merge uses GitHub-hosted runners.
This preserves source trust but can retain some queue delay.
- Roll back placement by disabling AWS_POST_MERGE_CI_ENABLED and
rerunning the whole workflow.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 14:13:01 -07:00
Devin Foley 4fde92107e
fix(ci): reuse cloud source verification for npm canaries (#13233)
Reuse the exact master source-verification result before npm canary publication, removing a duplicate verification matrix while preserving fail-closed release checks.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-11 09:26:26 -07:00
Devin Foley d56be3f3fc
fix(ci): verify deployable cloud artifacts independently (#13192)
Verify source, build the cloud image, and wait for exact-source migrator packages concurrently. Emit Cloud deployable v1 only when every prerequisite succeeds for the merged full SHA.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-10 21:08:29 -07:00