## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys verified images from merged source commits.
> - Cloud readiness waits for every release verification check.
> - Runner verification currently runs long TypeScript tests before Rust
checks.
> - These checks can run on independent runners with their own build
directories.
> - This PR runs them in parallel while preserving all checks and the
shared dependency cache.
## Linked Issues or Issue Description
Refs #13194. Related prior work: #13142 and #13259. A search found no
duplicate parallel release-check change.
**What existing behavior does this improve?**
Time from merge to Cloud source verification and deployment readiness.
**Current behavior**
Recent successful runs take roughly 13 minutes from merge to deployable.
In run 34705914878, Runner verification took 11m23s. Protocol tests
finished before Rust tests and API authority checks started.
**Proposed behavior**
Run protocol and Rust verification in two matrix jobs. Cloud readiness
still requires both jobs to pass.
**Reason and benefit**
Remove the serial dependency between independent checks. Expected
improvement is about 2–3 minutes on a typical cached run, until the
image build or server tests become the longest job. This is an estimate;
post-merge timing will confirm it.
**Breaking changes**
Individual release Runner job names gain a lane suffix. Cloud source and
readiness marker names stay the same. PR runner routing is unchanged.
## What Changed
- Split release Runner checks into protocol and Rust lanes. Keep every
constituent of `check:all` exactly once.
- Restore the existing Rust dependency cache in both lanes. Allow only
the Rust lane to save it after warming both build profiles.
- Add coverage and cache authorization regressions. Document the
parallel verification and single cache writer.
## Verification
- Passed 477 workflow and source-verification tests with `node --test
.github/scripts/tests/*.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`.
- Passed `actionlint`, `git diff --check`, and the private AWS routing
regression suite.
- Passed local `pnpm -r typecheck` and the standalone `check:runner &&
check:api-authority` lane, including all 1,671 API tests before the
protocol lane had built TypeScript output.
- The broad local protocol run under Node 25 had four failures. The two
affected files passed under CI's Node 24.19.0: 67 passed, 6 platform
skips.
- Local `pnpm test:run` aborted when disk space ran out; local `pnpm
build` could not run afterward. These are local verification limits.
[Linux CI run
34710421424](https://github.com/paperclipai/paperclip/actions/runs/34710421424)
passed full typecheck, all grouped tests, native verification, build,
release dry run, and browser checks. Native protocol CI passed 1,986
tests, plus 1,671 API tests and the Rust suites.
- Latest-head Greptile is 5/5 with no open findings. All 33 current-head
checks are successful or intentionally skipped.
## Risks
- Uses one additional short-lived verification runner per release
verification. The existing AWS exact-master restriction remains in
place.
- The Rust lane warms debug dependencies so its cache save also serves
protocol tests. Both lanes always rebuild workspace code.
- A workflow regression could omit a check. The new coverage test
compares the matrix checks directly with `check:all`; Cloud readiness
depends on the complete reusable workflow.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployment requires source verification for the exact merged
commit.
> - Contributor PRs leave lockfile updates to a separate bot PR.
> - Most release checks can refresh an outdated lockfile while
installing dependencies.
> - Two Runner checks still require a frozen lockfile and fail after
dependency changes.
> - This PR gives those checks the same install policy as the other
release checks.
> - A valid dependency change can become deployable without waiting for
another merge.
## Linked Issues or Issue Description
Refs #13257. The dependency change in #13256 exposed this gap. The
separate lockfile update is #13279. Related #12115 addresses the bot PR
check trigger; this PR fixes exact-source cloud verification itself.
**What happened?**
[Cloud readiness for
2083bf6](https://github.com/paperclipai/paperclip/actions/runs/34651761811)
failed in the Runner scorer and chaos jobs with
`ERR_PNPM_OUTDATED_LOCKFILE`. The commit added `svix` to server
dependencies. The tracked lockfile still describes the previous
manifest. The other release checks install with `--no-frozen-lockfile`.
**Expected behavior**
Every source check installs and tests the same checked-out commit. A
pending bot lockfile PR must not block cloud readiness.
**Steps to reproduce**
1. Check out master commit 250deab, which retains the manifest/lockfile
mismatch.
2. Run `pnpm install --ignore-scripts --frozen-lockfile`. It fails with
the same outdated-lockfile error.
3. Run `pnpm install --ignore-scripts --no-frozen-lockfile
--resolution-only`. It succeeds.
4. Restore the generated lockfile. This PR does not commit it.
## What Changed
- Use `--no-frozen-lockfile` in the release Runner scorer job.
- Use the same option in the reusable Runner chaos workflow.
- Document why cloud source checks allow a job-local lockfile refresh.
- Update the existing Runner scorer workflow assertion to match its
install policy.
## Verification
- All 457 workflow tests pass across `.github/scripts/tests/*.test.mjs`
and `scripts/__tests__/release-verify-workflow.test.mjs`.
- `actionlint` passes for both changed workflows.
- Reproduced the frozen install failure against the real tracked
manifest and lockfile. The refresh command passes in 4.6 seconds.
- `git diff --check` passes. No lockfile changes remain.
- No application source changes. Full local application typecheck,
build, and test commands were not rerun in this dependency-free workflow
worktree. Current-head GitHub CI must pass before merge.
- After merge, verify both affected jobs pass on the exact master source
even if the lockfile bot PR remains pending.
## Risks
pnpm can resolve allowed dependency ranges when a manifest outgrows the
tracked lockfile. This matches the existing release install policy. The
resulting lockfile stays in the job workspace. Verification commands and
runner routing are unchanged. The security reviewer explicitly accepted
this existing dependency-policy tradeoff for both jobs after reviewing
repository policy and the source/authorization checks. A future shared
immutable dependency artifact would improve reproducibility across jobs.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] Local verification passes: all 457 workflow tests, actionlint, and
the stale-lockfile reproduction described above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployments consume a verified image and exact-source
migrator.
> - An image alone is not deployable until source checks and artifact
checks pass.
> - GitHub-hosted queues delayed those checks and the final readiness
signal.
> - This PR gives trusted master work a separate concurrency allowance
on existing AWS runners.
> - Community PRs and arbitrary source inputs keep the GitHub-hosted
fallback.
## Linked Issues or Issue Description
Refs #13243.
**What existing behavior does this improve?**
Time from a master merge to the Cloud deployable v1 signal.
**Current behavior**
For merge d0b7ba4, the image was available after 7m 42s, but readiness
took 16m 05s. Typecheck queued for 6m 40s and the final readiness job
queued for 1m 46s.
**Proposed behavior**
Allow up to 36 concurrent post-merge verification and migrator jobs on
the existing four-vCPU, 16-GiB AWS runners. Workers launch on demand and
terminate after their job; no always-on worker pool or AWS Reserved
Instance purchase is introduced. Keep the combined runner ceiling
unchanged. A separate operator switch enables this route only after the
restricted runner group and Fleet exist.
**Reason and benefit**
Remove GitHub-hosted queue delays from the cloud deployment path. The
gain depends on queue pressure and which remaining job finishes last;
the observed queues are not additive savings.
**Breaking changes**
None to source verification or readiness contracts. Paid routing is
limited to canonical master push/manual events, with exact source checks
on reusable and migrator jobs.
## What Changed
- Route source verification, artifact waiting, dispatch, and readiness
jobs to the separate post-merge Fleet when enabled.
- Require source inputs to match the event's master SHA. Preview inputs
and raced older migrator dispatches stay GitHub-hosted.
- Keep npm publication on GitHub-hosted runners for trusted publishing.
- Bound AWS job timeouts below the 45-minute instance lifetime.
- Document activation, capacity reservation, and rollback.
- Exercise each actual runner selector against allowed and rejected
event/source combinations.
## Verification
- 268 routing and timeout cases pass, including unapproved PR, fork,
branch/tag, arbitrary ref, and disabled-switch cases.
- All 461 focused workflow, preview, and readiness tests pass. The 284
routing/preview cases also pass after the review fixes.
- actionlint passes for changed workflows with the existing
SC2012/SC2016/SC2129 warnings excluded.
- Full local typecheck and build pass (167s and 206s). `pnpm test:run`
completed: 10,600 passed, 65 skipped, and 13 failed in the unchanged
company-skills-service/runtime-skill-cache suites with local filesystem
permission errors. Linux CI is the required test gate; this is not a
claim of a fully passing local suite. Current-head Linux CI is green,
Greptile is 5/5, and all findings are resolved. The final Build retry
passed on a verified 60 GiB AWS runner after correcting the earlier
disk-capacity failure.
- After activation, verify a master run selects the separate group and
all readiness prerequisites pass.
## Risks
- A missing or incorrectly restricted runner group can leave eligible
jobs queued. Enable the switch only after Fleet and group verification.
- PR bursts have 64 slots after reserving 36 for post-merge work. The
image Fleet retains eight, for the same 108-runner total.
- A migrator dispatch racing a newer merge uses GitHub-hosted runners.
This preserves source trust but can retain some queue delay.
- Roll back placement by disabling AWS_POST_MERGE_CI_ENABLED and
rerunning the whole workflow.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud images become deployable only after source verification
passes.
> - The typecheck job builds the native Runner binary through the server
package.
> - Fresh runners repeatedly compile Rust dependencies for that binary.
> - This PR caches those dependencies for exact-source master
verification.
> - Workspace code and every typecheck still rebuild or run as before.
## Linked Issues or Issue Description
Refs #13243 and #13257.
**What existing behavior does this improve?**
The typecheck portion of post-merge cloud source verification.
**Current behavior**
The typecheck job has no Rust dependency cache. An observed release
build in this job took 4m 13s, including dependency compilation.
**Proposed behavior**
Restore dependency build outputs for canonical master pushes with the
exact source SHA. Use a separate cache key from the Runner verification
job, which builds other profiles.
**Reason and benefit**
A warm cache should remove roughly 2–3 minutes of dependency compilation
from this job. Overall deployment gains depend on the remaining critical
path. The first cache population still compiles from scratch.
**Breaking changes**
None. All checks remain enabled. Non-master callers compile without
restoring or saving this cache.
## What Changed
- Select the pinned Rust toolchain before the typecheck cache lookup.
- Reuse the existing pinned Rust cache action with a typecheck-specific
key.
- Exclude workspace crates and installed cargo executables.
- Test restore/save trust boundaries and document cache behavior.
## Verification
- All workflow script tests pass locally, including nine new cache
trust/contract cases.
- actionlint passes for release-verify.yml.
- Full local typecheck and build pass on the same source base (167s and
206s); `pnpm test:run` is still running and is recorded with #13257.
This PR changes only the workflow, cache guard tests, and documentation.
- All 32 current-head checks are successful or intentionally skipped,
including the complete Linux test matrix, build, and Greptile 5/5 with
no unresolved findings. Verify cache population and subsequent restore
on actual master runs.
## Risks
- The first run and any toolchain/dependency invalidation compile from
scratch.
- Cache restore/save overhead reduces the benefit for small dependency
graphs.
- Disable the cache step to roll back; the existing uncached build
remains valid.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. Exact serving model ID and context window are not exposed by
this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud waits for source verification before deploying a new image.
> - The slowest server verification job spends about ten minutes running
tests.
> - Each job uses one test worker to preserve test isolation.
> - This pull request distributes those suites across ten standard
hosted runners.
> - The benefit is a shorter verification path with the same test
coverage.
## Linked Issues or Issue Description
**Current behavior**
In [readiness run
34572340764](https://github.com/paperclipai/paperclip/actions/runs/34572340764),
the slowest server job ran for 638 seconds. Test execution used 594
seconds. This held readiness behind the image job.
**Proposed behavior**
Use ten general server jobs in the reusable release verification
workflow. Keep the three chat jobs and every existing prerequisite. The
complete partition test verifies that no server suite is omitted or
duplicated.
**Reason and benefit**
Reduce merge-to-deployable time on the existing runner type. The next
longest prerequisite was Runner verification at 526 seconds, so the
initial expected total gain is about two minutes rather than a halving
of readiness time. Measure actual queue and execution time before
claiming a result.
Related: #13198 introduced the separate chat lane. #12577 refreshes
duration estimates; this change leaves that manifest alone.
## What Changed
- Increase the general server matrix from five jobs to ten.
- Verify the ten-way partition covers the complete server suite when
combined with the chat lane.
- Document runner demand and the unchanged local and PR grouping.
## Verification
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs`: 29 passed.
- `actionlint .github/workflows/release-verify.yml`: passed.
- Full local `pnpm -r typecheck` and `pnpm build`: passed.
- All latest-head GitHub CI checks passed, including the complete Linux
test partition, build, typecheck, and browser gates. Greptile: 5/5 with
zero open findings.
- [Ten-shard timing
probe](https://github.com/paperclipai/paperclip/actions/runs/34606772388):
all 16 jobs passed; slowest server job 6m 23s versus 10m 38s in the
earlier five-shard sample. This compares the server lane, not total
readiness, and is not a controlled same-source A/B.
- The full local `pnpm test:run` is also running. It has reproduced
previously observed macOS-only failures in unchanged skill-cache and
native-session suites; the corresponding Linux CI suites passed. Final
local results will be attached separately. No affected-workflow test
failed.
## Risks
Five additional concurrent jobs per release verification run increase
runner demand and repeated setup work. Queueing can offset the gain.
Test workers, timeouts, permissions, and readiness requirements stay
unchanged. Revert the matrix and its partition test to restore the
previous split.
## Model Used
OpenAI GPT-6 / Codex, with reasoning, tool use, and code execution. The
exact serving model identifier and context-window size are not exposed
by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the affected workflow tests locally and they pass;
full-suite macOS limitations are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Split release chat verification into three validated test-line shards and balance other server suites across five runners using the measured native Runner integration cost. Retire each chat case's fixtures after assertions, preserve complete test coverage, and exercise the real shard CLI in PR tests.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Cache external Rust dependencies in trusted master release verification after selecting the package-owned toolchain. Keep source compilation and all validation unconditional; restrict both restore and save to the matching master push.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push publishes a canary through release.yml, gated by
release-verify.yml — the fleet's staging deploys and the
nightly/beta/stable chain all start from those canaries
> - release-verify splits the server test suite across three shards with
a 20-minute job cap, while pr-trusted splits the same suite across five
> - The server suite grew on 2026-09-10 and the three shards moved to
17-19 minutes; that evening every push-triggered canary run was
cancelled by the 20-minute cap mid-verify, and no canary published after
18:50 UTC
> - This pull request mirrors pr-trusted's five-way server split in
release-verify, putting shards back at the 10-15 minute range with real
headroom
> - The benefit is a canary lane that reports test verdicts instead of
dying on an infrastructure cap
## Linked Issues or Issue Description
**What happened?**
Push-triggered Release runs stopped publishing canaries on 2026-09-10.
Runs at 19:34, 22:30, and 22:37 UTC were all cancelled by "The job has
exceeded the maximum execution time of 20m0s" on a `verify_canary /
General tests (server (N/3))` shard. No canary published after 18:50
UTC, which also starves the staging fleet's continuous deploys.
**Expected behavior**
release-verify's server shards finish well inside the 20-minute cap and
runs conclude with a test verdict, as pr-trusted's five-way split of the
same suite does (10-15 minutes per shard).
**Steps to reproduce**
1. Compare server shard durations in the `verify_canary` job across
2026-09-10: 11-14 minutes in the morning, 17-19 minutes from 15:06 UTC,
over 20 minutes by evening.
2. Observe runs 34521169020, 34537798488, and 34538332689 cancelled at
the cap.
**Paperclip version or commit**
`master` at `d1ba17eec` (current tip; its canary run was one of the
cancelled ones).
## What Changed
- `release-verify.yml`: the `general-server` matrix goes from three
shards to five, byte-for-byte the shape `pr-trusted.yml` already runs,
with a comment recording why.
## Verification
- The identical five-way split runs green on every pr-trusted run (10-15
minutes per shard today, including on PRs merged this evening).
- The suite's own growth (slower chat-connector tests) is being
addressed separately; this PR only removes the artificial cliff.
## Risks
- Low risk: two more runners per verify run; no test content changes. If
shard durations regress further, the cap fires again — which is the
correct signal once shards have honest headroom.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5` (Claude Fable 5),
extended thinking, tool use via Claude Code CLI.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip uses GitHub Actions to verify changes before release.
> - The Paperclip Runner has a separate verification boundary.
> - The build job currently runs this verification before the workspace
build.
> - This pull request moves runner verification into its own parallel
job.
> - The benefit is clearer CI results and less wait time for independent
work.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The trusted PR and release verification workflows run Paperclip Runner
verification inside the Build job.
**Subsystem affected**
Cross-cutting (GitHub Actions CI workflows).
**Current behavior**
The Build job runs `pnpm --filter @paperclipai/paperclip-runner
check:all` before it builds the workspace. A runner verification failure
appears as a Build failure. The workspace build cannot run in parallel
with runner verification.
**Proposed behavior**
Each workflow has a `Verify Paperclip Runner` job with the same
checkout, dependency install, and command. The Build job only builds its
required outputs. Both jobs run after the same gate and policy jobs.
**Reason and benefit**
The runner command is an independent verification boundary. A dedicated
job gives it a clear status and allows it to run in parallel with Build.
**Breaking changes**
None. The same runner verification command still runs in both workflows.
## What Changed
- Added a dedicated `Verify Paperclip Runner` job to the trusted PR
workflow.
- Added a dedicated `Verify Paperclip Runner` job to the release
verification workflow.
- Kept the Build jobs independent and retained their existing build
commands.
- Updated the trusted-workflow policy test for the additional
dependency-install job.
## Verification
- Ran `git diff --check`.
- Ran `node --test ./scripts/__tests__/e2e-shard.test.mjs`.
- Ran `pnpm exec prettier --check .github/workflows/pr-trusted.yml
.github/workflows/release-verify.yml`.
- Confirmed both jobs retain their prior runner, dependency, and policy
prerequisites.
## Risks
Low risk. The runner verification job repeats the existing setup. It
adds one parallel GitHub Actions runner to each affected workflow.
## Model Used
OpenAI Codex, GPT-5.6, 128k context window, reasoning and tool-use
capabilities.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner subsystem executes agent work across local and managed
provider backends.
> - The lower pull requests restore the task runtime, provider backends,
and managed-provider control plane.
> - The restored system needs repeatable full-stack checks before it can
ship safely.
> - Paid live checks also need clear access, cost, and secret controls.
> - This pull request adds acceptance, live evaluation, chaos, and
release gates for the restored runner stack.
> - The benefit is measurable runner parity with safer release
decisions.
## Linked Issues or Issue Description
**Subsystem affected**
Cross-cutting. This change covers runner tests, release workflows,
server contracts, and evaluation tools.
**Problem or motivation**
The runner stack did not have one complete acceptance surface for native
Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could
miss provider drift, task-view regressions, cost-policy errors, and
destructive cleanup errors.
**Proposed solution**
Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid
workflows. Add live evaluation, chaos, cost-limit, redaction, and
release contract checks. Add AWS AgentCore infrastructure and guarded
provisioning tools. Keep the native runner experimental flag off by
default.
**Alternatives considered**
We considered manual smoke tests only. They do not give repeatable
evidence and they do not protect release branches. We also considered
one large pull request. The stacked pull requests keep each review below
the Greptile file limit.
**Roadmap alignment**
This work supports the shipped Cloud / Sandbox agents milestone and the
shipped Agent evals & feedback milestone in `ROADMAP.md`.
Related stack:
- #12699 adds managed provider backends and lifecycle support.
- #12691 adds qualified OpenCode and ACPX provider backends.
- #12685 restores task runtime rendering and steering.
## What Changed
- Add the runner full-stack harness with 57 catalog cells and 60 unit
tests.
- Add a Daytona runner image with digest-pinned base images and
base-aware image-content checks.
- Add guarded live evaluation and chaos workflows with a fixed
40-execution matrix; live and full-stack paid schedules now run only on
Sundays or by manual dispatch.
- Add in-flight reported-usage cost stops, post-turn cost caps,
exact-threshold failure classification, secret redaction, retry
classification, and actor authorization.
- Reattach stream and hard-budget listeners before restart-recovery
continuations so restored paid sessions cannot bypass in-flight
interruption.
- Preserve OpenCode usage and cost across tool-loop messages and turns
while exposing an explicit current-run delta to durable accounting.
- Keep PNG/WebM evidence in access-controlled artifacts only, reject
SVG, and publish only pruned inert structured per-attempt evidence.
- Add AWS AgentCore infrastructure, provisioning checks, and smoke
tools; reject unsafe model identifiers, require exact stack ownership
markers, and make failed-stack replacement explicit.
- Add evaluation-session contracts and capability reports.
- Add release workflow checks for immutable action pins, frozen
dependency installs, exact weekly cron shape, paid-run guards,
provider-secret isolation, and chaos test paths.
- Reauthorize the original and triggering numeric actor IDs as the first
step of every provider-secret job, including partial reruns, before
checkout or provider access.
- Give each full-stack matrix cell only its matching provider
credential, expose Daytona only to Daytona cells, and disable shared
dependency caches anywhere paid credentials or OIDC write access are
present.
- Protect the legacy manual E2E workflow with the same default-branch,
allowlist, environment, and per-job authorization boundary.
- Rotate live-eval candidates by week and retain 120 days of compatible
history so the seven-week trend window remains viable.
- Restore the root runner-acceptance commands and reconcile reported
snapshots,
raw receipts, and terminal usage without double counting or losing late
usage.
- Mark ACPX token deltas exact only when every budget field is present,
keep
cumulative cost/request authority separate, reject non-USD cost
labeling,
and include thought tokens in output-token budgets.
- Keep `enableNativeRunner` off by default. The acceptance harness
enables it only in its isolated test instance.
## Verification
Passed locally:
- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm test:runner-acceptance:typecheck`
- `pnpm test:runner-acceptance` (19 tests)
- focused OpenCode proxy, driver, runnerd transport, live-session, and
turn-stream tests (106 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/clean-room-server.test.ts` (22 tests)
- `pnpm test:e2e:runner:typecheck`
- `pnpm test:e2e:runner:unit` (62 tests)
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-workflow-evals` (22 tests)
- `pnpm -r typecheck`
- `pnpm build`
- `node --test
packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs`
(6 tests)
- `git diff --check`
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--lib --locked` (161 tests)
- focused ACPX provider-event tests (10 tests)
- The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged.
I did not run paid live provider jobs or provision AWS resources. Those
checks need credentials and can create cost.
## Risks
The paid workflows can create provider cost. They require an allowlisted
original and triggering actor, the protected `runner-e2e-paid`
environment, explicit opt-in variables, and cost limits. The four
provider credentials exist only in that master-only environment, which
requires allowlisted reviewer approval and disables administrator
bypass; repository and organization Actions scopes contain no copies.
Provider usage arrives after a billable request, so the live guard
cannot prevent one request from crossing a threshold. It interrupts
immediately on the first reported threshold hit and permits no
continuation.
Visual evidence can contain secrets rendered as pixels. PNG/WebM remain
only in access-controlled workflow artifacts; SVG and per-attempt XML
are excluded, and S3/Pages receive a pruned structured dashboard.
The AWS scripts can create cloud resources. They use explicit commands,
least-privilege roles, KMS encryption, saved nonsecret metadata, and
explicit teardown.
This pull request does not enable the experimental native runner for
existing instances.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex with GPT-5. The model used extended reasoning, tool use,
code execution, and parallel subagents.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner package now has protocol, transport, provider, catalog,
and authorization foundations.
> - Its first upstream package boundary should expose only the
implemented runtime and test-helper surfaces.
> - Rust correctness belongs in the repository existing build
verification, without introducing a parallel release process.
> - Direct package creation must build the files declared by the package
manifest.
> - This pull request defines the minimal package API and verifies the
optimized runner binaries in the existing PR and release Build jobs.
> - The benefit is a production-ready runner package boundary with
minimal build-process change.
## Linked Issues or Issue Description
Refs #11962
This pull request replaces one bounded part of the archived large runner
change. It follows the package-local authorization change in #12126.
## What Changed
- Export only `@paperclipai/paperclip-runner` and
`@paperclipai/paperclip-runner/testing`.
- Keep Node-only fixture loading and semantic conformance helpers out of
the runtime root.
- Add a provider-neutral semantic conformance kit with stable JSON
comparison and fail-closed input checks.
- Keep deferred SDK, eval, browser, React, lab, and command surfaces
private.
- Pin the runner Rust toolchain to 1.97.1 with the minimal profile and
`rustfmt`.
- Run the Rust workspace tests in release mode.
- Launch the optimized `paperclip-runnerd` and fake-harness binaries in
process-level integration coverage.
- Add one `pnpm --filter @paperclipai/paperclip-runner check:all` step
to each existing PR and release Build job.
- Make the existing server `prepack` lifecycle run its existing build
after it prepares UI assets.
- Document that no production adapter starts runnerd yet.
This revision adds no standalone GitHub Actions job. It adds no server
runner dependency or runner vendoring. It adds no Docker bootstrap or
clean-consumer harness. It does not change `pnpm-lock.yaml`.
## Verification
- `pnpm --filter @paperclipai/paperclip-runner check:all`
- 66 TypeScript tests
- 8 protocol contract tests
- 56 Rust unit and integration tests
- Release-mode integration coverage launches the optimized runnerd and
fake-harness binaries.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/server-package-build-script.test.ts` (2 tests)
- Clean `pnpm pack` from `server/` rebuilt the server and produced both
`package/dist/index.js` and `package/dist/index.d.ts`.
- `node --test scripts/__tests__/release-verify-workflow.test.mjs` (8
tests)
- `pnpm -r typecheck`
- `pnpm build`
- `pnpm check:token-gates`
- `git diff --check`
- No `pnpm-lock.yaml` diff.
- The diff changes 12 files.
## Risks
The runner adds Rust work to the existing Build jobs. These jobs can
take longer on a cold cache. The pinned toolchain makes contributor and
CI behavior reproducible. Cargo tests use `--release` to verify
optimized executables. The server prepack lifecycle now performs the
build that its published entry points require. This can make direct
server packing slower. This pull request does not wire runnerd into the
server. It does not select runnerd for any adapter. Existing application
execution and finalization paths remain unchanged.
## Model Used
OpenAI Codex with GPT-5. Agentic coding mode used repository tools, code
execution, and automated tests.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
Split the slow workspaces-a CI lane into two Vitest native shards and keep release verification in parity.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip uses CI to keep control-plane changes safe and mergeable.
> - The PR workflow splits serialized server tests across isolated
runners.
> - A recent successful run spent 305 seconds in serialized shard 2/4.
> - That job was the slowest check in the run.
> - The four shards reported about 739 seconds of Vitest suite time.
> - This pull request adds a fifth serialized shard and keeps release
verification aligned.
> - The benefit is a shorter PR critical path with no loss of test
coverage.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The PR and release verification workflows run serialized server tests in
four shards.
**Current behavior**
Successful PR run 30876682788 spent 305 seconds in `Verify serialized
server suites (2/4)`. The test step used 256 seconds and made this job
the slowest check.
**Proposed behavior**
Run the same serialized suite set in five complete and non-overlapping
shards.
**Reason and benefit**
The measured suites reported about 739 seconds of total Vitest time.
Five runners reduce the expected average suite time from about 185
seconds to about 148 seconds before setup overhead.
**Breaking changes**
None. The change only alters CI partition size.
## What Changed
- Split serialized server tests into five shards in the PR workflow.
- Apply the same five-shard layout to release verification.
- Add a partition test that proves complete and non-overlapping
serialized coverage.
- Update release workflow coverage tests for five shards.
## Verification
- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`
- `git diff --check`
## Risks
- Low risk. CI uses one additional runner for the serialized lane.
- Round-robin partition weights can still vary as suite timings change.
> This change does not overlap with planned core work in `ROADMAP.md`.
Related PR #10663 optimized the separate general-server lane.
## Model Used
- OpenAI Codex, GPT-5, agentic coding with reasoning, tool use, and code
execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Releases publish the same app and package set that operators
install, so release verification should keep full release-strength
coverage.
> - The release workflow currently verifies stable and canary releases
with one serial job that typechecks, runs all tests, and builds.
> - The PR workflow already proves the test surface can be split into
grouped general suites and serialized shards without changing coverage.
> - This pull request extracts the release verify work into a reusable
workflow and fans out the independent lanes.
> - The benefit is faster stable and canary release verification while
preserving the existing publish and preview gates.
## Linked Issues or Issue Description
No public GitHub issue exists for this CI improvement.
**Subsystem affected**
Cross-cutting (multiple of the above)
**Problem or motivation**
Release verification spends most of its wall time in a single serial
test step even though the same stable test surface is already
partitioned for PR CI. Stable dispatches and master-push canaries
therefore wait on one long runner after setup, typecheck, tests, and
build run sequentially.
**Proposed solution**
Add a reusable release verification workflow with parallel typecheck,
grouped general tests, serialized test shards, and build lanes. Have
both stable and canary release verification call it with the ref they
need to verify.
**Alternatives considered**
Keeping the serial `pnpm test:run` job preserves the old shape but keeps
stable and canary releases waiting on one long runner. Skipping
verification when a source SHA already has green CI would be faster, but
adds stale-check and lookup risk beyond this change.
**Roadmap alignment**
No overlapping item found in `ROADMAP.md`; this is release CI
maintenance.
**Additional context**
The new workflow keeps the release-strength full `pnpm -r typecheck`,
uses the existing stable test grouping/sharding entry points, and leaves
publish/preview jobs unchanged.
## What Changed
- Added `.github/workflows/release-verify.yml` as a `workflow_call`
workflow accepting a `ref` input.
- Split release verification into parallel `typecheck`, `general_tests`,
`serialized_tests`, and `build` jobs with 20-minute lane timeouts.
- Mirrored the PR workflow's stable test partition: `general-server`
shards 1-3, `general-workspaces-a`, `general-workspaces-b`, and four
serialized shards.
- Replaced `release.yml` `verify_canary` and `verify_stable` job bodies
with calls to the reusable workflow while leaving publish and preview
jobs unchanged.
- Added a Node test that guards the release workflow delegation and
split verify surface.
## Verification
- `actionlint 1.7.12 .github/workflows/release.yml
.github/workflows/release-verify.yml`
- `node ./scripts/release-package-map.mjs check`
- `node --test ./scripts/__tests__/release-verify-workflow.test.mjs
./scripts/__tests__/run-vitest-stable-shard.test.mjs`
- `git diff --check`
## Risks
- Release verification now starts more jobs per release event,
increasing total runner setup/install minutes. This matches the existing
PR CI tradeoff and should reduce release wall time substantially.
- The called workflow checks out the requested ref shallowly. That is
intentional for verify lanes; publish and preview jobs still retain
their existing full-history checkouts.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex, GPT-5-class coding agent in local tool-use mode with shell
execution, repository editing, GitHub connector access, and medium
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Devin Foley
Nicky Leach
Dotta
dependabot[bot]