## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud needs a verified image for each merged source
commit.
> - Fresh builders restore compiled native dependencies from registry
caches.
> - The current workflow imports up to eleven historical cache manifests
at once.
> - Live builds missed native layers that a fresh builder reused from
one manifest.
> - This PR selects the nearest available cache and tests reuse across
fresh builders.
## Linked Issues or Issue Description
Refs #13329 and #13330. A search of open cache PRs found no duplicate of
this change.
**What existing behavior does this improve?**
Remote Docker cache reuse on fresh Cloud image builders.
**Current behavior**
[Cloud run
34714483272](https://github.com/paperclipai/paperclip/actions/runs/34714483272/job/103609096836)
imported the previous cache manifest successfully but rebuilt
`cargo-chef` and Rust dependencies. The dependency compile took 3m43s.
The preceding image build had already exported those layers.
A controlled [fresh-builder
diagnostic](https://github.com/paperclipai/paperclip/actions/runs/34715336530)
used the same source and registry cache. The single-manifest job reused
both layers immediately. The multiple-manifest job rebuilt them and
failed the cache assertion. Both jobs used GitHub-hosted runners with
read-only access.
**Proposed behavior**
Inspect cache manifests in first-parent order and import only the
nearest available one. Keep full-SHA cache exports, the ten-commit
search bound, and the legacy fallback. If caches cannot be read, permit
a cold build.
**Reason and benefit**
Avoid the observed cache misses without changing image contents or
builder sizes. Expected savings include about four minutes of native
tool/dependency compilation when those inputs are unchanged. The final
merge-to-deployable gain still needs a post-merge measurement.
**Breaking changes**
No image, artifact, deployment, or runner-routing contract changes.
## What Changed
- Select one available ancestor cache after Docker login and Buildx
setup.
- Preserve separate writable cache tags for each full source SHA.
- Test cache ordering, missing caches, registry errors, and workflow
integration.
- Add the selector tests to the existing release-registry suite.
- Export a local test cache, remove the first builder, and verify a
source rebuild on a fresh builder.
- Document cache selection and the stronger Docker check.
## Verification
- Passed 456 focused workflow, routing, readiness, preview-artifact, and
cache-selector tests.
- Passed shell syntax, ShellCheck for the changed probe, actionlint
workflow validation, and `git diff --check`. actionlint's shell checks
were disabled for the workflow validation because unchanged
migration-label commands trigger existing SC2012 notes.
- The fresh-builder registry diagnostic proves the single-cache
behavior. The [permanent two-builder probe
passed](https://github.com/paperclipai/paperclip/actions/runs/34715771048/job/103612624090),
including a changed real binary and dependency-declaration invalidation.
- Passed all 35 latest-head checks (green or intentionally skipped),
including full typecheck, test, build, and browser suites in [PR CI run
34715771217](https://github.com/paperclipai/paperclip/actions/runs/34715771217).
- The real selector CLI inspected registry metadata and chose the
nearest available ancestor cache.
- Fresh Greptile review is 5/5 with no open findings. The PR title was
corrected to meet the source-change naming rule; the review check passed
after that correction.
- Local full-suite runs and Docker builds are unavailable because the
local Docker daemon is unresponsive after disk exhaustion. CI provides
the Linux verification.
## Risks
- Missing or unreadable caches cause a slower cold build. The selector
logs that condition and preserves image publication.
- Inspecting several missing ancestors adds lookup time. Each lookup has
a ten-second timeout and the search is bounded.
- The Docker test now exports a local cache. It removes the first
builder before starting the second to release disk space, then cleans up
its builders and files.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud consumes versioned readiness markers for each merged
source commit.
> - Those markers can be published only after the required checks and
artifacts pass.
> - The marker jobs currently wait for the same AWS runner capacity as
builds and tests.
> - A busy builder pool can delay readiness after all required work has
finished.
> - This PR moves the small readiness jobs to GitHub-hosted runners
while retaining every dependency gate.
## Linked Issues or Issue Description
Refs #13326 and #13328.
**What existing behavior does this improve?**
Time from completed Cloud verification to a deployable marker, and AWS
capacity occupied by artifact polling.
**Current behavior**
In [Cloud readiness run
34711557083](https://github.com/paperclipai/paperclip/actions/runs/34711557083),
all builds and tests finished at 18:43:05 UTC. The source marker did not
start until 18:44:21, and the deployable marker did not start until
18:44:37. Merge-to-deployable was 11m33s, although the prerequisite work
finished in 9m44s.
**Proposed behavior**
Run the artifact wait and both versioned marker jobs on `ubuntu-latest`.
Keep the compute jobs on the approved post-merge AWS fleet.
**Reason and benefit**
Avoid builder-pool queue delays after verification finishes. This also
removes the long artifact-wait job from AWS capacity. Expected savings
depend on queue depth: the observed run had over 90 seconds of avoidable
marker waiting. The marker commands themselves take only seconds.
**Breaking changes**
Runner placement changes for three bookkeeping jobs. Marker names,
exact-source artifact checks, required verification, and image
verification stay the same.
**Additional context**
Searched the related runner and Cloud readiness work. This addresses
queue time observed after the parallel verification change.
## What Changed
- Place the artifact wait, source-verification marker, and deployable
marker on GitHub-hosted runners.
- Keep all existing job dependencies, source guards, permissions, and
commands.
- Extend routing regressions to enforce this placement and retain
fail-closed readiness gates.
- Document why readiness bookkeeping uses separate runner capacity.
## Verification
- Passed 433 workflow, routing, source-verification, and Cloud readiness
tests with `node --test .github/scripts/tests/*.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/cloud-readiness.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`.
- Passed `actionlint` and `git diff --check`.
- Passed all latest-head CI gates in [run 34712624340, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/34712624340),
including typecheck, build, browser, Runner, and all general/serialized
tests.
- Attempt 1 had one localhost readiness timeout in an unchanged test. A
single targeted retry passed all 166 files (3,225 tests passed, one
existing skip), including all seven tests in that file. No timeout,
assertion, or application source was changed; the retry is documented in
the PR comment.
- Local full-suite verification is limited by local disk exhaustion; the
focused checks above pass.
- Fresh Greptile review is 5/5 with no unresolved findings.
## Risks
- GitHub-hosted capacity can also queue, but these jobs no longer
compete with AWS build/test demand. The change does not reserve
instances or change box sizes.
- Readiness must still fail if any prerequisite fails. The existing
`needs` relationships and success-only execution are preserved and
tested.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images that contain the native Rust Runner.
> - The image already builds that Runner before copying ordinary app
source.
> - A Rust source change still invalidates its entire compiled
dependency layer.
> - Compiled dependencies can survive source changes when their recipe
is unchanged.
> - This PR adds a separate locked dependency build before compiling the
real workspace.
## Linked Issues or Issue Description
Refs #13195. A search of related Docker and Cargo cache PRs found no
duplicate dependency-recipe change.
**What existing behavior does this improve?**
Docker image build time after Rust source or embedded protocol changes.
**Current behavior**
The `runner-build` stage compiles dependencies and workspace code in one
layer. In Cloud readiness run 34698143548, that stage took about 3m48s
when its cache was unavailable.
**Proposed behavior**
Generate a recipe with pinned cargo-chef 0.1.73. Build locked release
dependencies in `runner-deps`, then copy and compile real Rust source
and embedded protocol inputs in `runner-build`. Source edits can reuse
the dependency layer from the existing registry cache.
**Reason and benefit**
Reduce dependency recompilation during source changes and merge bursts.
Expected savings are roughly 2–4 minutes when the old native layer would
miss but dependency layers are available. Full cold builds also pay for
the recipe tool installation. Ordinary app-only cache hits gain little
from this change.
**Breaking changes**
None to the shipped application or image tags. The recipe tool and
compiled dependencies remain in build stages.
## What Changed
- Install a pinned recipe generator with its locked dependencies and the
existing package-owned compiler.
- Add recipe planning and compiled dependency stages. Use the same
release profile, package, binary, and lockfile enforcement as the real
native build.
- Remove generated source stubs before copying actual source. Preserve
protocol inputs, timestamp normalization, binary staging, and
application checks.
- Add Docker cache wiring regressions and update the Docker cache
documentation.
- Run a two-build probe in Docker Runner check. It requires dependency
reuse, changed real binary metadata after a source edit, and a changed
recipe after a dependency declaration edit. It uses a disposable
tracked-source context and exports only small metadata files.
## Verification
- Passed all five Docker build-stamp and dependency-cache tests with
`pnpm exec vitest run server/src/__tests__/docker-build-stamp.test.ts`.
- Passed the local ARM64 `docker buildx build --target runner-build
--progress plain`. Local Docker then hit storage errors during a runtime
probe; cache invalidation verification continues on GitHub-hosted Linux.
- Passed `bash -n scripts/check-docker-runner-cache.sh`, `actionlint`,
and `git diff --check`.
- Passed a [Linux AMD64 cache
probe](https://github.com/paperclipai/paperclip/actions/runs/34711042199)
against the PR source: dependencies compiled in 3m49s for the baseline
and were `CACHED` after a source edit; real source compilation took
about 37 seconds. Binary metadata changed and dependency declaration
changes altered the recipe. The permanent probe is also running in
latest-head Docker Runner check.
- Passed latest-head [Docker Runner
check](https://github.com/paperclipai/paperclip/actions/runs/34711145160),
including the permanent source/dependency invalidation probe.
- Passed full [PR
verification](https://github.com/paperclipai/paperclip/actions/runs/34711145352/attempts/2):
typecheck, all grouped tests, native verification, build, release dry
run, and browser checks. One unrelated signoff-policy browser test
failed waiting for a heartbeat run on attempt 1; only that failed shard
and dependent checks were retried, and passed.
- Latest-head Greptile is 5/5 with no unresolved findings. Full local
tests/build were limited by local disk exhaustion; Linux CI completed
those checks.
## Risks
- The two-build CI probe has a 20-minute job limit to cover the cold
build and source rebuild. It adds no AWS routing.
- A fully cold build must install cargo-chef and populate the dependency
layer. Both become reusable registry layers; no Actions cache is added.
- The recipe and final build must keep the same compiler, build profile,
package, binary, and directory layout. A source-change rebuild probe
checks real cache reuse and binary invalidation.
- Dependency or compiler changes still require rebuilding dependencies.
Existing image verification and full-SHA publication gates remain
unchanged.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys verified images from merged source commits.
> - Cloud readiness waits for every release verification check.
> - Runner verification currently runs long TypeScript tests before Rust
checks.
> - These checks can run on independent runners with their own build
directories.
> - This PR runs them in parallel while preserving all checks and the
shared dependency cache.
## Linked Issues or Issue Description
Refs #13194. Related prior work: #13142 and #13259. A search found no
duplicate parallel release-check change.
**What existing behavior does this improve?**
Time from merge to Cloud source verification and deployment readiness.
**Current behavior**
Recent successful runs take roughly 13 minutes from merge to deployable.
In run 34705914878, Runner verification took 11m23s. Protocol tests
finished before Rust tests and API authority checks started.
**Proposed behavior**
Run protocol and Rust verification in two matrix jobs. Cloud readiness
still requires both jobs to pass.
**Reason and benefit**
Remove the serial dependency between independent checks. Expected
improvement is about 2–3 minutes on a typical cached run, until the
image build or server tests become the longest job. This is an estimate;
post-merge timing will confirm it.
**Breaking changes**
Individual release Runner job names gain a lane suffix. Cloud source and
readiness marker names stay the same. PR runner routing is unchanged.
## What Changed
- Split release Runner checks into protocol and Rust lanes. Keep every
constituent of `check:all` exactly once.
- Restore the existing Rust dependency cache in both lanes. Allow only
the Rust lane to save it after warming both build profiles.
- Add coverage and cache authorization regressions. Document the
parallel verification and single cache writer.
## Verification
- Passed 477 workflow and source-verification tests with `node --test
.github/scripts/tests/*.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`.
- Passed `actionlint`, `git diff --check`, and the private AWS routing
regression suite.
- Passed local `pnpm -r typecheck` and the standalone `check:runner &&
check:api-authority` lane, including all 1,671 API tests before the
protocol lane had built TypeScript output.
- The broad local protocol run under Node 25 had four failures. The two
affected files passed under CI's Node 24.19.0: 67 passed, 6 platform
skips.
- Local `pnpm test:run` aborted when disk space ran out; local `pnpm
build` could not run afterward. These are local verification limits.
[Linux CI run
34710421424](https://github.com/paperclipai/paperclip/actions/runs/34710421424)
passed full typecheck, all grouped tests, native verification, build,
release dry run, and browser checks. Native protocol CI passed 1,986
tests, plus 1,671 API tests and the Rust suites.
- Latest-head Greptile is 5/5 with no open findings. All 33 current-head
checks are successful or intentionally skipped.
## Risks
- Uses one additional short-lived verification runner per release
verification. The existing AWS exact-master restriction remains in
place.
- The Rust lane warms debug dependencies so its cache save also serves
protocol tests. Both lanes always rebuild workspace code.
- A workflow regression could omit a check. The new coverage test
compares the matrix checks directly with `check:all`; Cloud readiness
depends on the complete reusable workflow.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Conversations must use the same tasks, controls, and execution
history.
> - Users need an ongoing chat with an agent without managing task
properties.
> - Agents should clarify and plan work, then hand execution to assigned
project tasks.
> - This pull request combines the reviewed Agent Chat stack for one
squash merge.
> - The benefit is persistent conversation with normal task governance
and shared UI.
## Linked Issues or Issue Description
**Subsystem affected**
Task lifecycle, agent runtime tools, shared task UI, and browser/paid
runner tests.
**Problem or motivation**
Users need one persistent conversation with each agent. A separate chat
store or renderer would duplicate task behavior and bypass existing
controls.
**Proposed solution**
Use a task-backed chat per company, user, and agent. Reuse the task
composer and transcript. Clarify and plan in chat, then create assigned
project tasks with the relevant plan. Keep Agent Chat behind its own
disabled-by-default experimental setting.
**Roadmap alignment**
This implements the task-backed direction in [CEO
Chat](https://github.com/paperclipai/paperclip/blob/master/ROADMAP.md#-ceo-chat).
Related proposals: #2504 and #9693. Related request: #7981. The
maintainer requested one squash merge of the complete stack.
Consolidates the reviewed runtime
[#13281](https://github.com/paperclipai/paperclip/pull/13281), backend
[#13282](https://github.com/paperclipai/paperclip/pull/13282), and UI
[#13283](https://github.com/paperclipai/paperclip/pull/13283) layers
with this PR's E2E coverage. All four layers passed CI and received
Greptile 5/5 before consolidation. This PR targets master and includes
the complete feature.
## What Changed
- Add personal canonical chat tasks with ordinary company visibility,
immutable identity, idempotent first sends, and an idle waiting state.
- Process `/new` in queue order. Preserve history, release a chat pause,
and fence old provider context and delayed writes.
- Keep chat lifecycle rules across recovery, finalization, assignment,
task lists, and rollups.
- Support research and plan revision in chat. Hand plans to ordinary
assigned project tasks before execution starts. Reject new chat
subtasks.
- Add repository-aware project creation and discovery tools, including
multiple repository IDs and GitHub URLs, authorization, idempotency, and
durable project-created cards.
- Reuse task UI components for chat, with starred/recent agent
navigation and a separate `enableAgentChat` experimental flag.
- Add deterministic browser tests and 24 paid chat cells across four
Codex/Claude profiles, with validated reports and screenshots.
- Integrate current master recovery, controller lease, queued-message,
and task UI changes. Gate chat interruption and deferred promotion on
ownership/feature policy. Guarantee lease renewal and active controls
are stopped even if teardown fails.
- Preserve master's migration 0273 and generate chat migration 0274 with
idempotent replay for development databases.
## Verification
- Prior exact heads of all four PRs passed Linux CI, including build,
typecheck, general/serialized tests, and browser E2E. Each had Greptile
5/5 and no unresolved findings.
- Integrated local verification passed: full repository typecheck and
production build, Storybook build, token gates, 340 focused UI tests,
all 20 deterministic chat browser tests, two migration replay tests, 88
focused chat/queue/native/controller tests, and provider/session
regressions including real lease expiry. These include the three
lifecycle regressions for the final admission/teardown fixes; server
typecheck also passes. Current head
`1268eda16cc2af892055917e7292f068820be135` has Greptile 5/5 with no
unresolved findings and passing security scans. All final-head CI gates
passed: build, full Runner verification, typecheck/release registry,
canary, all general/serialized test shards, and all browser E2E shards
([CI
run](https://github.com/paperclipai/paperclip/actions/runs/34696739927)).
Local PostgreSQL startup contention required serialized retries; skipped
fixtures do not count as passing coverage.
- The earlier paid campaign passed all 24 chat cells and retained 32
screenshots:
[report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-34648511170-1/index.html?report=agent-chat#suite-agent-chat).
It tested `abacbdfd2f660709ec37312cdb758284c8399d04`; it is prior
evidence, not a paid run of this integrated head.
- Manual check: enable Agent Chat in Experimental settings, open an
agent, clarify and revise a plan, then hand off to an assigned project
task. Stop a reply, send `/new`, and verify fresh context with retained
history. Disable the setting and verify agent shortcuts/new chat turns
are blocked.
## Risks
- Queue/session integration can affect retries and delayed writes. Tests
cover ownership, cancellation, reset boundaries, idle recovery, and
ordinary task behavior.
- Migration 0274 adds conversation fields and constraints. Replay is
idempotent and preserves existing development chat history.
- This combines the previously reviewed stack at the maintainer's
request. Agent Chat remains off by default and is separate from
Conference Room.
## Model Used
OpenAI Codex, GPT-6 Astra (`gpt-6-astra`), with reasoning, code
execution, browser tools, and parallel review. The exact context-window
size is not exposed in this session. Codex and Claude also ran as test
subjects in the linked paid campaign.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - PR checks share a repository cache budget with post-merge cloud
verification.
> - Per-PR dependency store copies consume about 700 MB each and
displace useful build caches.
> - The reviewed workflow now restores shared dependency caches without
saving PR copies.
> - The public entry point must pin an approved immutable workflow
before it can use AWS runners.
> - This PR activates the merged workflow after the runner group
accepted its exact SHA.
> - Native verification and the app build also run in parallel, as
defined in the merged workflow.
## Linked Issues or Issue Description
Refs #13300. Refs #13301.
The cache definition is merged, and its exact SHA is authorized in the
restricted runner group. This PR activates it. Its base also contains
the two synthetic fixture fixes from #13301. No duplicate activation PR
was found.
## What Changed
- Advance the `pr-trusted.yml` pin from
`03609aa6ecc9a047ed53d6b6469d8be554fbc46d` to merged commit
`44dde2dec42a22746a2f36b595acacc9ccfa1df6`.
- Update the adjacent pin comment to identify #13300 and the activated
behavior.
- Activate restore-only pnpm caching and removal of redundant Node setup
steps from #13300.
- Activate the already-merged separate native verification job, required
by the aggregate `verify` check. The app build no longer waits for
native verification inside the same job.
- Activate the merged module-boundary and source-verification checks and
explicit Runner Evalbook viewer build.
## Verification
- Passed all 505 workflow, routing, cache, sharding, and
source-verification tests.
- Passed `actionlint` for both workflow files and `git diff --check`.
- Passed the runner infrastructure's workflow routing regression tests.
- Confirmed that the gate is unchanged between the old and new workflow
definitions. The restricted runner group permits the exact merged SHA
and retains its existing workflow pins.
- Full workspace typecheck and build passed on the timeout-fix branch.
Its 107 targeted runner tests and all Linux CI groups passed. The broad
local Mac test command had 15 unrelated permission and
missing-skill-path failures, documented in #13301; it stopped before
later groups.
- [Current-head CI run
34662664879](https://github.com/paperclipai/paperclip/actions/runs/34662664879)
passed. All 33 checks are green or intentionally skipped at
`ded4f1d093644b9279cdf56457f69a196fcf8cc2`. Greptile is 5/5, with no
unresolved findings.
- The live build restored the master pnpm cache, downloaded the resolved
lockfile artifact, and completed a frozen install. It had no
dependency-cache upload step. GitHub reports zero cache entries for this
PR. Native verification and the app build started together and both
passed.
- [Post-merge cloud readiness for
#13301](https://github.com/paperclipai/paperclip/actions/runs/34662389233)
passed all 30 jobs in 12m42s from merge. Native verification restored
the master Rust cache and passed the formerly failing fixtures.
## Risks
A new PR-only dependency can require a download until a master cache
contains it. The new pin also activates the merged workflow changes
listed above, so the live PR must pass all checks. The repository
storage cap is still 10 GB; this PR does not increase it. No allowlist
or runner permission logic changes.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud releases wait for source verification before deployment.
> - That verification reuses compiled Rust dependencies to finish
sooner.
> - PR jobs save large pnpm stores under separate merge refs and
different lockfile keys.
> - Those copies compete with master build caches for the repository's
10 GB cache limit.
> - This PR makes PR dependency caches restore-only and reuses
master-compatible keys.
> - A separate pin update will activate the reviewed workflow.
## Linked Issues or Issue Description
**What happened?**
PR merge refs accumulated roughly 700 MB copies of the same pnpm store.
Master Rust caches disappeared, and Cloud readiness run
[34656098157](https://github.com/paperclipai/paperclip/actions/runs/34656098157)
rebuilt dependencies after cache misses. The repository currently has a
10 GB limit. GitHub rejected a request for 50 GB; that setting needs
separate organization/billing access.
**Expected behavior**
PR jobs should reuse downloaded packages without evicting post-merge
compilation caches through duplicate uploads.
**Steps to reproduce**
1. Run several PRs while the checked-in lockfile needs policy
regeneration.
2. Compare the setup-node keys in PR jobs and master jobs.
3. List Actions caches by ref, key, and archive size. The PR keys repeat
across merge refs.
**Paperclip version or commit**
f12b647ae, before this change.
**Deployment mode**
GitHub Actions, with GitHub-hosted and allowlisted AWS PR runners.
Refs #13267 (empty pnpm store prevention). Searched open issues and PRs
for pnpm cache duplication and found no duplicate implementation. This
change leaves the paused capacity documentation PR #13280 alone.
## What Changed
- Replace setup-node cache writes with pinned `actions/cache/restore` in
all seven PR install job definitions.
- Restore against the checked-in lockfile before downloading the
regenerated policy artifact. Keep every install frozen against that
artifact.
- Allow an OS/architecture-specific pnpm fallback and disable automatic
setup-node caching.
- Remove dependency-store caching from the resolution-only policy job.
- Add eight regression tests, update the existing stacked-lockfile cache
assertion, and document cache behavior and storage settings.
## Verification
- Passed 505 workflow, routing, cache, and source-verification tests:
`node --test '.github/scripts/tests/*.test.mjs'
scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs`.
- Passed `actionlint .github/workflows/pr-trusted.yml` and `git diff
--check`.
- The AWS routing gate is unchanged. Author, event sender, and rerun
actor must still be allowlisted.
- This definition PR does not change the active `pr.yml` pin. After
review and merge, authorize its immutable merge SHA additively and
activate it in a separate PR. Verify a populated restore and no cache
uploads in an allowlisted PR.
- All 32 checks passed or were intentionally skipped on
`44b31eca590f61b75cae646de43c491b6c4deae7`, including full native Runner
verification, application build, server/workspace tests, and browser
shards. Current-head Greptile is 5/5 with no findings. No application
code changes in this PR.
## Risks
- New dependencies present only in a PR may download again on each run
until master saves a cache containing them. Frozen installation remains
the source of dependency resolution.
- Missing or expired stores fall back to normal package downloads.
- Existing PR copies remain until expiry or a separate targeted cleanup.
No cache entries are deleted here.
- The workflow only takes effect after the separate immutable pin
rotation. Storage billing settings are not changed by this PR.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployment requires source verification for the exact merged
commit.
> - Contributor PRs leave lockfile updates to a separate bot PR.
> - Most release checks can refresh an outdated lockfile while
installing dependencies.
> - Two Runner checks still require a frozen lockfile and fail after
dependency changes.
> - This PR gives those checks the same install policy as the other
release checks.
> - A valid dependency change can become deployable without waiting for
another merge.
## Linked Issues or Issue Description
Refs #13257. The dependency change in #13256 exposed this gap. The
separate lockfile update is #13279. Related #12115 addresses the bot PR
check trigger; this PR fixes exact-source cloud verification itself.
**What happened?**
[Cloud readiness for
2083bf6](https://github.com/paperclipai/paperclip/actions/runs/34651761811)
failed in the Runner scorer and chaos jobs with
`ERR_PNPM_OUTDATED_LOCKFILE`. The commit added `svix` to server
dependencies. The tracked lockfile still describes the previous
manifest. The other release checks install with `--no-frozen-lockfile`.
**Expected behavior**
Every source check installs and tests the same checked-out commit. A
pending bot lockfile PR must not block cloud readiness.
**Steps to reproduce**
1. Check out master commit 250deab, which retains the manifest/lockfile
mismatch.
2. Run `pnpm install --ignore-scripts --frozen-lockfile`. It fails with
the same outdated-lockfile error.
3. Run `pnpm install --ignore-scripts --no-frozen-lockfile
--resolution-only`. It succeeds.
4. Restore the generated lockfile. This PR does not commit it.
## What Changed
- Use `--no-frozen-lockfile` in the release Runner scorer job.
- Use the same option in the reusable Runner chaos workflow.
- Document why cloud source checks allow a job-local lockfile refresh.
- Update the existing Runner scorer workflow assertion to match its
install policy.
## Verification
- All 457 workflow tests pass across `.github/scripts/tests/*.test.mjs`
and `scripts/__tests__/release-verify-workflow.test.mjs`.
- `actionlint` passes for both changed workflows.
- Reproduced the frozen install failure against the real tracked
manifest and lockfile. The refresh command passes in 4.6 seconds.
- `git diff --check` passes. No lockfile changes remain.
- No application source changes. Full local application typecheck,
build, and test commands were not rerun in this dependency-free workflow
worktree. Current-head GitHub CI must pass before merge.
- After merge, verify both affected jobs pass on the exact master source
even if the lockfile bot PR remains pending.
## Risks
pnpm can resolve allowed dependency ranges when a manifest outgrows the
tracked lockfile. This matches the existing release install policy. The
resulting lockfile stays in the job workspace. Verification commands and
runner routing are unchanged. The security reviewer explicitly accepted
this existing dependency-policy tradeoff for both jobs after reviewing
repository policy and the source/authorization checks. A future shared
immutable dependency artifact would improve reproducibility across jobs.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] Local verification passes: all 457 workflow tests, actionlint, and
the stale-lockfile reproduction described above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI installs dependencies before it verifies and builds cloud
artifacts.
> - Install jobs share a pnpm package-store cache with lockfile refresh.
> - Lockfile refresh resolves versions without downloading packages.
> - That job saved an empty cache before full install jobs could save
theirs.
> - This PR prevents lockfile refresh from publishing that empty entry.
> - Full install jobs can then populate the cache and reuse
dependencies.
## Linked Issues or Issue Description
Refs #13259 for the related cloud verification cache work. No duplicate
empty-cache fix was found.
**What happened?**
Refresh Lockfile run 34517514932 saved a 216-byte default-branch pnpm
cache at 18:58:08 UTC on September 10. Full install jobs still restore
that empty entry. The cache API reports 216 bytes for master and about
703 MB for populated entries with the same key and cache version in PR
scopes.
**Expected behavior**
A job that installs dependencies should populate the shared
package-store cache.
**Steps to reproduce**
1. Run lockfile refresh with a new lockfile cache key.
2. Its resolution-only command leaves the package store empty.
3. The Node action saves the empty archive before a full install
finishes.
4. Later jobs report a cache hit but download packages again.
**Paperclip version or commit**
Observed on master 6728e133f8 and still
present at a23ae894a5.
**Deployment mode**
GitHub Actions cloud verification and release workflows.
## What Changed
- Disable package-manager caching in Refresh Lockfile.
- Document how to remove the existing empty default-branch entry and
verify a populated replacement.
- Add regression coverage for explicit and automatic package-manager
cache selection in a resolution-only job.
## Verification
- actionlint and git diff checks pass.
- All 175 existing workflow-script tests pass. Both new regression cases
pass and fail against the original workflow, covering the explicit pnpm
cache and automatic npm cache paths. This change adds no application
behavior.
- [The cache creator
job](https://github.com/paperclipai/paperclip/actions/runs/34517514932/job/103006542158)
logs a 216-byte upload under the same key still used by cloud
verification.
- The batch-wide local full typecheck and build passed. The local full
test run reported 10,600 passed, 65 skipped, and 13 permission failures
in unchanged runtime-skill suites. These checks were not repeated in
this dependency-free worktree. Current-head Linux CI passes. The
unchanged chat and browser suites passed on their single retry; all
final checks are green. Greptile is 5/5 with all threads resolved.
- After merge, delete only the existing empty master cache entry. Verify
that a master install saves a populated archive and subsequent jobs
reuse packages. Measure the net install-time change before claiming a
latency gain.
## Risks
- Lockfile resolution can require fresh registry metadata. It does not
need a cached package store.
- The existing empty cache must be removed once; this change prevents
its recreation by this workflow.
- Cache benefits vary with download speed and archive extraction time.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused workflow checks
pass; the batch-wide local test limitation is disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployments consume a verified image and exact-source
migrator.
> - An image alone is not deployable until source checks and artifact
checks pass.
> - GitHub-hosted queues delayed those checks and the final readiness
signal.
> - This PR gives trusted master work a separate concurrency allowance
on existing AWS runners.
> - Community PRs and arbitrary source inputs keep the GitHub-hosted
fallback.
## Linked Issues or Issue Description
Refs #13243.
**What existing behavior does this improve?**
Time from a master merge to the Cloud deployable v1 signal.
**Current behavior**
For merge d0b7ba4, the image was available after 7m 42s, but readiness
took 16m 05s. Typecheck queued for 6m 40s and the final readiness job
queued for 1m 46s.
**Proposed behavior**
Allow up to 36 concurrent post-merge verification and migrator jobs on
the existing four-vCPU, 16-GiB AWS runners. Workers launch on demand and
terminate after their job; no always-on worker pool or AWS Reserved
Instance purchase is introduced. Keep the combined runner ceiling
unchanged. A separate operator switch enables this route only after the
restricted runner group and Fleet exist.
**Reason and benefit**
Remove GitHub-hosted queue delays from the cloud deployment path. The
gain depends on queue pressure and which remaining job finishes last;
the observed queues are not additive savings.
**Breaking changes**
None to source verification or readiness contracts. Paid routing is
limited to canonical master push/manual events, with exact source checks
on reusable and migrator jobs.
## What Changed
- Route source verification, artifact waiting, dispatch, and readiness
jobs to the separate post-merge Fleet when enabled.
- Require source inputs to match the event's master SHA. Preview inputs
and raced older migrator dispatches stay GitHub-hosted.
- Keep npm publication on GitHub-hosted runners for trusted publishing.
- Bound AWS job timeouts below the 45-minute instance lifetime.
- Document activation, capacity reservation, and rollback.
- Exercise each actual runner selector against allowed and rejected
event/source combinations.
## Verification
- 268 routing and timeout cases pass, including unapproved PR, fork,
branch/tag, arbitrary ref, and disabled-switch cases.
- All 461 focused workflow, preview, and readiness tests pass. The 284
routing/preview cases also pass after the review fixes.
- actionlint passes for changed workflows with the existing
SC2012/SC2016/SC2129 warnings excluded.
- Full local typecheck and build pass (167s and 206s). `pnpm test:run`
completed: 10,600 passed, 65 skipped, and 13 failed in the unchanged
company-skills-service/runtime-skill-cache suites with local filesystem
permission errors. Linux CI is the required test gate; this is not a
claim of a fully passing local suite. Current-head Linux CI is green,
Greptile is 5/5, and all findings are resolved. The final Build retry
passed on a verified 60 GiB AWS runner after correcting the earlier
disk-capacity failure.
- After activation, verify a master run selects the separate group and
all readiness prerequisites pass.
## Risks
- A missing or incorrectly restricted runner group can leave eligible
jobs queued. Enable the switch only after Fleet and group verification.
- PR bursts have 64 slots after reserving 36 for post-merge work. The
image Fleet retains eight, for the same 108-runner total.
- A migrator dispatch racing a newer merge uses GitHub-hosted runners.
This preserves source trust but can retain some queue delay.
- Roll back placement by disabling AWS_POST_MERGE_CI_ENABLED and
rerunning the whole workflow.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud images become deployable only after source verification
passes.
> - The typecheck job builds the native Runner binary through the server
package.
> - Fresh runners repeatedly compile Rust dependencies for that binary.
> - This PR caches those dependencies for exact-source master
verification.
> - Workspace code and every typecheck still rebuild or run as before.
## Linked Issues or Issue Description
Refs #13243 and #13257.
**What existing behavior does this improve?**
The typecheck portion of post-merge cloud source verification.
**Current behavior**
The typecheck job has no Rust dependency cache. An observed release
build in this job took 4m 13s, including dependency compilation.
**Proposed behavior**
Restore dependency build outputs for canonical master pushes with the
exact source SHA. Use a separate cache key from the Runner verification
job, which builds other profiles.
**Reason and benefit**
A warm cache should remove roughly 2–3 minutes of dependency compilation
from this job. Overall deployment gains depend on the remaining critical
path. The first cache population still compiles from scratch.
**Breaking changes**
None. All checks remain enabled. Non-master callers compile without
restoring or saving this cache.
## What Changed
- Select the pinned Rust toolchain before the typecheck cache lookup.
- Reuse the existing pinned Rust cache action with a typecheck-specific
key.
- Exclude workspace crates and installed cargo executables.
- Test restore/save trust boundaries and document cache behavior.
## Verification
- All workflow script tests pass locally, including nine new cache
trust/contract cases.
- actionlint passes for release-verify.yml.
- Full local typecheck and build pass on the same source base (167s and
206s); `pnpm test:run` is still running and is recorded with #13257.
This PR changes only the workflow, cache guard tests, and documentation.
- All 32 current-head checks are successful or intentionally skipped,
including the complete Linux test matrix, build, and Greptile 5/5 with
no unresolved findings. Verify cache population and subsequent restore
on actual master runs.
## Risks
- The first run and any toolchain/dependency invalidation compile from
scratch.
- Cache restore/save overhead reduces the benefit for small dependency
graphs.
- Disable the cache step to roll back; the existing uncached build
remains valid.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. Exact serving model ID and context window are not exposed by
this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images built from master commits.
> - Cloud image builds share GitHub-hosted capacity with other
workflows.
> - The organization already operates AWS runners through RunsOn Fleet.
> - This pull request allows approved master builds to use a dedicated
cloud Fleet.
> - The benefit is separate build capacity with a quick operator
rollback.
## Linked Issues or Issue Description
Refs #13189, #13192.
**What existing behavior does this improve?**
Placement of the Docker cloud build after a master merge.
**Current behavior**
Every Docker cloud build uses a GitHub-hosted runner. Busy periods delay
the job.
**Proposed behavior**
An operator variable enables the approved cloud Fleet for canonical
master pushes and manual master builds. Other events, refs, and
repositories use GitHub-hosted runners.
## What Changed
- Add a guarded AWS runner selector to the Docker cloud job.
- Keep the existing image cache, verification, and publication steps.
- Test the selector against master, branch, tag, PR, fork, and disabled
contexts.
- Document provisioning requirements, placement checks, and rollback.
## Verification
- 29 focused Node tests pass for routing, readiness, and disk handling.
- The full workflow-script Node suite passes.
- `pnpm -r typecheck` passes locally.
- The pinned PR routing regression suite passes. The first live PR run
assigned 21 jobs to the approved AWS PR group. AWS then reclaimed 16
Spot instances. The failed run is being repeated on GitHub-hosted
runners while the Fleet moves to On-Demand.
- Actionlint passes with existing shellcheck findings excluded (SC2012,
SC2016, SC2129).
- `git diff --check` passes.
- Greptile reports 5/5 on commit
`764d505a41dd2023751c3f361906fa9ea35bf0c6`, with no review threads.
- All 30 current-head CI checks pass, including typecheck, build, all
server/workspace test shards, Runner verification, and browser tests.
Two Storybook checks are intentionally skipped for this change. Run:
https://github.com/paperclipai/paperclip/actions/runs/34630550799
- The broader local test/build sequence is still running. This Mac has
reported failures in unchanged application suites; their complete Linux
CI shards pass. Local targeted workflow tests and typecheck pass.
- Both On-Demand Fleets are deployed and healthy. Live master
cloud-build verification follows the merge.
## Risks
- Missing Fleet capacity or runner-group authorization can leave an AWS
job queued. Disable `AWS_CLOUD_BUILDS_ENABLED` and rerun the workflow to
use GitHub-hosted capacity.
- The runner group must restrict access to this repository and the
master version of `docker-cloud.yml`.
- Docker needs more disk space than the PR Fleet. Provision 120 GiB
disks and retain the free-space check.
- This changes image build placement only. Source verification and
migrator publication remain separate prerequisites.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact serving model identifier and context-window size
are not exposed by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Reuse the exact master source-verification result before npm canary publication, removing a duplicate verification matrix while preserving fail-closed release checks.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud waits for source verification before deploying a new image.
> - The slowest server verification job spends about ten minutes running
tests.
> - Each job uses one test worker to preserve test isolation.
> - This pull request distributes those suites across ten standard
hosted runners.
> - The benefit is a shorter verification path with the same test
coverage.
## Linked Issues or Issue Description
**Current behavior**
In [readiness run
34572340764](https://github.com/paperclipai/paperclip/actions/runs/34572340764),
the slowest server job ran for 638 seconds. Test execution used 594
seconds. This held readiness behind the image job.
**Proposed behavior**
Use ten general server jobs in the reusable release verification
workflow. Keep the three chat jobs and every existing prerequisite. The
complete partition test verifies that no server suite is omitted or
duplicated.
**Reason and benefit**
Reduce merge-to-deployable time on the existing runner type. The next
longest prerequisite was Runner verification at 526 seconds, so the
initial expected total gain is about two minutes rather than a halving
of readiness time. Measure actual queue and execution time before
claiming a result.
Related: #13198 introduced the separate chat lane. #12577 refreshes
duration estimates; this change leaves that manifest alone.
## What Changed
- Increase the general server matrix from five jobs to ten.
- Verify the ten-way partition covers the complete server suite when
combined with the chat lane.
- Document runner demand and the unchanged local and PR grouping.
## Verification
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs`: 29 passed.
- `actionlint .github/workflows/release-verify.yml`: passed.
- Full local `pnpm -r typecheck` and `pnpm build`: passed.
- All latest-head GitHub CI checks passed, including the complete Linux
test partition, build, typecheck, and browser gates. Greptile: 5/5 with
zero open findings.
- [Ten-shard timing
probe](https://github.com/paperclipai/paperclip/actions/runs/34606772388):
all 16 jobs passed; slowest server job 6m 23s versus 10m 38s in the
earlier five-shard sample. This compares the server lane, not total
readiness, and is not a controlled same-source A/B.
- The full local `pnpm test:run` is also running. It has reproduced
previously observed macOS-only failures in unchanged skill-cache and
native-session suites; the corresponding Linux CI suites passed. Final
local results will be attached separately. No affected-workflow test
failed.
## Risks
Five additional concurrent jobs per release verification run increase
runner demand and repeated setup work. Queueing can offset the gain.
Test workers, timeouts, permissions, and readiness requirements stay
unchanged. Revert the matrix and its partition test to restore the
previous split.
## Model Used
OpenAI GPT-6 / Codex, with reasoning, tool use, and code execution. The
exact serving model identifier and context-window size are not exposed
by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the affected workflow tests locally and they pass;
full-suite macOS limitations are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip helps people manage AI agents for work.
> - Maintainers use Storybook to review the board UI.
> - Reviews need public previews of selected repository branches.
> - Each branch needs its own URL so previews do not replace each other.
> - This pull request adds manual, CODEOWNER-controlled publishing to S3
and CloudFront.
> - The action returns stable branch links and permanent build links in
its summary and a Markdown artifact.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The existing Storybook build and manual visual-review workflow.
**Current behavior**
The repository has no manual branch-preview publisher. A single GitHub
Pages site cannot support independent publishers without combining their
output.
**Proposed behavior**
A CODEOWNER selects a source branch and approves publication. Each
branch has a stable CloudFront URL. A completed build becomes the branch
target only after its upload succeeds. The action attaches
`storybook-deployment.md` with the preview links and source commit.
**Reason and benefit**
Maintainers can share multiple branch previews at the same time. Branch
builds have no repository token permissions or AWS credentials.
Dependency caching and install hooks are disabled. The publisher cannot
write runner dashboard files or delete objects.
**Breaking changes**
None. Normal visual checks keep their existing behavior. This does not
change application code or GitHub Pages settings.
**Additional context**
Searched public issues and PRs for Storybook deployment work. No
duplicate deployment proposal was found. This is maintainer
infrastructure, not a roadmap-level core feature.
## What Changed
- Add `Storybook Deploy` with a source-branch input and a manual entry
through `Storybook Visual`.
- Check the original actor and rerunner against default-branch
CODEOWNERS. Require a protected deployment environment with CODEOWNER
reviewers.
- Separate public-source builds with no repository permissions from an
OIDC publisher restricted to the Storybook S3 prefix.
- Publish distinct branch URLs and retain build URLs. Preserve Storybook
deep links across the branch redirect.
- Add the run summary, a downloadable Markdown deployment report,
focused tests, and operator setup docs and IAM policies.
## Verification
- `node --test scripts/__tests__/storybook-deploy.test.mjs`: 19 tests
pass.
- `actionlint .github/workflows/storybook-deploy.yml
.github/workflows/storybook-visual.yml`: passes.
- [Feature branch live publication and deployment-only
rerun](https://github.com/paperclipai/paperclip/actions/runs/34533202273):
passed.
- [Master branch live
publication](https://github.com/paperclipai/paperclip/actions/runs/34533204743):
passed.
- Both public branch URLs render a component story without browser
errors. A deployment-only rerun updates only the selected branch entry
and preserves the previous build URL.
- AWS policy simulation allows Storybook uploads and denies dashboard
writes and object deletion.
- Full local typechecking passes. Full local tests, build, and
current-head PR checks are running.
- [Revised build and Markdown artifact
validation](https://github.com/paperclipai/paperclip/actions/runs/34605623088):
passed. Downloaded the report and verified its branch URL, build URL,
and source commit.
- The public verifier also checks that the stable branch URL points to
this build and rejects stale targets.
## Risks
- Storybook previews are public. Maintainers must publish only public UI
fixtures.
- Retained builds accumulate until an operator prunes them.
- Environment reviewers must stay synchronized with CODEOWNERS. The
workflow fails closed if its environment loses required protection.
- The existing CloudFront distribution is shared with runner reports.
Separate S3 prefixes and a dedicated role prevent the publisher from
overwriting those reports.
## Model Used
OpenAI GPT-6 via Codex, with reasoning, shell tools, and browser
verification. The exact runtime model ID and context-window size are not
exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Managed deployments start from the image built by the Cloud
workflow.
> - The managed runtime requests user and group 1001.
> - The image currently builds the node user as 1000.
> - Startup must remap that user, which can walk a large mounted home
directory.
> - This pull request uses the existing Docker build arguments to bake
user and group 1001 into Cloud images.
> - Matching the runtime identity removes that startup work and helps
avoid health-check retries.
## Linked Issues or Issue Description
Refs #13208, #1923, and #7861. Searched open and closed PRs for the
Cloud UID change. The older #7861 addresses build context and volume
ownership repair. This change uses the existing identity arguments in
the Cloud workflow and preserves ownership repair.
**What happened?**
A measured rollout had a container log `Updating node UID to 1001` after
startup. The container stayed at this step for at least 2 minutes 55
seconds before rollback stopped it. The baked node identity was 1000,
while the managed runtime requested 1001. A health check timed out and
the target required a second deployment attempt.
**Expected behavior**
Cloud images should already have the managed runtime identity. A
matching image should skip user and group remapping. Fresh or mismatched
volumes must still receive ownership repair.
**Steps to reproduce**
1. Build the current Cloud image with its default build arguments.
2. Start it with `USER_UID=1001`, `USER_GID=1001`, and a populated home
volume.
3. Observe the startup user remap before the application starts.
**Paperclip version or commit**
`fc06f7f05f42c675be71ff0927b6334405d520ed`
**Deployment mode**
Docker on managed hosts.
## What Changed
- Pass `USER_UID=1001` and `USER_GID=1001` to the Cloud image build.
- Check the pushed digest's baked identity before the entrypoint can
repair it. Then check the normal entrypoint's effective identity and
writable home before publishing the verified full-SHA tag.
- Add a workflow regression and two entrypoint cases for a matching
Cloud identity, including a mismatched volume.
- Document the runtime identity and the first-build cache cost.
## Verification
- Focused workflow and artifact tests: 27 passed.
- Entrypoint tests: 11 passed. Actionlint passed. Full local `pnpm -r
typecheck` passed. Full local `pnpm build` passed. The manual [Cloud
image
build](https://github.com/paperclipai/paperclip/actions/runs/34575473213)
passed on the exact PR head. It checked Sentry, baked and effective
identity, writable home, orphan reaping, and full-SHA publication. The
new identity check took one second. All 30 PR checks passed; the
Storybook workflow was intentionally skipped. Greptile reviewed commit
`114d408f637a0b53e2e2b1339c263779b1e4ae54` at 5/5 with no findings or
open threads.
- The full local suite for the same application source was already run
in #13205. Its macOS general-server phase had 10,471 passes and 70
failures in seven unchanged files. Those failures included missing
Runner fixtures, filesystem errors, timeouts, a port conflict, and a
load-count mismatch. After configuring Cargo and rebuilding fixtures, 37
of 38 native tests passed; one unchanged native-resume assertion still
failed. Linux PR CI passed. This change adds entrypoint tests and does
not change application code.
## Risks
- The first build must rebuild layers that depend on the base image
identity. Later builds can reuse them.
- A future managed runtime identity change must update these build
arguments and checks together.
- The Dockerfile's self-hosted defaults remain 1000. Runtime overrides
and mounted-volume ownership repair remain supported.
- The observed startup delay supports this change, but fleet timing also
includes provider startup, image pull, canary order, and retries. No
fixed end-to-end gain is claimed before a live rollout.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, code execution, and tool
use. The exact serving model ID and context-window size are not exposed
in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused workflow tests;
full-suite limitations are listed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployments require verified artifacts for the merged source
commit.
> - Cloud readiness and the npm release independently run the same
source checks.
> - Their shared chaos workflow used only the source ref as its
concurrency key.
> - One caller could cancel the other caller's required job for the same
commit.
> - This pull request scopes that key to the caller workflow and source
ref.
> - Both callers can finish their checks without blocking deployment
readiness.
## Linked Issues or Issue Description
Refs #13192 and #13205. Searched for related open issues and PRs; no
duplicate fix was found.
**What happened?**
The master push for `398d304e15739d1ee6105633bd8a0e42c929d33f` started
Cloud readiness and Release together. GitHub cancelled the Cloud
readiness chaos job before it acquired a runner. Its annotation reported
a higher-priority waiting request for the same concurrency group. The
required readiness gate cannot pass after that cancellation.
**Expected behavior**
Cloud readiness and Release must each finish source verification for the
same SHA. Standalone chaos evals must also have a separate group.
**Steps to reproduce**
Merge a commit to master while the npm release queue is empty. Both
callers reach the reusable chaos workflow with the same source SHA. See
[the cancelled
job](https://github.com/paperclipai/paperclip/actions/runs/34569569760/job/103168603926).
**Paperclip version or commit**
`398d304e15739d1ee6105633bd8a0e42c929d33f`.
**Deployment mode**
GitHub Actions on master.
## What Changed
- Add the caller workflow name to the chaos workflow concurrency group.
Retain source isolation and cancellation of duplicate calls within the
same workflow.
- Add a regression test that evaluates the group for Cloud readiness,
Release, and standalone evals at the same source SHA.
- Document the concurrency boundary in the readiness runbook.
## Verification
- `node --test scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs` passed: 26 tests.
- The new regression test fails against the previous concurrency key and
passes with this fix.
- `actionlint -shellcheck= -pyflakes=
.github/workflows/runner-chaos-evals.yml
.github/workflows/release-verify.yml
.github/workflows/cloud-readiness.yml` passed.
- `git diff --check` passed.
- The full local typecheck passed for the same application source in
#13205. Its macOS general-server test phase had 10,471 passes and 70
failures in seven unchanged application test files: missing Cargo/Runner
test binaries, filesystem permissions, timeouts, a port conflict, and a
load-test count mismatch. Linux CI test checks passed. The full local
build passed with Cargo on PATH. This PR changes workflow configuration,
its test, and documentation only.
- All CI checks pass on the final head, including typecheck, tests,
browser suites, build, and canary dry run. Greptile is 5/5 with no open
findings. After merge, verify both callers' chaos jobs complete for the
same master SHA and record the resulting readiness time.
## Risks
- Two callers may now run chaos tests at the same time. This uses two
existing GitHub runners, which is the intended cost of independent
verification.
- Renaming a caller changes its concurrency group. The fixed prefix
keeps this child group separate from caller-level concurrency groups.
- The readiness gate continues to require every verification
prerequisite. No gate is bypassed.
## Model Used
- OpenAI GPT-6 / Codex, with reasoning, repository editing, and
command/API tools. Exact serving model ID and context-window size are
not exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (26 focused
workflow/artifact tests)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
Verify source, build the cloud image, and wait for exact-source migrator packages concurrently. Emit Cloud deployable v1 only when every prerequisite succeeds for the merged full SHA.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Publish exact-source shared and database migrator packages for each master merge through the existing trusted Release workflow, independently of the full release and image build.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Skip cloud runner disk cleanup when both the Docker and workspace filesystems have at least 64 GiB free. Preserve the existing cleanup for low, unavailable, or invalid measurements and verify the actual shell behavior across eight scenarios.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Build cloud images independently for each master commit through a reusable workflow. Preserve production release dependencies and image runtime checks, and write cloud registry caches per commit with bounded ancestor imports to prevent overlapping builds from replacing each other's cache.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Compile the native Runner from its complete Cargo and protocol inputs in a separate cached Docker stage. Preserve Cargo validation and generated-contract checks during the normal application build, normalize input timestamps across checkouts, and compile the isolated target in PR CI.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Split release chat verification into three validated test-line shards and balance other server suites across five runners using the measured native Runner integration cost. Retire each chat case's fixtures after assertions, preserve complete test coverage, and exercise the real shard CLI in PR tests.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Cache external Rust dependencies in trusted master release verification after selecting the package-owned toolchain. Keep source compilation and all validation unconditional; restrict both restore and save to the matching master push.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Publish the full-source-SHA cloud image tag only after the pushed digest passes runtime, revision, and platform checks. This makes verified images directly resolvable by Cloud.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every master push publishes a canary through release.yml, gated by
release-verify.yml — the fleet's staging deploys and the
nightly/beta/stable chain all start from those canaries
> - release-verify splits the server test suite across three shards with
a 20-minute job cap, while pr-trusted splits the same suite across five
> - The server suite grew on 2026-09-10 and the three shards moved to
17-19 minutes; that evening every push-triggered canary run was
cancelled by the 20-minute cap mid-verify, and no canary published after
18:50 UTC
> - This pull request mirrors pr-trusted's five-way server split in
release-verify, putting shards back at the 10-15 minute range with real
headroom
> - The benefit is a canary lane that reports test verdicts instead of
dying on an infrastructure cap
## Linked Issues or Issue Description
**What happened?**
Push-triggered Release runs stopped publishing canaries on 2026-09-10.
Runs at 19:34, 22:30, and 22:37 UTC were all cancelled by "The job has
exceeded the maximum execution time of 20m0s" on a `verify_canary /
General tests (server (N/3))` shard. No canary published after 18:50
UTC, which also starves the staging fleet's continuous deploys.
**Expected behavior**
release-verify's server shards finish well inside the 20-minute cap and
runs conclude with a test verdict, as pr-trusted's five-way split of the
same suite does (10-15 minutes per shard).
**Steps to reproduce**
1. Compare server shard durations in the `verify_canary` job across
2026-09-10: 11-14 minutes in the morning, 17-19 minutes from 15:06 UTC,
over 20 minutes by evening.
2. Observe runs 34521169020, 34537798488, and 34538332689 cancelled at
the cap.
**Paperclip version or commit**
`master` at `d1ba17eec` (current tip; its canary run was one of the
cancelled ones).
## What Changed
- `release-verify.yml`: the `general-server` matrix goes from three
shards to five, byte-for-byte the shape `pr-trusted.yml` already runs,
with a comment recording why.
## Verification
- The identical five-way split runs green on every pr-trusted run (10-15
minutes per shard today, including on PRs merged this evening).
- The suite's own growth (slower chat-connector tests) is being
addressed separately; this PR only removes the artificial cliff.
## Risks
- Low risk: two more runners per verify run; no test content changes. If
shard durations regress further, the cap fires again — which is the
correct signal once shards have honest headroom.
## Model Used
- Claude (Anthropic), model ID `claude-fable-5` (Claude Fable 5),
extended thinking, tool use via Claude Code CLI.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip uses GitHub Actions to verify changes before release.
> - The Paperclip Runner has a separate verification boundary.
> - The build job currently runs this verification before the workspace
build.
> - This pull request moves runner verification into its own parallel
job.
> - The benefit is clearer CI results and less wait time for independent
work.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The trusted PR and release verification workflows run Paperclip Runner
verification inside the Build job.
**Subsystem affected**
Cross-cutting (GitHub Actions CI workflows).
**Current behavior**
The Build job runs `pnpm --filter @paperclipai/paperclip-runner
check:all` before it builds the workspace. A runner verification failure
appears as a Build failure. The workspace build cannot run in parallel
with runner verification.
**Proposed behavior**
Each workflow has a `Verify Paperclip Runner` job with the same
checkout, dependency install, and command. The Build job only builds its
required outputs. Both jobs run after the same gate and policy jobs.
**Reason and benefit**
The runner command is an independent verification boundary. A dedicated
job gives it a clear status and allows it to run in parallel with Build.
**Breaking changes**
None. The same runner verification command still runs in both workflows.
## What Changed
- Added a dedicated `Verify Paperclip Runner` job to the trusted PR
workflow.
- Added a dedicated `Verify Paperclip Runner` job to the release
verification workflow.
- Kept the Build jobs independent and retained their existing build
commands.
- Updated the trusted-workflow policy test for the additional
dependency-install job.
## Verification
- Ran `git diff --check`.
- Ran `node --test ./scripts/__tests__/e2e-shard.test.mjs`.
- Ran `pnpm exec prettier --check .github/workflows/pr-trusted.yml
.github/workflows/release-verify.yml`.
- Confirmed both jobs retain their prior runner, dependency, and policy
prerequisites.
## Risks
Low risk. The runner verification job repeats the existing setup. It
adds one parallel GitHub Actions runner to each affected workflow.
## Model Used
OpenAI Codex, GPT-5.6, 128k context window, reasoning and tool-use
capabilities.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
A fleet rollout failed with `image_manifest_not_found` for a commit that
had merged and gone green. Tracing that back: the fleet resolves
releases against the `-cloud` image, that image comes from `docker.yml`,
and `docker.yml`'s runs on master have been reading `cancelled` for a
long stretch. The cloud half was fine; the production half was hanging
and taking the run down with it — and, because a run holds the
concurrency slot for its whole duration, starving later commits of a
build at all.
## Linked Issues or Issue Description
No tracking issue — described inline, per CONTRIBUTING.md.
**What's wrong.** `build-and-push` builds `linux/amd64,linux/arm64` on
an x86 runner, so arm64 runs under QEMU. It hangs there —
deterministically, in the same step:
```
#111 [linux/arm64 build 8/10] RUN pnpm --filter @paperclipai/server build
```
…then emits nothing until `timeout-minutes: 60` kills it. Three
consecutive runs on 2026-09-04, silent for **38, 43 and 45 minutes**
respectively. The amd64 leg reached `production 5/5` minutes earlier in
every one.
**Why it stayed hidden.** A timed-out job is reported by GitHub as
**cancelled, not failed**. The run conclusion reads "cancelled", which
looks like supersession rather than breakage, so the production image
quietly stopped publishing.
**The knock-on.** A run that burns the full hour holds the top-level
concurrency slot for that hour. `cancel-in-progress: false` keeps
exactly one pending slot, so merges arriving faster than one an hour
supersede each other while queued. Sampling the last ten master commits,
**five produced no image at all** — their Docker runs have zero job
records because they never started.
**Expected.** Both architectures publish, and a commit merged during a
busy period still gets an image.
## What Changed
`build-and-push` becomes a two-leg matrix, each on a runner of its own
architecture:
| platform | runner |
|---|---|
| `linux/amd64` | `ubuntu-latest` |
| `linux/arm64` | `ubuntu-24.04-arm` |
Each leg pushes **by digest** (`push-by-digest=true`, untagged), and a
new `merge-and-push` job names the digests into one manifest list with
the real lane tags. Nothing is publicly tagged until the merge, so a
half-published multi-arch image is never a pullable state.
Two supporting changes:
- **Per-arch BuildKit cache refs** (`:buildcache-amd64` /
`:buildcache-arm64`). Separate runners sharing one ref would overwrite
each other on every build.
- **The PID-1 orphan-reaping check moves to the merge job**, since that
is where a tagged, pullable image first exists. It still runs against
the pushed image rather than a local build, for the same reason as
before.
**arm64 is kept, not dropped.** The cloud variant is amd64-only and can
be — managed hosts are amd64. This is the self-hosted image and ARM
hosts consume it, so dropping arm64 would break them. GitHub-hosted
arm64 runners are free for public repositories, which this is.
`build-and-push-cloud` is untouched. It was already `platforms:
linux/amd64` and has been succeeding in ~14 minutes throughout — that is
why `-cloud` images exist at all.
## Verification
Parsed the workflow and asserted its shape (jobs, matrix, `needs`, step
order, that the cloud job is unchanged). The artifact actions are pinned
by SHA with version comments, matching the repo's dominant convention —
`upload-artifact` v7 and `download-artifact` v8, the same pins used
across the other workflows; v8 is required for the `pattern` /
`merge-multiple` inputs the merge job uses.
**This PR's CI does not exercise the change.** `docker.yml` triggers on
master and tag pushes, never on pull requests — deliberately, since it
publishes release images. The first real run is after merge, so the
check is: the next master push produces a `Docker` run whose
`build-and-push (amd64)`, `build-and-push (arm64)` and `merge-and-push`
jobs all succeed, and whose conclusion is `success` rather than
`cancelled`.
## Risks
- **Not testable before merge**, per above. If the matrix is wrong the
next master push fails loudly rather than silently — which is already
better than the current state, where the failure mode is an invisible
"cancelled".
- **First use of `ubuntu-24.04-arm` in this repo.** No other workflow
uses an ARM runner. They are free for public repos, but if the label is
unavailable the arm64 leg will fail to schedule and the merge will not
run — no image, same as today, and visible.
- **Digest-push changes the publish shape.** Between the legs finishing
and the merge running, digests exist untagged in ghcr. Anything watching
for tags sees no intermediate state; anything enumerating untagged
manifests will see more of them.
- **Cache refs change name**, so the first build after this lands is
cold on both legs and will be slower than steady state.
- **Does not fix the underlying QEMU hang** — it avoids it. If arm64
ever has to build under emulation again, the same stall is presumably
still there.
- No application code, schema, server or persistence change.
## Model Used
Anthropic Claude — Opus 5, model ID `claude-opus-5`, run through Claude
Code.
Extended thinking enabled. Tool use throughout: GitHub Actions API to
correlate run/job outcomes and read build logs, `git` for ancestry
checks, and a YAML parser to validate the rewritten workflow's
structure.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
A workflow change has no unit test to add, and `docker.yml` cannot run
on a PR; the verification section states what to check on the first
master run instead.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin Foley <devin@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Docker workflow publishes the server images that all deployments
pull, including the `sha-*` images that downstream consumers deploy
> - The `build-and-push` job builds for linux/amd64 and QEMU-emulated
linux/arm64, and that build now takes more than its 60 minute job
timeout
> - Every run dies at the timeout, and each doomed hour-long run holds
the per-ref concurrency slot, so queued master pushes supersede each
other and no image publishes at all
> - This pull request raises the multi-arch job timeout to 120 minutes
> - The benefit is that image publishing works again, with headroom for
the build to grow
## Linked Issues or Issue Description
Related: #12821 replaces the QEMU-emulated arm64 build with native
runners — that is the durable fix for the build duration itself. This PR
is the immediate unblock so images publish again while #12821 lands.
No existing issue for the outage. Description follows the bug report
template:
**What happened?**
The `docker.yml` `build-and-push` job hits its 60 minute
`timeout-minutes` cap on every run. The last fully successful
`docker.yml` run was September 2. Since then almost every run ends
`cancelled`: the multi-arch build is killed at the timeout, and runs
queued behind it are superseded by newer master pushes before they can
start. The amd64-only `build-and-push-cloud` job often still succeeds
inside those cancelled runs, which masked the breakage.
**Expected behavior**
Every master push and canary tag dispatch publishes its `sha-*`
production and cloud images, and the `promote_canary_channel` job runs.
**Steps to reproduce**
Look at the runs of the Docker workflow on master: `gh run list
--workflow docker.yml --branch master`. Nearly every run since September
5 ends `cancelled` or `failure`. Open a cancelled run: the
`build-and-push` job runs for 61+ minutes and its "Build and push" step
ends `cancelled` at the job timeout. The last runs that succeeded
(September 2) took 39 to 54 minutes for the same job.
## What Changed
- Raise `timeout-minutes` on the `build-and-push` job from 60 to 120,
with a comment that explains why. The amd64-only `build-and-push-cloud`
job keeps its 60 minute cap.
## Verification
- `actionlint .github/workflows/docker.yml` reports no issues in this
change (only pre-existing info-level shellcheck notes in untouched
steps).
- Compared job durations across the last successful runs (39-54 minutes)
and the recent timeout kills (61+ minutes) to confirm the cap is the
failure cause.
- After merge, the next master push should produce a `docker.yml` run
that completes with both build jobs green.
## Risks
Low risk. The change only gives the existing build more time. A
genuinely hung build now occupies a runner for up to 120 minutes instead
of 60. The slow arm64 emulated build itself is worth a separate look
(native arm runners or splitting the platforms), but that is a larger
change than this outage fix.
## Model Used
Claude (Anthropic) — Fable 5 (`claude-fable-5`), extended thinking,
agentic tool use via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no code paths changed;
workflow linted with actionlint)
- [x] I have added or updated tests where applicable (not applicable for
a CI timeout value)
- [x] I have updated relevant documentation to reflect my changes (the
workflow comment documents the rationale)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every master merge publishes an npm canary, and downstream managed
deployments consume the matching `sha-<short>-cloud` Docker image.
> - The canary image relies on the master-push `docker.yml` run, whose
single pending concurrency slot is superseded by every newer push.
> - On a busy day the image build never survives: five consecutive
canaries shipped npm packages with no cloud image on 2026-09-06, and
downstream deploys starved for ~18 hours while master kept moving.
> - The nightly and beta lanes already solve exactly this by dispatching
`docker.yml` at their tag ref after pushing it.
> - This pull request wires the canary lane into the same mechanism, so
every canary tag gets an image build no master push can supersede.
> - The benefit is that a published canary always has its images, and
downstream release resolution stops walking back to day-old builds
during busy merge windows.
## Linked Issues or Issue Description
Refs #12769 / #12855 (the earlier image-publishing incident in the same
pipeline, different failure mode).
**What happened?**
`docker.yml` serialises per ref with one pending slot (`concurrency:
docker-${{ github.ref }}`, `cancel-in-progress: false`). Master pushes
arriving faster than the ~50-minute build supersede the pending build
indefinitely. On 2026-09-06, canaries `2026.906.0-canary.1` through `.3`
(and the commits between) published to npm with no `sha-<short>-cloud`
image on GHCR — the Docker run list shows `cancelled, cancelled,
cancelled` for their commits. Downstream managed rollouts correctly
refused to deploy image-less canaries and pinned at `canary.0` for ~18
hours.
**Expected behavior**
Every published canary has its Docker images. A busy merge window must
not be able to prevent image publication for tagged releases.
**Steps to reproduce**
1. Merge to master more often than the Docker build takes to complete,
for several hours.
2. Observe npm canaries advancing while every `Docker` run for their
commits completes `cancelled`.
3. Observe `ghcr.io/paperclipai/paperclip:sha-<short>-cloud` returning
404 for each of those canaries.
**Paperclip version or commit**
Master at `83987210` (diagnosis time); the starved canaries were
`2026.906.0-canary.1`–`.3`.
**Deployment mode**
GitHub Actions release + image pipeline; consumed by managed cloud
deployments.
## What Changed
- `.github/workflows/release.yml`, `publish_canary` job:
- New step after "Push canary tag": dispatch `docker.yml` at
`refs/tags/canary/v<version>` — identical to the nightly and beta lanes'
existing step, including the step-summary note. The dispatched run keys
its concurrency off the tag ref, so master pushes cannot supersede it,
and `docker.yml`'s `type=sha` mapping publishes `sha-<short>` /
`sha-<short>-cloud` for any ref.
- The job gains `actions: write`, mirroring `publish_nightly`.
- No `docker.yml` changes: it already accepts `workflow_dispatch` for
exactly this pattern.
- No dry-run guard needed: `publish_canary` runs only on `push` events,
so the `dry_run` dispatch input cannot reach it.
## Verification
- YAML lints clean; the step is a line-for-line mirror of the proven
nightly-lane step (tag source swapped for the job's existing
`canary_tag` output).
- The concurrency claim is docker.yml's own documented behavior: groups
are per-ref, and a tag ref is distinct from `refs/heads/master`.
- Not run: a live canary publish — the next master merge after this
lands exercises it end to end; the worst case (double build for a canary
whose master-push run also survives) is benign, since both runs push
identical content-addressed tags.
## Risks
- Low. Roughly one additional Docker build per canary on busy days (on
quiet days the master-push run and the dispatched run both execute —
duplicate work, identical images, no conflict since the sha tags are
content-equivalent for the same commit).
- `actions: write` on `publish_canary` matches the grant
`publish_nightly` already carries for the same purpose.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
Claude Fable 5 (Anthropic, model id `claude-fable-5`), extended
thinking, agentic tool use in Claude Code: GHCR manifest probing,
Actions run-list forensics, and workflow-lane comparison.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (workflow-only change; YAML
linted; no test harness targets release.yml)
- [x] I have added or updated tests where applicable (not applicable —
CI wiring mirroring an existing proven lane)
- [x] I have updated relevant documentation to reflect my changes (the
step's inline comment documents the invariant)
- [x] I have considered and documented the risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip manages AI agents and their work.
> - Managed deployments need a cloud image and a database migration
package.
> - Branch commits can lack both artifacts until a normal release runs.
> - Operators need to test an exact commit without advancing release
aliases.
> - This pull request adds a preview build mode to the existing release
workflow.
> - Builds use an immutable source SHA and publish isolated, reusable
artifacts.
## Linked Issues or Issue Description
**Subsystem affected**
Release automation, cloud Docker images, and shared/database npm
packages.
**Problem or motivation**
An operator cannot deploy an unpublished branch with new migrations
using only
the normal release artifacts. Publishing it through a normal lane would
also
advance shared release aliases.
**Proposed solution**
Dispatch the trusted release workflow on master with a full source SHA
and a
request UUID. Build missing SHA images and, when needed, deterministic
preview
shared/DB packages. Publish packages under the preview dist-tag with
exact
workspace pins. Reuse matching artifacts on retries.
**Roadmap alignment**
This extends release tooling for operator validation. It does not add a
core
product feature or duplicate a planned product capability. Related PR
searches
found no duplicate preview deployment workflow.
## What Changed
- Add the preview channel, request correlation, artifact checks, and
result artifact.
- Compile source packages in a separate job from the npm publisher. The
publisher
uses trusted master code and disables package lifecycle scripts.
- Publish only SHA cloud image tags. Preserve release aliases. Use
full-SHA tags and no shared build cache.
- Verify full source identity for reused packages and images. Both image
and npm publishers
use isolated jobs and the externally master-restricted npm-canary
environment. Fail on registry
authentication errors, outages, or artifact identity mismatches.
- Let bundled-package preparation use patches from the requested source
checkout.
- Document publishing configuration, artifact contracts, and deployment
order.
## Verification
- Passed `pnpm -r typecheck` and `pnpm build`.
- Passed `pnpm test:release-registry`: 107 tests, including eight
preview tests.
- Passed `actionlint -shellcheck= .github/workflows/release.yml`.
- Built real shared and DB preview tarballs from an isolated exact-SHA
checkout.
Verified package source identity and all 244 SQL files and journal
entries.
- Verified the full revision behind an existing published SHA cloud
image.
- `pnpm test:run` exposed missing local embedded PostgreSQL library
symlinks.
The package's postinstall repair restored initdb; all 12 previously
affected
suites passed on rerun (95 tests). Additional local matrix reruns are in
progress.
The complete PR CI matrix is green, including general/serialized tests,
e2e,
typecheck, build, release registry, canary dry run, and the required
verify gate.
- Live preview publication and staging deployment require this workflow
on master
and the compatible control-plane backend. They have not run yet. No
production
deployment was performed.
## Risks
Preview npm versions are immutable public artifacts. Both packages must
retain
their trusted publisher for release.yml in environment npm-canary.
Source builds
must remain separated from privileged npm publishing. The deploying
control plane
must verify source identity, integrity, and migration compatibility
before use.
Normal release jobs retain their existing conditions. Roll back by
stopping preview
dispatches and reverting the workflow/tooling. Published preview
versions remain
isolated from normal release tags.
## Model Used
OpenAI GPT-6 through Codex, with repository tools, code execution, and
test runs.
The session does not expose a more specific model version or
context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with the available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs
- [x] I have described the issue in-PR following the feature template
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [ ] I have run the full tests locally and they pass
- [x] I have added tests for the new behavior
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open recommendations or follow-ups
- [x] I will address review comments before requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Direct Runner evals retain evidence across model configurations.
> - Evalbook already has a grid and a read-only Runner Lab chat viewer.
> - Public projection stripped the view and selected a second plain
result page.
> - This change uses the existing viewer for public and private results.
> - The data access differs, but the presentation does not.
## Linked Issues or Issue Description
Refs #12931, #12945. Related open runtime-contract PR #11634 does not
contain this report-only change.
**What happened?**
The public direct-eval campaign opened plain result pages. The
access-controlled artifact used the chat viewer. Users could not follow
the same recorded interaction from the published grid.
**Expected behavior**
Every newly generated Runner Evalbook opens the existing chat viewer.
The grid and durable run history remain. Public evidence has explicit
redactions.
**Steps to reproduce**
Open campaign gha-34062394019-1 from the direct-eval history. Click a
result, then compare its plain page with the corresponding Actions
artifact.
**Paperclip version or commit**
Reproduced at 83987210d6.
**Deployment mode**
Static GitHub Actions artifacts and S3/CloudFront publication.
Companion site-theme renderer:
https://github.com/paperclipai/paperclip-evals/pull/19. This removes the
Python light theme and links the same built stylesheet.
## What Changed
- Add a closed public chat projection. Require mock isolation evidence
before publishing recorded text. Scrub private references and withhold
tool payloads, reasoning and provider state.
- Validate public HTML against the exact trusted viewer shell and asset
bytes. Validate the public DTO and local links. Keep CSP restrictions on
outbound requests and forms.
- Make the workflow render both data projections with the canonical
viewer. Pass a viewer-only artifact to the trusted publisher. Reject an
old renderer pin before paid execution.
- Fix report-only start position, missing-state inspector, read-only
controls and redaction labels. Tool evidence links select and highlight
the Evidence tab even when reopened. Runner execution and the full-stack
E2E workflow are unchanged.
- Add a no-model report refresh command. Preserve original campaign
identity, measurements and immutable history; label report revisions.
- Document the single presentation and public/private evidence boundary.
- Use one Runner Lab stylesheet and local fonts for the grid, Latest,
test design, inventory, server gate and S3 history index. Keep static
styles scoped away from live chat.
- Emit exact published report/history URLs to the Actions summary and
job outputs only after successful upload. Set the deployment link from
that output.
- Switch public Eval and Evidence panes without rendering both at once.
- Add all-run history with like-for-like pass-rate and cost timelines,
regression/recovery lists, exact commit links, source refs and Actions
links.
- Record all-attempt costs including retries. Keep provider list cost
separate from estimates. Label missing coverage and historical
final-only costs.
- Retain all run records beyond 200, backfill a separate derived
analytics projection, and exclude report refreshes from measurements.
## Verification
- Focused report/publishing/projection/workflow/adapter tests: 29
passed.
- Workflow Evalbook adapter tests: three passed.
- Viewer unit tests: eight passed; Vite viewer build passed.
- Companion renderer tests: 59 total, 57 passed, two inventory tests
skipped because the expected sibling checkout is absent.
- Re-rendered all 375 retained attempts from the completed campaign and
validated the public bundle. Zero new model calls; 356/358 selected
cells still pass.
- Browser walkthrough: grid to failed chat; prompt, named tool calls,
correct blocked status, visible assertions, no loading spinner or
composer. Public payload redactions are explicit.
- Full typecheck and build passed locally. test:run ended with 17 failed
files and 19 failed tests in unrelated server/worktree areas (3772 tests
passed). Latest-head CI is the final merge gate. The browser sources
also typecheck with a temporary TypeScript-7-compatible path
configuration; the checked-in browser config still uses removed baseUrl
options and is unchanged here.
- Real Chromium verification passed for passing, failing and
missing-recording attempts in both the full and public bundles. It
checks tool expansion, navigation, reload, read-only controls, narrow
viewport visibility and the public no-network boundary. Future
publications run it automatically.
- Fresh live proof: three gpt-5.4-mini native Codex cases passed on the
first attempt (get-task-context, create-child-task,
workflow-context-document-progress); estimated total $0.00632625.
Generated the full canonical report and verified all three file://
pages, all seven DevTools tabs, and evidence cross-links in Chromium.
Private screenshots remain local.
- Follow-up 99697c2c5: viewer build, eight unit tests, browser token
gate and browser-source typecheck (existing TS7 configuration
workaround) passed. Added repeated tool-to-evidence navigation to the
publication browser gate.
- Site-theme follow-up: 26 publishing/security tests, 59 Python tests
(57 passed, two existing skips), eight viewer unit tests and the viewer
build passed. Chromium verified shared colors and
grid/test-design/Latest navigation.
- Re-rendered all 375 retained attempts with the shared site theme,
without new provider calls.
- Successful hosted publication: [refreshed
Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34062394019-1-report-site-theme-v3/index.html)
and [themed
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html).
Real browser verified grid → test design → chat → All results and
history → run.
- Full maintained live suite completed on AWS: [Actions run
34074939112](https://github.com/paperclipai/paperclip/actions/runs/34074939112),
Paperclip 856813ba3a, evals
34e1846c06a39e641182dadce5de7ea739f657f1. All 358 cells across 11
configurations ran; 355 passed (99.2%), two behavior failures and one
infrastructure failure. Nine configurations are entirely green. All 360
retained attempts were rendered using the new design and published as an
immutable, zero-provider-call report refresh: [full
Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34074939112-1-report-history-v1/index.html).
The trusted master workflow ran the models; this PR's viewer rendered
their results afterward.
- Remaining live failures: native Codex mini did not emit the expected
discovery event for lazy-unauthorized-undisclosed (no mutation
occurred); GLM 5.3 scheduled a wake before approval in
workflow-governed-wait; GLM 5.3 timed out on create-task-document on
both attempts. No scoring rules were relaxed.
- Retry-inclusive recorded estimated model cost is at least $7.067692,
with usage for 357/360 attempts. Provider-reported list cost is a
separate alternative (at least $14.002806), not an additive cost.
Missing usage is unknown, not zero; AWS compute is excluded.
- [Production
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
now contains nine model runs and two separately labeled report
refreshes. Matching full-suite comparison reports three newly failing
cells and two recoveries versus the previous run. This is observed
run-to-run variation, not proof of a deterministic code regression. Live
HTTP checks verify the newest report link, shared theme, exact source
SHAs and cost analytics.
- The new full report passed Chromium checks for passing, failing and
missing-recording chat pages, tool-to-Evidence links, grid/design/Latest
navigation, reload, read-only controls and narrow layouts. The Mac was
locked during final hosted verification; the new hosted history was
checked by HTTP and generated-page browser checks, not a fresh
interactive desktop walkthrough.
- History follow-up: 31 publishing/security/metrics tests pass. Viewer
build passes. Chromium checks desktop and narrow history pages with no
document-level horizontal overflow.
### Visual verification
Generated from the scrubbed completed campaign; no private provider
identities or raw tool payloads are included. Full private pages were
also browser-tested, but their private metadata is not published as
screenshots.
Latest shared-theme proof:
## Risks
- Public chat text is newly visible, but only for the isolated mock
boundary. The producer excludes raw payloads and the publisher fails
closed on unknown fields, secrets, shell changes and asset
substitutions.
- Requires the companion canonical renderer revision and an updated
RUNNER_PROTOCOL_EVALS_SHA after merge. Old pins fail before paid
execution.
- Existing published campaigns remain immutable. A report refresh is a
separate history entry, not a new model qualification.
- Successfully published the immutable site-theme refresh with the
configured report-bucket SSO profile. Original run records and
qualification pointers are preserved.
## Model Used
OpenAI Codex, GPT-5-based coding agent with reasoning, shell and browser
tools. Exact deployment model ID and context-window size are not exposed
in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
cad99dbf04
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on cad99dbf04
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip runs AI agents through qualified provider backends.
> - The direct live eval workflow builds one immutable Runner runtime
for every matrix cell.
> - The workflow reinstalled the packed Runner with npm.
> - That install discarded pnpm patches and selected provider
dependencies outside the qualified lock.
> - The first pnpm deployment model also placed its virtual-store marker
at the wrong level; a real deployment keeps `.pnpm` beside the scoped
Runner package.
> - AgentCore enforced the current context-aware harness but the direct
eval CLI did not supply the production v3 runtime context that harness
requires.
> - This pull request preserves the qualified dependency graph, resolves
the real deployment layout, and makes direct evals exercise the
production runtime-context contract.
> - The benefit is that live eval cells reach their provider turn with
the same artifacts and context contract that Paperclip qualified.
## Linked Issues or Issue Description
Refs: #12931
**What happened?**
The full direct live eval campaign failed every ACPX cell during
`session.open`. The portable runtime had an incorrect dependency root.
Its npm install also discarded the qualified ACP server patches.
AgentCore cells first failed because Runner enforced
`aws-agentcore-harness-v1` while the provisioned stack and eval profile
use `aws-agentcore-harness-context-v2`; after aligning that revision,
the direct eval CLI still omitted the required v3 runtime context.
**Expected behavior**
The direct eval runtime must preserve the frozen pnpm dependency graph
and patched provider bytes. Runner, server validation, OpenAPI, and the
deployed AgentCore stack must use one qualification revision. Direct
eval attempts must supply the same immutable native runtime-context
contract as production.
**Steps to reproduce**
1. Dispatch `Runner Direct Live Protocol Evals` from `master`.
2. Select an ACPX Claude, ACPX Codex, or AgentCore roster.
3. Observe a pre-turn provider bootstrap failure.
**Paperclip version or commit**
`d96452db059338b329b458ba8fe359fef72f1363`
**Deployment mode**
GitHub Actions on the RunsOn Linux x64 fleet.
## What Changed
- Build the reusable direct-eval runtime with `pnpm deploy --prod`.
- Resolve ACPX dependencies from the actual scoped-package layout of a
self-contained pnpm deployment.
- Align AgentCore configuration and qualification checks on
`aws-agentcore-harness-context-v2`.
- Materialize a minimal immutable v3 runtime context for each isolated
direct eval attempt.
- Add workflow, package-authority, runtime-context, Rust, and server
regression coverage.
- Document the qualified packaging, runtime-context, and AgentCore
revision contracts.
## Verification
- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport.test.ts` (70 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/cli/eval-session-contract.test.ts` (14 tests)
- Focused Runner contract tests (36 tests)
- Focused server profile tests (47 tests)
- Focused Rust managed-provider and native-selector tests (19 tests)
- `node --test
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs`
- `actionlint .github/workflows/runner-protocol-live-evals.yml`
- A local `pnpm deploy --prod` produced both qualified ACP server
digests.
- A Linux reproduction of the first follow-up smoke identified the real
deployment root and the missing AgentCore runtime context.
## Risks
The AgentCore revision change rejects profiles that still use the
obsolete v1 value. This is intentional because the provisioned
context-aware harness and current eval profile use v2. Direct eval
prompts now receive the same fixed runtime-context preamble as
production, so behavior scores may move; that is the intended
qualification surface. The workflow package layout changes, but tests
assert the new entrypoint and dependency root. This change does not
modify the browser full-stack E2E workflow.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex with GPT-5.6. The context-window size is not exposed in
this session. The model used extended reasoning, repository tools, code
execution, Docker-based Linux reproduction, and GitHub Actions
diagnostics.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (for example, `docs/...` or
`fix/...`) and contains no internal Paperclip ticket id or
instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner includes an issue-thread viewer for direct
Evalbook reports.
> - The runner package uses Vite 6.4.3.
> - Dependabot changed the React plugin from version 4.7.0 to version
6.1.1.
> - React plugin 6.1.1 requires Vite 7 or Vite 8 package internals.
> - The live evaluation workflow found this mismatch when it built the
viewer on a clean Linux worker.
> - This pull request restores the compatible plugin and adds the viewer
build to pull request CI.
> - The benefit is that CI detects this class of report-viewer build
failure before a paid evaluation campaign starts.
## Linked Issues or Issue Description
**What happened?**
The direct live evaluation workflow failed before model execution. The
`build:issue-thread` command could not load `@vitejs/plugin-react@6.1.1`
with Vite 6.4.3. The plugin imported the unavailable `vite/internal`
package path.
**Expected behavior**
The Evalbook issue-thread viewer must build from a clean frozen-lockfile
installation before the live evaluation matrix starts.
**Steps to reproduce**
1. Check out commit `165ca56a22adb60e5fda56045442d9c8498116a8`.
2. Run `pnpm install --frozen-lockfile --ignore-scripts`.
3. Run `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`.
4. Observe the `ERR_PACKAGE_PATH_NOT_EXPORTED` error for
`vite/internal`.
**Paperclip version or commit**
`165ca56a22adb60e5fda56045442d9c8498116a8`
**Deployment mode**
Built from source in GitHub Actions on Ubuntu.
## What Changed
- Restore `@vitejs/plugin-react` 4.7.0 in the Vite 6 runner package.
- Follow the repository policy: trusted PR CI regenerates and verifies
the lockfile artifact, and the master refresh workflow commits the
lock-only update after merge.
- Build the Runner Evalbook viewer in pull request CI.
## Verification
- `ci / policy`: regenerated the dependency lock artifact successfully
- `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`
- `actionlint .github/workflows/pr-trusted.yml
.github/workflows/runner-protocol-live-evals.yml`
- `node --test
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs`
- `git diff --check`
## Risks
Low risk. This change restores the previous React plugin major version
for one package. The selected version declares support for Vite 6. Pull
request CI now builds the affected viewer directly.
> This bug fix does not add or change a roadmap feature.
## Model Used
- OpenAI Codex with GPT-5. Tool use and code execution were enabled. The
Codex app managed the context window.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip uses a paid full-stack campaign to verify runner behavior
across providers and environments.
> - The campaign already creates an interactive report, workflow logs,
and retained evidence artifacts.
> - The merge job summary shows result totals but does not link to those
resources.
> - Reviewers must search several workflow jobs and artifacts to find
the executed cells.
> - This pull request adds direct and safe links to the exact campaign,
each cell, the workflow logs, and the artifacts.
> - The benefit is that a reviewer can inspect a result from the Actions
summary with one click.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
This improves the `Merge and enforce campaign result` summary in the
`Runner Full-Stack E2E` workflow.
**Subsystem affected**
The runner E2E report generator and its GitHub Actions workflow are
affected.
**Current behavior**
The summary lists each selected cell and its result. It does not link to
the published campaign report, the workflow logs, or the evidence
artifacts.
**Proposed behavior**
The summary includes a `View results` section. It links to the exact
immutable campaign report, the workflow logs, and the artifacts. Each
cell name links to its stable section in the campaign report.
**Reason and benefit**
The current summary does not show reviewers where to inspect the run.
Direct links make the result evidence discoverable without manual URL
construction or artifact searches.
**Breaking changes**
None. This change only adds links and stable HTML anchors to existing
report output.
**Additional context**
Related: #12904. The cited successful campaign is [run
34026735033](https://github.com/paperclipai/paperclip/actions/runs/34026735033).
## What Changed
- Add a safe URL builder for public campaign, workflow, and artifact
links.
- Add a `View results` section to the GitHub Actions campaign summary.
- Link each summary table cell to its exact section in the immutable
campaign report.
- Add stable execution anchors to the generated dashboard.
- Reject non-HTTPS, credential-bearing, malformed, and ambiguous link
destinations.
- Document the new links and their retention or publication timing.
## Verification
- `pnpm test:e2e:runner:unit` — 116 tests passed.
- `pnpm test:e2e:runner:typecheck` — passed.
- `pnpm typecheck` — passed, including migration safety.
- `pnpm build` — passed.
- `pnpm exec prettier --check ...` for all changed files — passed.
- `git diff --check origin/master...HEAD` — passed.
- The full local server suite also ran. One unrelated macOS
workspace-runtime file passed 157 tests and failed 4 existing path and
port assumptions. Two failures compare `/var` with `/private/var`. Two
failures cannot reserve a port outside a hard-coded range. This PR does
not change that file or its dependencies.
## Risks
- The immutable campaign link becomes available after the history
publisher completes. The workflow and artifact links remain available
while publication runs.
- The artifact link requires GitHub access and follows the existing
30-day retention period.
- Invalid configured URLs are omitted instead of being rendered into the
summary.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex desktop agent with GPT-5. The runtime does not expose the
context-window size. The agent used repository inspection, agentic
reasoning, code execution, and GitHub CLI tools.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g.
`docs/no-internal-issue-references`, `fix/sandbox-secret-resolution`,
`feat/adapter-retry-backoff`) and contains no internal Paperclip ticket
id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - The merged direct live eval workflow must read the private
`paperclip-evals` repository at an exact commit.
> - Its first hosted dispatch failed before provider execution because
the GitHub App token was minted from the `paperclip` repository
installation.
> - GitHub returned 404 while resolving the private eval commit, proving
that token did not have the required repository scope.
> - Minting each short-lived token from the exact private eval
repository installation supplies only the cross-repository read boundary
the workflow needs.
> - A workflow regression now verifies every eval-token block keeps that
exact scope.
## Linked Issues or Issue Description
The first default-branch run of Runner Direct Live Protocol Evals failed
in its immutable eval-commit verification step with HTTP 404. No
provider jobs ran and no provider spend occurred.
**What existing behavior does this improve?**
It allows the protected direct live eval workflow to verify and check
out the private `paperclipai/paperclip-evals` repository.
**Current behavior**
All four eval-token blocks set `GH_REPO` to `paperclipai/paperclip`,
selecting a token installation that cannot read the private eval
repository.
**Proposed behavior**
Set `GH_REPO` to the exact `paperclipai/paperclip-evals` repository in
authorization, catalog, matrix, and report jobs.
**Reason and benefit**
The app mints a short-lived token from the correct repository
installation while the main repository continues to use its ordinary
read-only workflow token.
**Breaking changes**
None.
## What Changed
- Scoped all four private-eval installation tokens to
`paperclipai/paperclip-evals`.
- Added a regression requiring that exact scope in every token block.
## Verification
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-protocol-eval-publish` — 15 passed.
- `node --test .github/scripts/tests/get-bot-token.test.mjs` — 3 passed.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` —
passed.
- `git diff --check` — passed.
## Risks
- The workflow reads a private repository. The token is still
short-lived, repository-specific, masked immediately, and used only by
the protected default-branch workflow.
- This changes no provider execution, Runner behavior, report content,
S3 publishing, or browser E2E behavior.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, repository inspection, code
editing, GitHub Actions diagnostics, and test execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation where applicable
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner executes agents through native and managed provider
drivers.
> - The direct live eval layer had drifted from the current Runner
contracts.
> - The old local workflow did not provide a complete parallel campaign
or durable report history.
> - The Runner also needed current native OpenCode and OpenRouter
qualification.
> - This pull request restores the direct campaign, corrects the runtime
gaps that the campaign found, and adds safe hosted Evalbook history.
> - The benefit is repeatable model comparison against an immutable
Runner and eval source revision.
## Linked Issues or Issue Description
Refs #11297
Refs #11634
**What existing behavior does this improve?**
This improves the direct live `paperclip-runner` eval workflow, provider
execution contract, and static Evalbook reporting path.
**Current behavior**
The direct evals do not have one maintained full campaign on current
`master`. OpenCode has no qualified multi-model OpenRouter roster.
Parallel provider bursts can compact committed events before the
transport observes them. Local reports do not have a separate safe S3
history index.
**Proposed behavior**
Run one immutable roster-plus-case matrix. Use the shared paid AWS
runner fleet. Keep raw artifacts access-controlled. Publish a sanitized
canonical Evalbook report under the separate `runner-protocol-evals` S3
prefix. Keep immutable campaign directories plus root history, latest,
and latest-green pointers.
**Reason and benefit**
Maintainers can compare native Codex, native OpenCode, ACPX, Claude
Managed, and AWS AgentCore behavior over time. They can inspect failures
without mixing this direct protocol layer with browser full-stack E2E.
**Breaking changes**
None. The new workflow and S3 prefix are additive. The existing Runner
full-stack E2E workflow and report remain separate.
## What Changed
- Added a trusted two-shard direct live workflow for up to 393
roster-plus-case cells.
- Reused the numeric actor allowlist, protected paid environment, and
RunsOn fleet controls from Runner full-stack E2E.
- Added immutable Runner and eval revision resolution, exact credential
boundaries, bounded retries, and cost ceilings.
- Added a public report projection that removes sessions, transcripts,
tool payloads, state, traces, raw failures, remote profile identities,
and credential-shaped values.
- Added additive S3 history under `runner-protocol-evals`, with
immutable campaigns and mutable root index pointers.
- Added native OpenCode model injection and current OpenRouter pricing
contracts.
- Fixed direct eval completion, workflow execution, semantic discovery,
warm-attach state reset, executable binding, and event-burst handling.
- Kept Runner browser full-stack E2E behavior and publication separate.
- Documented local and hosted direct eval operation.
## Verification
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-protocol-eval-publish` — 15 passed.
- `pnpm --filter @paperclipai/paperclip-runner build:typescript` —
passed.
- `actionlint .github/workflows/runner-protocol-live-evals.yml
.github/workflows/runner-full-stack-e2e.yml` — passed.
- Local current matrix at the revision in
[paperclip-evals#17](https://github.com/paperclipai/paperclip-evals/pull/17)
— 323 cells across 10 enabled configurations completed.
- Final local current matrix — 269 passed, 11 behavior failures, and 43
expected macOS-only ACPX platform failures.
- Targeted Runner checks — 13/13 eval-session tests, 15/15
publisher/security tests, and package typecheck passed; complete PR CI
is green, including all browser E2E shards.
## Risks
- Paid live campaigns can consume provider budget. Actor authorization,
exact per-cell ceilings, protected environments, and explicit schedule
enablement bound this risk.
- Public reports can leak provider data. The workflow publishes only a
separately projected report and validates every file before upload.
- The new workflow cannot publish until it is present on the default
branch. This pull request does not change the existing
`runner-full-stack-e2e` publication path.
- The campaign is large. It uses two GitHub matrices and caps combined
concurrency at the shared fleet limit.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, code editing, browser
inspection, repository tools, and live provider execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip manages AI agent work and the execution state for each
task.
> - Remote agents run in sandbox environments such as Daytona.
> - Daytona keeps files while a sandbox is stopped, but deletion removes
those files.
> - Runner Codex did not copy successful remote workspace changes back
to the host workspace.
> - A warm sandbox could therefore hide data loss until Daytona replaced
or deleted the sandbox.
> - This pull request makes the host workspace durable after every
successful turn and keeps verified reusable sandboxes warm.
> - The benefit is reliable multi-turn work across warm reuse, restart,
stop, and sandbox replacement.
## Linked Issues or Issue Description
**What happened?**
A successful native Codex turn in Daytona could leave workspace changes
only in the remote sandbox. A later warm turn appeared to work because
it reused that filesystem. A replacement sandbox could start from stale
host data and lose the successful changes.
**Expected behavior**
Paperclip must merge each successful remote turn into the authoritative
host workspace before it completes the run. A verified warm lease may
reuse its remote files. A replacement lease must reconstruct the exact
durable workspace seed.
**Steps to reproduce**
1. Run Codex in a reusable Daytona environment.
2. Write a file during one successful turn.
3. Replace the Daytona sandbox before the next turn.
4. Observe that the next turn can start without the prior file on the
unpatched code.
Related remote workspace foundation: #10070.
## What Changed
- Added explicit `host_current`, `durable_seed`, and `adopt_remote`
workspace preparation modes.
- Added atomic, versioned native workspace descriptors and seed archives
under `PAPERCLIP_HOME`.
- Added real native sandbox export and three-way host merge before
terminal result completion.
- Added workspace-only recovery after a proposed result. Recovery does
not submit another provider turn or consume the provider retry budget.
- Added fail-closed handling when a sandbox with unexported changes is
gone.
- Kept healthy reusable Daytona sandboxes started for legacy Codex and
Runner Codex.
- Kept the Runner Codex process and provider session across verified
warm turns.
- Added the paid `daytona-warm-continuity` browser suite. It contains
exactly the legacy Codex and Runner Codex cells. Each cell performs
three measured turns.
- Documented `pnpm test:e2e:runner -- --suite daytona-warm-continuity`.
No package script was added.
- Added no database migration. The metadata format is backward
compatible and idempotent.
## Verification
- `pnpm typecheck`
- `pnpm test:e2e:runner:unit` — 114 passed
- Native workspace, finalizer, session, and environment tests — 232
passed
- Daytona provider tests — 150 passed
- Workspace staging and merge tests — 98 passed
- Runner transport tests — 63 passed
- Legacy Codex restore tests — 5 passed
- Rust format and compile checks pass through root typecheck
- The paid Daytona suite was not run locally because the required
Daytona, OpenAI, and immutable image credentials are not present.
## Risks
- The main risk is an incorrect workspace identity or merge after a
crash. Durable descriptors bind the run, workspace, lease, provider
lease, local root, remote root, and baseline digest. Ambiguous evidence
fails closed.
- The host merge may conflict with concurrent host edits. The existing
three-way merge and exclusion rules handle this case and surface
failures.
- A deleted sandbox cannot recover unexported bytes. Paperclip now
blocks with `workspace_sync_out_unrecoverable` instead of reporting
success or rerunning the provider.
- There is no database migration. Descriptor writes and recovery are
atomic and idempotent.
## Model Used
OpenAI Codex with GPT-5. The run used agentic reasoning, repository
inspection, code execution, test execution, Git, and GitHub CLI tools.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server recovery service monitors active runs and applies
watchdog decisions
> - The watchdog rules and database operations lived in one large
recovery service
> - This structure made the rules harder to test and made company
scoping harder to inspect
> - This pull request moves the watchdog into domain, application, and
adapter layers
> - The benefit is a smaller recovery service, pure policy tests, and
clear company-scoped ports
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The active-run output watchdog that detects silence, suppression, and
terminal evidence.
**Current behavior**
The recovery service contains the watchdog policy, use cases, database
operations, and process control in one file.
**Proposed behavior**
A feature module separates pure policy, use cases and ports, and
Postgres and process adapters. The recovery service delegates its public
watchdog methods to this module.
**Reason and benefit**
The separation makes policy decisions easy to test. Company identifiers
on every reader and writer port make tenant scope clear. Smaller service
methods reduce change risk.
**Breaking changes**
None. The recovery service keeps its public methods and call sites.
Related public watchdog work includes
[#7043](https://github.com/paperclipai/paperclip/pull/7043) and
[#7770](https://github.com/paperclipai/paperclip/pull/7770).
## What Changed
- Add the `server/src/modules/active-run-watchdog/` feature module with
domain, application, and adapter layers.
- Move watchdog policy, use cases, Postgres access, and local process
control into the module.
- Keep the recovery service public methods and delegate them to the
module.
- Add 53 pure module test cases and retain 8 Postgres integration cases.
- Add company scoping and transaction rollback coverage.
## Verification
- Run `vitest run --config vitest.config.ts src/modules` and confirm 3
files and 53 cases pass.
- Run `vitest run --config vitest.config.ts
src/__tests__/heartbeat-active-run-output-watchdog.test.ts` and confirm
1 file and 8 cases pass.
- Run the full server suite in pull request CI.
- Compare the type-check result with a fresh baseline on the same
checkout.
## Risks
The main risk is a behavior change in recovery decisions during the move
across layers. The pure policy tests cover the moved rules. The
integration tests cover database behavior, company scope, and
transaction rollback. Pull request CI runs the full server suite.
## Model Used
OpenAI Codex, GPT-5, runtime-managed context window, tool use, code
execution, and repository review support.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip uses runner end-to-end reports to compare agent profiles
and execution environments
> - The report dashboard shows each reviewed final-state screenshot as a
thumbnail and gallery item
> - The public history publisher removed all per-attempt images before
it regenerated the dashboard
> - Therefore the public dashboard had the new layout but could not show
the screenshots from the run
> - The publisher needs a narrow rule that keeps only screenshots from
the exact live fixture issue route
> - This pull request keeps those trusted PNG files in every future
public S3 and GitHub Pages report
> - The benefit is that each future report can show its screenshot
gallery without exposing logs, traces, videos, archives, arbitrary
images, or generated report trees
## Linked Issues or Issue Description
**What happened?**
The runner E2E job captured final-state screenshots in its private
artifact. The public S3 and GitHub Pages publication step removed those
screenshots before it regenerated the dashboard. As a result, the public
report showed the new dashboard controls but no screenshot thumbnails or
gallery items.
**Expected behavior**
Each future public runner E2E report must include reviewed PNG
screenshots from the live fixture issue. Other captures and active or
unsafe evidence must stay private.
**Steps to reproduce**
1. Run the runner full-stack E2E workflow on `master` before this
change.
2. Open the private `runner-e2e-report-*` artifact and confirm that it
contains per-attempt PNG screenshots.
3. Open the public campaign URL and confirm that the dashboard has no
screenshot gallery items.
**Paperclip version or commit**
The issue was reproduced on commit `64d8929`, after the report design
change in PR #12889.
**Deployment mode**
GitHub Actions with the public S3 and GitHub Pages report publishers.
Related design work: Refs #12889.
## What Changed
- Mark screenshots from the exact server-created live fixture issue
route with `public-runner-fixture`.
- Keep marked PNG files in both the S3 history bundle and the GitHub
Pages bundle.
- Keep captures from other issue routes, sensitive routes, and external
origins private.
- Bind public files to the normalized execution ID, attempt, and safe
PNG base name.
- Validate every retained image with the existing PNG signature and 12
MiB size checks.
- Skip missing-artifact sentinel results with attempt `0` when they have
no public screenshots.
- Continue to remove unmarked images, videos, traces, archives,
generated HTML reports, and other private evidence.
- Update publisher tests, workflow checks, report copy, and the
public-evidence security documentation.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/history.test.ts tests/runner-e2e/report.test.ts`
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/workflow-security.test.ts -t "uses environment-scoped
OIDC"`
- `pnpm test:e2e:runner:typecheck`
- `PAPERCLIP_PLAYWRIGHT_CHANNEL=chrome pnpm exec playwright test
--config tests/e2e/playwright.config.ts
tests/e2e/runner-e2e-dashboard.spec.ts`
- `pnpm -r typecheck`
- `pnpm build`
- Regenerated the dashboard from retained evidence for Actions run
`33968240659` without a paid matrix rerun. The public-stage proof
contained 121 screenshot gallery items and thumbnail frames, with zero
generated HTML report files. The trusted-fixture marker and route gate
have separate focused tests.
- All pull request CI checks pass on commit `ccd2b49e1`.
## Risks
- This change intentionally makes marked fixture screenshots public at
the campaign URL. A screenshot can show data that a raw-byte secret scan
cannot detect.
- The capture helper marks a screenshot only on the exact loopback issue
route for the fixture that the harness created. A different issue,
sensitive page, or external origin stays private.
- The publisher also requires the marker, a safe normalized path, a
valid PNG signature, and the size limit.
- The change does not publish videos, logs, traces, archives, arbitrary
images, or generated browser report trees.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex, model `gpt-5.6-sol`, with high reasoning, repository
tool use, shell execution, browser inspection, and GitHub CLI access.
The working context was the Codex desktop task context.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The full-stack runner campaign checks local and Daytona runner
behavior.
> - A Daytona image content miss starts a cold multi-stage Docker build.
> - Stable dependency and agent CLI layers take most of the image build
time.
> - Development targets must not write shared cache state.
> - This pull request adds a registry cache with a default-branch write
gate.
> - It also puts volatile source inputs after stable install layers.
> - The benefit is a shorter Daytona image build without weaker secret
isolation.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
This improves the Daytona runner image stage in the full-stack E2E
workflow.
**Subsystem affected**
The GitHub Actions runner E2E workflow and its Daytona Docker image are
affected.
**Current behavior**
Each new Daytona image content ID starts with an empty BuildKit cache. A
runner source change also invalidates dependency and agent CLI install
layers because volatile inputs occur before those layers.
**Proposed behavior**
All authorized campaigns can read one GHCR BuildKit cache. Only a
campaign whose target ref is the repository default branch can update
that cache. The Dockerfile installs dependencies and agent CLIs before
it consumes volatile runner source or revision metadata.
**Reason and benefit**
The paid runner matrix spends several minutes building the image before
any selected cell can start. Cache reuse removes repeated stable setup
work and makes focused Daytona iterations faster.
**Breaking changes**
None. The immutable content tag, digest inspection, Cosign signature,
image labels, pinned base images, and provider credential boundary stay
unchanged.
## What Changed
- Read a registry-backed BuildKit cache for Daytona image content
misses.
- Export the cache only when the resolved target ref is the default
branch.
- Keep provider credentials outside the image build and cache.
- Install provider-pack dependencies before runner source is copied.
- Keep expensive agent CLI installs before source revision metadata.
- Add workflow and Docker layer-order contract checks.
## Verification
- `prettier --write .github/workflows/runner-full-stack-e2e.yml
tests/runner-e2e/daytona-image.test.ts
tests/runner-e2e/workflow-security.test.ts`
- `actionlint .github/workflows/runner-full-stack-e2e.yml`
- `git diff --check`
- I did not run a test suite or Docker image build locally. The
requested iteration policy reserves those checks for GitHub Actions.
## Risks
Low risk. BuildKit can use a cache record only when its content key
matches the build instruction and input. Development targets have
read-only cache access. The cache contains public source and build
outputs, but it does not receive provider credentials or the GitHub
token as Docker build inputs.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex with GPT-5, tool use, and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip manages AI agents and their work.
> - The runner executes agent turns on local and remote providers.
> - A remote per-turn session must save its state before Paperclip
releases its sandbox.
> - The session runtime returned after 100 milliseconds while the remote
checkpoint still ran.
> - The next turn also checked the local state path instead of the
verified remote backup.
> - This pull request waits for the bounded remote close and accepts
only a verified suspended backup.
> - The benefit is reliable multi-turn execution without weaker identity
checks.
## Linked Issues or Issue Description
**What happened?**
A successful remote agent turn released its sandbox before the runner
saved the verified continuation backup. The next turn failed with
`runner_state_identity_mismatch`.
**Expected behavior**
Paperclip must finish the bounded remote checkpoint before it releases
the sandbox. A later turn must validate and restore the digest-matched
suspended backup.
**Steps to reproduce**
1. Run a native ACPX Claude Plan test in a non-reusable Daytona sandbox.
2. Reject the first plan to start a second turn.
3. Observe that the second turn fails before provider execution.
**Paperclip version or commit**
The failure reproduced at `13775a90b078ff64872f50961ea1b83d575e7bc6`.
**Deployment mode**
GitHub Actions with a Daytona sandbox.
## What Changed
- Wait for the internally bounded remote runner close and checkpoint
before the host returns.
- Preserve the existing short cleanup bound for other providers.
- Validate remote continuation lifecycle from a complete digest-verified
backup when local runner state is absent.
- Keep corrupt, non-suspended, mismatched, and unverified state
fail-closed.
- Make native Plan completion and accepted-Plan wake prompts
deterministic.
## Verification
- A prior 45-cell local campaign passed 44 cells. The only failure was
the OpenCode Plan prompt variance fixed here.
- A focused OpenCode local Plan rerun passed.
- ACPX Claude Daytona message and question cells passed.
- Focused regressions cover delayed checkpoint close and verified remote
backup lifecycle.
- GitHub Build and the focused ACPX Claude Daytona Plan cell will
validate this exact head.
## Risks
Remote runnerd sessions now wait for their internally bounded
close/checkpoint path before returning; generic provider cleanup retains
the existing 100 millisecond bound. Durable run success still cannot be
reversed. The environment release guard still blocks sandbox destruction
when no verified backup stamp exists.
## Model Used
OpenAI Codex, GPT-5.6, extended reasoning, with code execution and
GitHub Actions inspection.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal task
id
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [ ] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Pull request CI runs through `pr.yml`, which pins the reusable
`pr-trusted.yml` workflow by commit SHA, so `pr-trusted.yml` changes
take effect only when the pin moves.
> - PRs #12855 and #12858 added the `docker_context_integrity` job and
wired it into the `verify` aggregate, but the pin still points at a
commit from before them.
> - Until the pin moves, a pull request that strips a committed Docker
build input still merges green and breaks every post-merge image build.
> - This pull request bumps the pin to the #12858 merge commit, the
standard second step of every `pr-trusted.yml` change.
> - The benefit is that the Docker context integrity gate now blocks
merges, which closes out the 2026-09-04 image-publishing incident end to
end.
## Linked Issues or Issue Description
Refs #12855 and #12858 (the gate this activates) and #12769 (the
incident that motivated it).
**What happened?**
The `docker_context_integrity` job exists on master but does not run on
pull requests, because `pr.yml` pins `pr-trusted.yml` at `a0a78ee6`,
which predates it.
**Expected behavior**
Pull requests run the gate, and the `verify` required check fails when a
change strips a committed Docker build input.
**Steps to reproduce**
1. Open any pull request before this change: the ci run shows no "Docker
context integrity" job.
2. After this change, the job runs on every full-CI pull request and
`verify` requires its result.
**Paperclip version or commit**
Pin moves from `a0a78ee60946a5f79f85b2bd0584fc766fae43bb` to `03609aa6`
(the #12858 merge commit).
**Deployment mode**
GitHub Actions pull request CI.
## What Changed
- `.github/workflows/pr.yml`: the `pr-trusted.yml` pin moves to the
#12858 merge commit. One line.
## Verification
- The pinned commit is master's current tip and contains the job, the
`verify` wiring, and both probe revisions; its own CI (on #12855 and
#12858) is fully green.
- This PR's ci run itself executes the newly pinned workflow, so the
gate's first live run is visible on this very pull request.
## Risks
- Low. Identical mechanism to every previous pin bump. If the new lane
misbehaves on some runner, reverting this one line restores the previous
pin.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
Claude Fable 5 (Anthropic, model id `claude-fable-5`), extended
thinking, agentic tool use in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (one-line pin bump; the
pinned workflow's tests ran green on #12855/#12858)
- [x] I have added or updated tests where applicable (covered by the pin
tests updated in #12855)
- [x] I have updated relevant documentation to reflect my changes (not
applicable to a pin bump)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub Actions builds the Docker images that ship Paperclip, and
downstream deployments consume the `-cloud` image variant on every
master merge.
> - PR #12769 slimmed the Docker build context with a broad
`.dockerignore` block for `packages/paperclip-runner`, and the block
also removed three files the image build itself reads.
> - The image build re-runs the runner's generated-file drift checks, so
it found no committed capability contract in the context and failed on
every master commit after the merge.
> - PR CI never runs those checks against the Docker context, so the
pull request stayed green and the breakage only appeared post-merge, on
every image build.
> - This pull request restores the three files with narrow
`.dockerignore` exceptions and adds a PR CI job that runs the drift
checks against the exact Docker build context.
> - The benefit is that image publishing works again now, and the next
context-slimming regression fails the pull request instead of every
post-merge image build.
## Linked Issues or Issue Description
Refs #12769 (the context-slimming change that exposed this) and #12608
(which committed the generated contract outputs the image build checks).
**What happened?**
Every `Docker` workflow run on master failed from 2026-09-04 12:58Z
onward, in both the `build-and-push` and `build-and-push-cloud` jobs.
The failing step reported `Generated contract drift:
generated/capability/capability-contract.md` from
`check:capability-contract` inside `pnpm --filter @paperclipai/server
build`. The committed contract file is current — regeneration on a full
checkout is a no-op. The file was simply absent from the build context:
the new `packages/paperclip-runner/**/*.md` ignore rule strips the
committed drift-check outputs
(`generated/capability/capability-contract.md`,
`generated/capability/downstream-handoff.md`), and the
`packages/paperclip-runner/docs` rule also strips
`docs/capability-contract.md`, which `check:capability-inventory` reads
next in the chain. No cloud image published for eight hours, which
stalled every downstream deployment that consumes the canary images.
**Expected behavior**
The Docker build context must contain every file the image build reads,
and a change that removes one must fail the pull request that introduces
it, not every image build after the merge.
**Steps to reproduce**
1. Check out master at any commit from `af3023f1` onward.
2. Run `docker buildx build -f .github/docker-context-checks.Dockerfile
.` (the probe added by this PR), or start the real `Docker` workflow
build.
3. Observe `Generated contract drift:
generated/capability/capability-contract.md` — while `node
packages/paperclip-runner/scripts/generate-capability-contract.mjs
--check` passes on the same checkout outside Docker.
**Paperclip version or commit**
`d593463ab` (master tip at diagnosis time; first failing commit
`af3023f1`).
**Deployment mode**
GitHub Actions image builds (`docker.yml`), consumed by managed cloud
deployments.
## What Changed
- `.dockerignore`: narrow exceptions (last match wins) re-include the
committed drift-check outputs
(`!packages/paperclip-runner/generated/**`) and the inventory check's
documentation input
(`!packages/paperclip-runner/docs/capability-contract.md`). Every other
exclusion from #12769 stays: no crate declares an explicit `[[test]]`
target, so cargo builds without the `tests` directories, and the image
build chain never runs the excluded smoke scripts.
- `.github/docker-context-checks.Dockerfile` (new): a small probe that
COPYs the real build context — identical `.dockerignore` semantics — and
runs the dependency-independent drift checks inside it
(`generate-capability-contract.mjs --check`,
`check-capability-inventory.mjs`). ajv installs in an isolated directory
for schema validation only; codegen checks such as
`generate-protocol-schema-module` stay out because their emitted bytes
vary with the ajv release and would raise false drift alarms outside the
locked dependency tree.
- `.github/workflows/pr-trusted.yml`: new `docker_context_integrity` job
builds the probe on every full-CI pull request, and the existing
`verify` aggregate now requires its result, so the guard gates merges
through the same required check as the other lanes.
- Activation note: `pr.yml` pins `pr-trusted.yml` by commit SHA, so the
new job starts gating pull requests after the usual follow-up `ci:
activate ...` pin bump once this merges. The `.dockerignore` fix needs
no activation — `docker.yml` reads it directly, so image builds recover
on the first master commit after this merges.
## Verification
- `docker buildx build -f .github/docker-context-checks.Dockerfile .` on
master (before the `.dockerignore` fix): fails with the exact production
error, `Generated contract drift:
generated/capability/capability-contract.md`.
- Same command with the `.dockerignore` exceptions applied: passes,
which also proves BuildKit honors the `!` exceptions, including the file
inside the excluded `docs` directory.
- `node scripts/generate-capability-contract.mjs --check` on a full
checkout: passes both before and after, which confirms the committed
contract was never stale — only missing from the context.
- Static sweep of every script in the image build chain (`build`,
`build:typescript` and their `check:*` steps) against the ignore rules:
the three restored files are the only build inputs the #12769 block
strips.
- YAML for `pr-trusted.yml` lints clean.
## Risks
- Low. The `.dockerignore` exceptions only re-add three committed files
to the build context; image contents do not change otherwise.
- The probe job adds one context transfer and two Node scripts per
full-CI pull request run (about one to two minutes, no dependency
install beyond one isolated ajv package).
- The `verify` aggregate now also requires the new job, mirroring the
existing pattern for the other lanes; on non-full-CI runs the job skips
and `verify` asserts the skip, unchanged from how the other lanes
behave.
- The new job only takes effect for pull requests after a follow-up pin
bump in `pr.yml` (same two-step flow as every `pr-trusted.yml` change).
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
Claude Fable 5 (Anthropic, model id `claude-fable-5`), extended
thinking, agentic tool use in Claude Code: GitHub Actions log forensics
to isolate the failing check, static analysis of the build-chain scripts
against the ignore rules, and local docker buildx runs to reproduce the
failure and verify the fix.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the docker probe, both
failing-before and passing-after; the drift checks themselves on a full
checkout)
- [x] I have added or updated tests where applicable (the probe IS the
regression test for this class)
- [x] I have updated relevant documentation to reflect my changes
(inline comments in `.dockerignore` and the probe explain the invariant)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
Paid cells now reuse the AWS image's system Chrome, but Playwright video
recording still resolves its revision-pinned FFmpeg helper from the
Playwright cache. Run 33875618534 proved Chrome qualification succeeds
and then failed before provider startup because that helper was absent.
The same run also exposed that generic lock repair can churn unrelated
package platform metadata, so the automated repair paths need
resolution-only regeneration rather than lockfile-only metadata refresh.
## What Changed
- install Playwright FFmpeg only on the AWS/system-Chrome path
- retry the small helper installation up to three times before provider
secrets are exposed
- keep the GitHub-hosted Chromium fallback unchanged
- bind static coverage to the exact FFmpeg step block and its pre-secret
ordering
- add pnpm `--resolution-only` to all four automated lock-repair paths
while retaining full transitive resolution
- require resolution-only repair in the shared workflow regression
The actual generated lockfile correction remains bot-owned by PR #12828
and is intentionally not committed here.
## Verification
- `node --test
.github/scripts/tests/lockfile-refresh-workflows.test.mjs`
- `actionlint -ignore SC2012` on all modified workflows
- focused Prettier checks
- `git diff --check`
- prior run 33875618534: system Chrome 151 qualified; missing Playwright
FFmpeg was the sole cell startup failure
## Risks
Low. The new network operation is limited to Playwright's pinned FFmpeg
payload, happens before paid credentials are exposed, and leaves the
hosted-runner path unchanged. Resolution-only is still a full
dependency-resolution pass, unlike lockfile-only, while avoiding
unrelated current-platform metadata churn.
## Model Used
GPT-5
## Thinking Path
> - Paperclip manages AI agents that perform work.
> - Paperclip Runner connects durable task runs to local provider
processes.
> - The full-stack paid matrix exposed failures after the runner
integrity repair.
> - Verified JavaScript entrypoints lost their relative module graph
when Linux executed them through descriptor paths.
> - Returned provider startup errors also remained pending and became
indeterminate after recovery.
> - Sparse Codex tool lifecycle events lost the `write_document`
identity before task transcript projection.
> - This pull request repairs those three boundaries and makes the
structured-question fixture deterministic.
> - The benefit is repeatable provider startup, exact failure replay,
and correct inline Plan placement.
## Linked Issues or Issue Description
Refs #12721 and #12700.
**What happened?**
The paid runner matrix failed ACPX and OpenCode startup before provider
session creation. The runner journal then replaced the original startup
error with an indeterminate recovery result. Native Codex saved a Plan
but rendered it only as a fallback card. A legacy Claude waiting reply
could also echo the reserved terminal marker before the answer arrived.
**Expected behavior**
Verified JavaScript providers must start from immutable
descriptor-backed artifacts. Returned startup failures must persist as
terminal failed command results. Native tool lifecycle updates must
preserve the `write_document` boundary. Pre-answer fixture output must
not contain the reserved terminal marker.
**Steps to reproduce**
1. Run the local provider cells in the Runner Full-Stack E2E workflow.
2. Observe ACPX and OpenCode fail during `session.open` before provider
execution.
3. Observe recovery report `execution_indeterminate` instead of the
original startup error.
4. Run the native Codex Plan cell and observe the fallback Plan card
after the tool activity row.
5. Run the legacy Claude structured-question resume cell and observe an
early marker echo in waiting prose.
**Paperclip version or commit**
`0f9452101740835ce0b1488a204bf48acd5bafc3`
**Deployment mode**
Local development with the paid GitHub Actions acceptance workflow.
## What Changed
- Bundle the ACPX sidecar and OpenCode proxy as self-contained Node ESM
entrypoints before hashing and verified descriptor launch.
- Anchor ACPX dynamic provider package resolution at a
controller-derived provider-pack root and keep that root out of the
provider child environment.
- Persist executor-returned startup errors as redacted durable failed
command results while retaining indeterminate recovery for true process
death.
- Coalesce sparse native tool items by stable ID so a late
`write_document` name, input, and result reach the transcript boundary
once.
- Forbid the structured-question fixture from spelling or announcing its
reserved terminal marker before the user answers.
## Verification
- Rust and TypeScript regression tests cover durable failed replay, true
crash ambiguity, bundle closure, package-root derivation, environment
filtering, exact Codex tool lifecycle coalescing, and prompt
determinism.
- Local execution is intentionally limited to formatters and static diff
checks. GitHub Actions will run tests, type checks, builds, and security
checks.
- After ordinary CI is green, scoped paid cells will validate one ACPX
launch, one OpenCode launch, native Codex Plan projection, and legacy
Claude structured resume before a complete matrix rerun.
- Prior failing matrix:
https://github.com/paperclipai/paperclip/actions/runs/33682434315
## Risks
- Bundling changes the bytes covered by provider launch hashes.
Provider-pack generation already hashes the final built files.
- ACPX still loads qualified provider packages dynamically. The
controller supplies a normalized package root, while existing version,
digest, path, and descriptor checks remain active.
- Durable `failed` is terminal. Replays return the same redacted result
and do not execute the provider effect twice.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex based on GPT-5 with agentic reasoning, repository
inspection, code editing, Git, parallel subagents, and GitHub Actions
coordination. The exact deployed snapshot and context-window size are
not exposed to this task.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked related public work or described the bug in
this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket id
- [ ] I have run tests locally and they pass (intentionally deferred to
GitHub Actions)
- [x] I have added or updated tests where applicable
- [x] No documentation change is required for this runtime repair
- [x] I have considered and documented the risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip validates every change through an immutable reusable PR
workflow.
> - That caller still pinned a revision that ran pnpm setup before Node
setup.
> - The implementation fix in #12808 is therefore present on master but
inactive for ordinary PR CI.
> - Advancing the immutable caller pin activates the already tested
Node-first workflow.
> - A focused contract prevents the caller from silently returning to
the old revision.
> - The benefit is a faster PR feedback loop without changing product
code or secret boundaries.
## Linked Issues or Issue Description
Refs #12808
## What Changed
- Pin ordinary PR CI to trusted workflow revision
`a0a78ee60946a5f79f85b2bd0584fc766fae43bb`.
- Assert that the reusable workflow call is canonical, unique, and
SHA-pinned to that audited revision.
## Verification
- Focused workflow security test: 8/8.
- Prettier passed.
- Actionlint passed.
- `git diff --check` passed.
## Risks
Low risk. The change only advances an immutable reusable-workflow pin to
a revision whose full ordinary CI and security checks passed. Product
code and credentials are unchanged.
## Model Used
OpenAI Codex GPT-5 with agentic reasoning and repository tool use.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have linked the related public PR
- [x] I have not referenced internal issue links
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have considered and documented risks
## Thinking Path
The trusted workflows currently invoke `pnpm/action-setup@v6` before
installing the repository Node version. On hosts whose ambient Node is
older than 22.13, the action downloads standalone `@pnpm/exe`, which has
repeatedly taken several minutes. Supplying Node 24 first lets the same
pinned pnpm action use its normal Node-backed path.
## What Changed
- install Node 24 before every trusted `pnpm/action-setup` invocation
- preserve the existing pnpm-store cache setup, pinned actions,
telemetry suppression, conditions, and secret boundaries
- enforce ordering, modern Node, condition parity, and cache counts in
the workflow security contract
## Verification
- workflow security: 7/7
- Prettier
- actionlint (excluding one pre-existing SC2129 in an untouched Daytona
shell block)
- `git diff --check`
## Risks
Low. Product code, providers, paid-runner selection, credentials, and
pnpm version are unchanged. Jobs that restore pnpm cache run
`setup-node` a second time after pnpm becomes available; the first setup
is deliberately cache-free.
## Model Used
Codex (GPT-5)
## Thinking Path
Trusted workflow-dispatch runs execute an authorized target SHA, but
GitHub context still describes the default-branch workflow revision.
Retained paid results and artifact names were therefore labeling
target-branch executions as master. The workflow must explicitly pass
its authorized target coordinates to target code and trusted reporting.
## What Changed
- emit the canonical authorized target ref alongside the immutable
target SHA
- pass those coordinates to paid cells and the trusted report
- name shared build/provider artifacts with the target SHA rather than
workflow SHA
- add workflow-security coverage for all trusted provenance wiring
## Verification
- focused workflow-security tests: 6/6 passed
- Prettier and git diff checks passed
- run 33823252706 independently proved the pre-fix defect: functionally
green target cells were retained as master SHA 0ad180b85 instead of
feature SHA 33c7646d3
## Risks
The execution checkout and secret boundary were already pinned
correctly; this changes retained attribution and artifact labels only.
Target-side report code on PR #12769 consumes these trusted environment
values and overwrites untrusted cell metadata.
## Model Used
Codex (GPT-5)
## Thinking Path
Every trusted PR and paid-workflow job invokes the pinned pnpm setup
action. Its internal npm install is currently waiting four to seven
minutes on npm audit telemetry before any Paperclip or provider code
runs. Audit, funding, and update notifications are not integrity
controls for this action; its committed lockfile still verifies
installed package bytes.
## What Changed
- disable npm audit, funding, and update-notifier telemetry narrowly on
all seven pinned setup steps in each of the trusted PR and full-stack
workflows
- add a workflow security contract proving every setup invocation
remains covered and the overrides do not leak elsewhere
## Verification
- focused workflow security tests: 6/6 passed
- Prettier and git diff checks passed
- observed unhealthy setup: 4-7+ minutes; historical healthy setup:
about four seconds
## Risks
This skips npm vulnerability-report telemetry for the setup action
bootstrap only. Repository dependency checks, lockfile integrity,
provider-secret authorization, and target-lock verification remain
unchanged.
## Model Used
Codex (GPT-5)
Devin Foley
Dotta
Nicky Leach
Tonio
