## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI installs dependencies before it verifies and builds cloud
artifacts.
> - Install jobs share a pnpm package-store cache with lockfile refresh.
> - Lockfile refresh resolves versions without downloading packages.
> - That job saved an empty cache before full install jobs could save
theirs.
> - This PR prevents lockfile refresh from publishing that empty entry.
> - Full install jobs can then populate the cache and reuse
dependencies.
## Linked Issues or Issue Description
Refs #13259 for the related cloud verification cache work. No duplicate
empty-cache fix was found.
**What happened?**
Refresh Lockfile run 34517514932 saved a 216-byte default-branch pnpm
cache at 18:58:08 UTC on September 10. Full install jobs still restore
that empty entry. The cache API reports 216 bytes for master and about
703 MB for populated entries with the same key and cache version in PR
scopes.
**Expected behavior**
A job that installs dependencies should populate the shared
package-store cache.
**Steps to reproduce**
1. Run lockfile refresh with a new lockfile cache key.
2. Its resolution-only command leaves the package store empty.
3. The Node action saves the empty archive before a full install
finishes.
4. Later jobs report a cache hit but download packages again.
**Paperclip version or commit**
Observed on master 6728e133f8 and still
present at a23ae894a5.
**Deployment mode**
GitHub Actions cloud verification and release workflows.
## What Changed
- Disable package-manager caching in Refresh Lockfile.
- Document how to remove the existing empty default-branch entry and
verify a populated replacement.
- Add regression coverage for explicit and automatic package-manager
cache selection in a resolution-only job.
## Verification
- actionlint and git diff checks pass.
- All 175 existing workflow-script tests pass. Both new regression cases
pass and fail against the original workflow, covering the explicit pnpm
cache and automatic npm cache paths. This change adds no application
behavior.
- [The cache creator
job](https://github.com/paperclipai/paperclip/actions/runs/34517514932/job/103006542158)
logs a 216-byte upload under the same key still used by cloud
verification.
- The batch-wide local full typecheck and build passed. The local full
test run reported 10,600 passed, 65 skipped, and 13 permission failures
in unchanged runtime-skill suites. These checks were not repeated in
this dependency-free worktree. Current-head Linux CI passes. The
unchanged chat and browser suites passed on their single retry; all
final checks are green. Greptile is 5/5 with all threads resolved.
- After merge, delete only the existing empty master cache entry. Verify
that a master install saves a populated archive and subsequent jobs
reuse packages. Measure the net install-time change before claiming a
latency gain.
## Risks
- Lockfile resolution can require fresh registry metadata. It does not
need a cached package store.
- The existing empty cache must be removed once; this change prevents
its recreation by this workflow.
- Cache benefits vary with download speed and archive extraction time.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused workflow checks
pass; the batch-wide local test limitation is disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployments consume a verified image and exact-source
migrator.
> - An image alone is not deployable until source checks and artifact
checks pass.
> - GitHub-hosted queues delayed those checks and the final readiness
signal.
> - This PR gives trusted master work a separate concurrency allowance
on existing AWS runners.
> - Community PRs and arbitrary source inputs keep the GitHub-hosted
fallback.
## Linked Issues or Issue Description
Refs #13243.
**What existing behavior does this improve?**
Time from a master merge to the Cloud deployable v1 signal.
**Current behavior**
For merge d0b7ba4, the image was available after 7m 42s, but readiness
took 16m 05s. Typecheck queued for 6m 40s and the final readiness job
queued for 1m 46s.
**Proposed behavior**
Allow up to 36 concurrent post-merge verification and migrator jobs on
the existing four-vCPU, 16-GiB AWS runners. Workers launch on demand and
terminate after their job; no always-on worker pool or AWS Reserved
Instance purchase is introduced. Keep the combined runner ceiling
unchanged. A separate operator switch enables this route only after the
restricted runner group and Fleet exist.
**Reason and benefit**
Remove GitHub-hosted queue delays from the cloud deployment path. The
gain depends on queue pressure and which remaining job finishes last;
the observed queues are not additive savings.
**Breaking changes**
None to source verification or readiness contracts. Paid routing is
limited to canonical master push/manual events, with exact source checks
on reusable and migrator jobs.
## What Changed
- Route source verification, artifact waiting, dispatch, and readiness
jobs to the separate post-merge Fleet when enabled.
- Require source inputs to match the event's master SHA. Preview inputs
and raced older migrator dispatches stay GitHub-hosted.
- Keep npm publication on GitHub-hosted runners for trusted publishing.
- Bound AWS job timeouts below the 45-minute instance lifetime.
- Document activation, capacity reservation, and rollback.
- Exercise each actual runner selector against allowed and rejected
event/source combinations.
## Verification
- 268 routing and timeout cases pass, including unapproved PR, fork,
branch/tag, arbitrary ref, and disabled-switch cases.
- All 461 focused workflow, preview, and readiness tests pass. The 284
routing/preview cases also pass after the review fixes.
- actionlint passes for changed workflows with the existing
SC2012/SC2016/SC2129 warnings excluded.
- Full local typecheck and build pass (167s and 206s). `pnpm test:run`
completed: 10,600 passed, 65 skipped, and 13 failed in the unchanged
company-skills-service/runtime-skill-cache suites with local filesystem
permission errors. Linux CI is the required test gate; this is not a
claim of a fully passing local suite. Current-head Linux CI is green,
Greptile is 5/5, and all findings are resolved. The final Build retry
passed on a verified 60 GiB AWS runner after correcting the earlier
disk-capacity failure.
- After activation, verify a master run selects the separate group and
all readiness prerequisites pass.
## Risks
- A missing or incorrectly restricted runner group can leave eligible
jobs queued. Enable the switch only after Fleet and group verification.
- PR bursts have 64 slots after reserving 36 for post-merge work. The
image Fleet retains eight, for the same 108-runner total.
- A migrator dispatch racing a newer merge uses GitHub-hosted runners.
This preserves source trust but can retain some queue delay.
- Roll back placement by disabling AWS_POST_MERGE_CI_ENABLED and
rerunning the whole workflow.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud images become deployable only after source verification
passes.
> - The typecheck job builds the native Runner binary through the server
package.
> - Fresh runners repeatedly compile Rust dependencies for that binary.
> - This PR caches those dependencies for exact-source master
verification.
> - Workspace code and every typecheck still rebuild or run as before.
## Linked Issues or Issue Description
Refs #13243 and #13257.
**What existing behavior does this improve?**
The typecheck portion of post-merge cloud source verification.
**Current behavior**
The typecheck job has no Rust dependency cache. An observed release
build in this job took 4m 13s, including dependency compilation.
**Proposed behavior**
Restore dependency build outputs for canonical master pushes with the
exact source SHA. Use a separate cache key from the Runner verification
job, which builds other profiles.
**Reason and benefit**
A warm cache should remove roughly 2–3 minutes of dependency compilation
from this job. Overall deployment gains depend on the remaining critical
path. The first cache population still compiles from scratch.
**Breaking changes**
None. All checks remain enabled. Non-master callers compile without
restoring or saving this cache.
## What Changed
- Select the pinned Rust toolchain before the typecheck cache lookup.
- Reuse the existing pinned Rust cache action with a typecheck-specific
key.
- Exclude workspace crates and installed cargo executables.
- Test restore/save trust boundaries and document cache behavior.
## Verification
- All workflow script tests pass locally, including nine new cache
trust/contract cases.
- actionlint passes for release-verify.yml.
- Full local typecheck and build pass on the same source base (167s and
206s); `pnpm test:run` is still running and is recorded with #13257.
This PR changes only the workflow, cache guard tests, and documentation.
- All 32 current-head checks are successful or intentionally skipped,
including the complete Linux test matrix, build, and Greptile 5/5 with
no unresolved findings. Verify cache population and subsequent restore
on actual master runs.
## Risks
- The first run and any toolchain/dependency invalidation compile from
scratch.
- Cache restore/save overhead reduces the benefit for small dependency
graphs.
- Disable the cache step to roll back; the existing uncached build
remains valid.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. Exact serving model ID and context window are not exposed by
this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused change tests
pass; full-suite local permission failures are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images built from master commits.
> - Cloud image builds share GitHub-hosted capacity with other
workflows.
> - The organization already operates AWS runners through RunsOn Fleet.
> - This pull request allows approved master builds to use a dedicated
cloud Fleet.
> - The benefit is separate build capacity with a quick operator
rollback.
## Linked Issues or Issue Description
Refs #13189, #13192.
**What existing behavior does this improve?**
Placement of the Docker cloud build after a master merge.
**Current behavior**
Every Docker cloud build uses a GitHub-hosted runner. Busy periods delay
the job.
**Proposed behavior**
An operator variable enables the approved cloud Fleet for canonical
master pushes and manual master builds. Other events, refs, and
repositories use GitHub-hosted runners.
## What Changed
- Add a guarded AWS runner selector to the Docker cloud job.
- Keep the existing image cache, verification, and publication steps.
- Test the selector against master, branch, tag, PR, fork, and disabled
contexts.
- Document provisioning requirements, placement checks, and rollback.
## Verification
- 29 focused Node tests pass for routing, readiness, and disk handling.
- The full workflow-script Node suite passes.
- `pnpm -r typecheck` passes locally.
- The pinned PR routing regression suite passes. The first live PR run
assigned 21 jobs to the approved AWS PR group. AWS then reclaimed 16
Spot instances. The failed run is being repeated on GitHub-hosted
runners while the Fleet moves to On-Demand.
- Actionlint passes with existing shellcheck findings excluded (SC2012,
SC2016, SC2129).
- `git diff --check` passes.
- Greptile reports 5/5 on commit
`764d505a41dd2023751c3f361906fa9ea35bf0c6`, with no review threads.
- All 30 current-head CI checks pass, including typecheck, build, all
server/workspace test shards, Runner verification, and browser tests.
Two Storybook checks are intentionally skipped for this change. Run:
https://github.com/paperclipai/paperclip/actions/runs/34630550799
- The broader local test/build sequence is still running. This Mac has
reported failures in unchanged application suites; their complete Linux
CI shards pass. Local targeted workflow tests and typecheck pass.
- Both On-Demand Fleets are deployed and healthy. Live master
cloud-build verification follows the merge.
## Risks
- Missing Fleet capacity or runner-group authorization can leave an AWS
job queued. Disable `AWS_CLOUD_BUILDS_ENABLED` and rerun the workflow to
use GitHub-hosted capacity.
- The runner group must restrict access to this repository and the
master version of `docker-cloud.yml`.
- Docker needs more disk space than the PR Fleet. Provision 120 GiB
disks and retain the free-space check.
- This changes image build placement only. Source verification and
migrator publication remain separate prerequisites.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact serving model identifier and context-window size
are not exposed by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Reuse the exact master source-verification result before npm canary publication, removing a duplicate verification matrix while preserving fail-closed release checks.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud waits for source verification before deploying a new image.
> - The slowest server verification job spends about ten minutes running
tests.
> - Each job uses one test worker to preserve test isolation.
> - This pull request distributes those suites across ten standard
hosted runners.
> - The benefit is a shorter verification path with the same test
coverage.
## Linked Issues or Issue Description
**Current behavior**
In [readiness run
34572340764](https://github.com/paperclipai/paperclip/actions/runs/34572340764),
the slowest server job ran for 638 seconds. Test execution used 594
seconds. This held readiness behind the image job.
**Proposed behavior**
Use ten general server jobs in the reusable release verification
workflow. Keep the three chat jobs and every existing prerequisite. The
complete partition test verifies that no server suite is omitted or
duplicated.
**Reason and benefit**
Reduce merge-to-deployable time on the existing runner type. The next
longest prerequisite was Runner verification at 526 seconds, so the
initial expected total gain is about two minutes rather than a halving
of readiness time. Measure actual queue and execution time before
claiming a result.
Related: #13198 introduced the separate chat lane. #12577 refreshes
duration estimates; this change leaves that manifest alone.
## What Changed
- Increase the general server matrix from five jobs to ten.
- Verify the ten-way partition covers the complete server suite when
combined with the chat lane.
- Document runner demand and the unchanged local and PR grouping.
## Verification
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs`: 29 passed.
- `actionlint .github/workflows/release-verify.yml`: passed.
- Full local `pnpm -r typecheck` and `pnpm build`: passed.
- All latest-head GitHub CI checks passed, including the complete Linux
test partition, build, typecheck, and browser gates. Greptile: 5/5 with
zero open findings.
- [Ten-shard timing
probe](https://github.com/paperclipai/paperclip/actions/runs/34606772388):
all 16 jobs passed; slowest server job 6m 23s versus 10m 38s in the
earlier five-shard sample. This compares the server lane, not total
readiness, and is not a controlled same-source A/B.
- The full local `pnpm test:run` is also running. It has reproduced
previously observed macOS-only failures in unchanged skill-cache and
native-session suites; the corresponding Linux CI suites passed. Final
local results will be attached separately. No affected-workflow test
failed.
## Risks
Five additional concurrent jobs per release verification run increase
runner demand and repeated setup work. Queueing can offset the gain.
Test workers, timeouts, permissions, and readiness requirements stay
unchanged. Revert the matrix and its partition test to restore the
previous split.
## Model Used
OpenAI GPT-6 / Codex, with reasoning, tool use, and code execution. The
exact serving model identifier and context-window size are not exposed
by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the affected workflow tests locally and they pass;
full-suite macOS limitations are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Managed deployments start from the image built by the Cloud
workflow.
> - The managed runtime requests user and group 1001.
> - The image currently builds the node user as 1000.
> - Startup must remap that user, which can walk a large mounted home
directory.
> - This pull request uses the existing Docker build arguments to bake
user and group 1001 into Cloud images.
> - Matching the runtime identity removes that startup work and helps
avoid health-check retries.
## Linked Issues or Issue Description
Refs #13208, #1923, and #7861. Searched open and closed PRs for the
Cloud UID change. The older #7861 addresses build context and volume
ownership repair. This change uses the existing identity arguments in
the Cloud workflow and preserves ownership repair.
**What happened?**
A measured rollout had a container log `Updating node UID to 1001` after
startup. The container stayed at this step for at least 2 minutes 55
seconds before rollback stopped it. The baked node identity was 1000,
while the managed runtime requested 1001. A health check timed out and
the target required a second deployment attempt.
**Expected behavior**
Cloud images should already have the managed runtime identity. A
matching image should skip user and group remapping. Fresh or mismatched
volumes must still receive ownership repair.
**Steps to reproduce**
1. Build the current Cloud image with its default build arguments.
2. Start it with `USER_UID=1001`, `USER_GID=1001`, and a populated home
volume.
3. Observe the startup user remap before the application starts.
**Paperclip version or commit**
`fc06f7f05f42c675be71ff0927b6334405d520ed`
**Deployment mode**
Docker on managed hosts.
## What Changed
- Pass `USER_UID=1001` and `USER_GID=1001` to the Cloud image build.
- Check the pushed digest's baked identity before the entrypoint can
repair it. Then check the normal entrypoint's effective identity and
writable home before publishing the verified full-SHA tag.
- Add a workflow regression and two entrypoint cases for a matching
Cloud identity, including a mismatched volume.
- Document the runtime identity and the first-build cache cost.
## Verification
- Focused workflow and artifact tests: 27 passed.
- Entrypoint tests: 11 passed. Actionlint passed. Full local `pnpm -r
typecheck` passed. Full local `pnpm build` passed. The manual [Cloud
image
build](https://github.com/paperclipai/paperclip/actions/runs/34575473213)
passed on the exact PR head. It checked Sentry, baked and effective
identity, writable home, orphan reaping, and full-SHA publication. The
new identity check took one second. All 30 PR checks passed; the
Storybook workflow was intentionally skipped. Greptile reviewed commit
`114d408f637a0b53e2e2b1339c263779b1e4ae54` at 5/5 with no findings or
open threads.
- The full local suite for the same application source was already run
in #13205. Its macOS general-server phase had 10,471 passes and 70
failures in seven unchanged files. Those failures included missing
Runner fixtures, filesystem errors, timeouts, a port conflict, and a
load-count mismatch. After configuring Cargo and rebuilding fixtures, 37
of 38 native tests passed; one unchanged native-resume assertion still
failed. Linux PR CI passed. This change adds entrypoint tests and does
not change application code.
## Risks
- The first build must rebuild layers that depend on the base image
identity. Later builds can reuse them.
- A future managed runtime identity change must update these build
arguments and checks together.
- The Dockerfile's self-hosted defaults remain 1000. Runtime overrides
and mounted-volume ownership repair remain supported.
- The observed startup delay supports this change, but fleet timing also
includes provider startup, image pull, canary order, and retries. No
fixed end-to-end gain is claimed before a live rollout.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, code execution, and tool
use. The exact serving model ID and context-window size are not exposed
in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused workflow tests;
full-suite limitations are listed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployments require verified artifacts for the merged source
commit.
> - Cloud readiness and the npm release independently run the same
source checks.
> - Their shared chaos workflow used only the source ref as its
concurrency key.
> - One caller could cancel the other caller's required job for the same
commit.
> - This pull request scopes that key to the caller workflow and source
ref.
> - Both callers can finish their checks without blocking deployment
readiness.
## Linked Issues or Issue Description
Refs #13192 and #13205. Searched for related open issues and PRs; no
duplicate fix was found.
**What happened?**
The master push for `398d304e15739d1ee6105633bd8a0e42c929d33f` started
Cloud readiness and Release together. GitHub cancelled the Cloud
readiness chaos job before it acquired a runner. Its annotation reported
a higher-priority waiting request for the same concurrency group. The
required readiness gate cannot pass after that cancellation.
**Expected behavior**
Cloud readiness and Release must each finish source verification for the
same SHA. Standalone chaos evals must also have a separate group.
**Steps to reproduce**
Merge a commit to master while the npm release queue is empty. Both
callers reach the reusable chaos workflow with the same source SHA. See
[the cancelled
job](https://github.com/paperclipai/paperclip/actions/runs/34569569760/job/103168603926).
**Paperclip version or commit**
`398d304e15739d1ee6105633bd8a0e42c929d33f`.
**Deployment mode**
GitHub Actions on master.
## What Changed
- Add the caller workflow name to the chaos workflow concurrency group.
Retain source isolation and cancellation of duplicate calls within the
same workflow.
- Add a regression test that evaluates the group for Cloud readiness,
Release, and standalone evals at the same source SHA.
- Document the concurrency boundary in the readiness runbook.
## Verification
- `node --test scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs` passed: 26 tests.
- The new regression test fails against the previous concurrency key and
passes with this fix.
- `actionlint -shellcheck= -pyflakes=
.github/workflows/runner-chaos-evals.yml
.github/workflows/release-verify.yml
.github/workflows/cloud-readiness.yml` passed.
- `git diff --check` passed.
- The full local typecheck passed for the same application source in
#13205. Its macOS general-server test phase had 10,471 passes and 70
failures in seven unchanged application test files: missing Cargo/Runner
test binaries, filesystem permissions, timeouts, a port conflict, and a
load-test count mismatch. Linux CI test checks passed. The full local
build passed with Cargo on PATH. This PR changes workflow configuration,
its test, and documentation only.
- All CI checks pass on the final head, including typecheck, tests,
browser suites, build, and canary dry run. Greptile is 5/5 with no open
findings. After merge, verify both callers' chaos jobs complete for the
same master SHA and record the resulting readiness time.
## Risks
- Two callers may now run chaos tests at the same time. This uses two
existing GitHub runners, which is the intended cost of independent
verification.
- Renaming a caller changes its concurrency group. The fixed prefix
keeps this child group separate from caller-level concurrency groups.
- The readiness gate continues to require every verification
prerequisite. No gate is bypassed.
## Model Used
- OpenAI GPT-6 / Codex, with reasoning, repository editing, and
command/API tools. Exact serving model ID and context-window size are
not exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (26 focused
workflow/artifact tests)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted deployments need a verified image and migrator for the same
source commit.
> - The cloud readiness workflow certifies those inputs before a
deployment consumer acts.
> - Its completion time does not show when a tenant runs the new commit.
> - This pull request documents each milestone from merge through target
health and fleet completion.
> - Operators can use the evidence to find the slow stage and measure a
complete deployment.
## Linked Issues or Issue Description
Refs #13192, #13188, and #13189. Searched related issues and PRs; no
duplicate timing documentation change was found.
**Issue type**
Missing documentation.
**Where is the issue?**
`doc/cloud-build-readiness.md`, Timing and rollout.
**What's wrong?**
The timing instructions stop at the readiness job. That omits consumer
queues, artifact resolution, and target deployment. An image can be
ready while the tenant still runs an older commit.
**Suggested fix**
Record separate merge, image, readiness, canary health, and fleet
completion timestamps for the same full source SHA. Keep
preparation-only runs out of deployment results.
## What Changed
- Define the evidence needed for each merge-to-deployment milestone.
- Explain how consumer queues can hide upstream build gains.
- Require target source identity as well as health, and report
exclusions, retries, cache state, and queue conditions.
## Verification
- `git diff --check` passed.
- `node --test scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs` passed: 25 tests.
- Cross-checked the readiness identity and artifact prerequisites
against the current workflows and consumer contract.
- Full local `pnpm -r typecheck` passed using the session's installed
Rust toolchain. The full local test suite and subsequent build are still
running.
- All CI checks pass and Greptile is 5/5 on the exact head, with no
unresolved findings. This changes one documentation file and adds no
runtime behavior.
## Risks
- Low risk: documentation only. Timing must still use trusted run
evidence and the actual target commit. A single measured run is not a
latency guarantee.
## Model Used
- OpenAI GPT-6 / Codex, with reasoning, repository editing, and
command/API tools. Exact serving model ID and context-window size are
not exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (25 focused
workflow/artifact tests; full checks pending)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
Verify source, build the cloud image, and wait for exact-source migrator packages concurrently. Emit Cloud deployable v1 only when every prerequisite succeeds for the merged full SHA.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Devin Foley