Build cloud images independently for each master commit through a reusable workflow. Preserve production release dependencies and image runtime checks, and write cloud registry caches per commit with bounded ancestor imports to prevent overlapping builds from replacing each other's cache.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Publish the full-source-SHA cloud image tag only after the pushed digest passes runtime, revision, and platform checks. This makes verified images directly resolvable by Cloud.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
A fleet rollout failed with `image_manifest_not_found` for a commit that
had merged and gone green. Tracing that back: the fleet resolves
releases against the `-cloud` image, that image comes from `docker.yml`,
and `docker.yml`'s runs on master have been reading `cancelled` for a
long stretch. The cloud half was fine; the production half was hanging
and taking the run down with it — and, because a run holds the
concurrency slot for its whole duration, starving later commits of a
build at all.
## Linked Issues or Issue Description
No tracking issue — described inline, per CONTRIBUTING.md.
**What's wrong.** `build-and-push` builds `linux/amd64,linux/arm64` on
an x86 runner, so arm64 runs under QEMU. It hangs there —
deterministically, in the same step:
```
#111 [linux/arm64 build 8/10] RUN pnpm --filter @paperclipai/server build
```
…then emits nothing until `timeout-minutes: 60` kills it. Three
consecutive runs on 2026-09-04, silent for **38, 43 and 45 minutes**
respectively. The amd64 leg reached `production 5/5` minutes earlier in
every one.
**Why it stayed hidden.** A timed-out job is reported by GitHub as
**cancelled, not failed**. The run conclusion reads "cancelled", which
looks like supersession rather than breakage, so the production image
quietly stopped publishing.
**The knock-on.** A run that burns the full hour holds the top-level
concurrency slot for that hour. `cancel-in-progress: false` keeps
exactly one pending slot, so merges arriving faster than one an hour
supersede each other while queued. Sampling the last ten master commits,
**five produced no image at all** — their Docker runs have zero job
records because they never started.
**Expected.** Both architectures publish, and a commit merged during a
busy period still gets an image.
## What Changed
`build-and-push` becomes a two-leg matrix, each on a runner of its own
architecture:
| platform | runner |
|---|---|
| `linux/amd64` | `ubuntu-latest` |
| `linux/arm64` | `ubuntu-24.04-arm` |
Each leg pushes **by digest** (`push-by-digest=true`, untagged), and a
new `merge-and-push` job names the digests into one manifest list with
the real lane tags. Nothing is publicly tagged until the merge, so a
half-published multi-arch image is never a pullable state.
Two supporting changes:
- **Per-arch BuildKit cache refs** (`:buildcache-amd64` /
`:buildcache-arm64`). Separate runners sharing one ref would overwrite
each other on every build.
- **The PID-1 orphan-reaping check moves to the merge job**, since that
is where a tagged, pullable image first exists. It still runs against
the pushed image rather than a local build, for the same reason as
before.
**arm64 is kept, not dropped.** The cloud variant is amd64-only and can
be — managed hosts are amd64. This is the self-hosted image and ARM
hosts consume it, so dropping arm64 would break them. GitHub-hosted
arm64 runners are free for public repositories, which this is.
`build-and-push-cloud` is untouched. It was already `platforms:
linux/amd64` and has been succeeding in ~14 minutes throughout — that is
why `-cloud` images exist at all.
## Verification
Parsed the workflow and asserted its shape (jobs, matrix, `needs`, step
order, that the cloud job is unchanged). The artifact actions are pinned
by SHA with version comments, matching the repo's dominant convention —
`upload-artifact` v7 and `download-artifact` v8, the same pins used
across the other workflows; v8 is required for the `pattern` /
`merge-multiple` inputs the merge job uses.
**This PR's CI does not exercise the change.** `docker.yml` triggers on
master and tag pushes, never on pull requests — deliberately, since it
publishes release images. The first real run is after merge, so the
check is: the next master push produces a `Docker` run whose
`build-and-push (amd64)`, `build-and-push (arm64)` and `merge-and-push`
jobs all succeed, and whose conclusion is `success` rather than
`cancelled`.
## Risks
- **Not testable before merge**, per above. If the matrix is wrong the
next master push fails loudly rather than silently — which is already
better than the current state, where the failure mode is an invisible
"cancelled".
- **First use of `ubuntu-24.04-arm` in this repo.** No other workflow
uses an ARM runner. They are free for public repos, but if the label is
unavailable the arm64 leg will fail to schedule and the merge will not
run — no image, same as today, and visible.
- **Digest-push changes the publish shape.** Between the legs finishing
and the merge running, digests exist untagged in ghcr. Anything watching
for tags sees no intermediate state; anything enumerating untagged
manifests will see more of them.
- **Cache refs change name**, so the first build after this lands is
cold on both legs and will be slower than steady state.
- **Does not fix the underlying QEMU hang** — it avoids it. If arm64
ever has to build under emulation again, the same stall is presumably
still there.
- No application code, schema, server or persistence change.
## Model Used
Anthropic Claude — Opus 5, model ID `claude-opus-5`, run through Claude
Code.
Extended thinking enabled. Tool use throughout: GitHub Actions API to
correlate run/job outcomes and read build logs, `git` for ancestry
checks, and a YAML parser to validate the rewritten workflow's
structure.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
A workflow change has no unit test to add, and `docker.yml` cannot run
on a PR; the verification section states what to check on the first
master run instead.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin Foley <devin@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Docker workflow publishes the server images that all deployments
pull, including the `sha-*` images that downstream consumers deploy
> - The `build-and-push` job builds for linux/amd64 and QEMU-emulated
linux/arm64, and that build now takes more than its 60 minute job
timeout
> - Every run dies at the timeout, and each doomed hour-long run holds
the per-ref concurrency slot, so queued master pushes supersede each
other and no image publishes at all
> - This pull request raises the multi-arch job timeout to 120 minutes
> - The benefit is that image publishing works again, with headroom for
the build to grow
## Linked Issues or Issue Description
Related: #12821 replaces the QEMU-emulated arm64 build with native
runners — that is the durable fix for the build duration itself. This PR
is the immediate unblock so images publish again while #12821 lands.
No existing issue for the outage. Description follows the bug report
template:
**What happened?**
The `docker.yml` `build-and-push` job hits its 60 minute
`timeout-minutes` cap on every run. The last fully successful
`docker.yml` run was September 2. Since then almost every run ends
`cancelled`: the multi-arch build is killed at the timeout, and runs
queued behind it are superseded by newer master pushes before they can
start. The amd64-only `build-and-push-cloud` job often still succeeds
inside those cancelled runs, which masked the breakage.
**Expected behavior**
Every master push and canary tag dispatch publishes its `sha-*`
production and cloud images, and the `promote_canary_channel` job runs.
**Steps to reproduce**
Look at the runs of the Docker workflow on master: `gh run list
--workflow docker.yml --branch master`. Nearly every run since September
5 ends `cancelled` or `failure`. Open a cancelled run: the
`build-and-push` job runs for 61+ minutes and its "Build and push" step
ends `cancelled` at the job timeout. The last runs that succeeded
(September 2) took 39 to 54 minutes for the same job.
## What Changed
- Raise `timeout-minutes` on the `build-and-push` job from 60 to 120,
with a comment that explains why. The amd64-only `build-and-push-cloud`
job keeps its 60 minute cap.
## Verification
- `actionlint .github/workflows/docker.yml` reports no issues in this
change (only pre-existing info-level shellcheck notes in untouched
steps).
- Compared job durations across the last successful runs (39-54 minutes)
and the recent timeout kills (61+ minutes) to confirm the cap is the
failure cause.
- After merge, the next master push should produce a `docker.yml` run
that completes with both build jobs green.
## Risks
Low risk. The change only gives the existing build more time. A
genuinely hung build now occupies a runner for up to 120 minutes instead
of 60. The slow arm64 emulated build itself is worth a separate look
(native arm runners or splitting the platforms), but that is a larger
change than this outage fix.
## Model Used
Claude (Anthropic) — Fable 5 (`claude-fable-5`), extended thinking,
agentic tool use via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no code paths changed;
workflow linted with actionlint)
- [x] I have added or updated tests where applicable (not applicable for
a CI timeout value)
- [x] I have updated relevant documentation to reflect my changes (the
workflow comment documents the rationale)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every master merge publishes an npm canary, and downstream managed
deployments consume the matching `sha-<short>-cloud` Docker image.
> - The canary image relies on the master-push `docker.yml` run, whose
single pending concurrency slot is superseded by every newer push.
> - On a busy day the image build never survives: five consecutive
canaries shipped npm packages with no cloud image on 2026-09-06, and
downstream deploys starved for ~18 hours while master kept moving.
> - The nightly and beta lanes already solve exactly this by dispatching
`docker.yml` at their tag ref after pushing it.
> - This pull request wires the canary lane into the same mechanism, so
every canary tag gets an image build no master push can supersede.
> - The benefit is that a published canary always has its images, and
downstream release resolution stops walking back to day-old builds
during busy merge windows.
## Linked Issues or Issue Description
Refs #12769 / #12855 (the earlier image-publishing incident in the same
pipeline, different failure mode).
**What happened?**
`docker.yml` serialises per ref with one pending slot (`concurrency:
docker-${{ github.ref }}`, `cancel-in-progress: false`). Master pushes
arriving faster than the ~50-minute build supersede the pending build
indefinitely. On 2026-09-06, canaries `2026.906.0-canary.1` through `.3`
(and the commits between) published to npm with no `sha-<short>-cloud`
image on GHCR — the Docker run list shows `cancelled, cancelled,
cancelled` for their commits. Downstream managed rollouts correctly
refused to deploy image-less canaries and pinned at `canary.0` for ~18
hours.
**Expected behavior**
Every published canary has its Docker images. A busy merge window must
not be able to prevent image publication for tagged releases.
**Steps to reproduce**
1. Merge to master more often than the Docker build takes to complete,
for several hours.
2. Observe npm canaries advancing while every `Docker` run for their
commits completes `cancelled`.
3. Observe `ghcr.io/paperclipai/paperclip:sha-<short>-cloud` returning
404 for each of those canaries.
**Paperclip version or commit**
Master at `83987210` (diagnosis time); the starved canaries were
`2026.906.0-canary.1`–`.3`.
**Deployment mode**
GitHub Actions release + image pipeline; consumed by managed cloud
deployments.
## What Changed
- `.github/workflows/release.yml`, `publish_canary` job:
- New step after "Push canary tag": dispatch `docker.yml` at
`refs/tags/canary/v<version>` — identical to the nightly and beta lanes'
existing step, including the step-summary note. The dispatched run keys
its concurrency off the tag ref, so master pushes cannot supersede it,
and `docker.yml`'s `type=sha` mapping publishes `sha-<short>` /
`sha-<short>-cloud` for any ref.
- The job gains `actions: write`, mirroring `publish_nightly`.
- No `docker.yml` changes: it already accepts `workflow_dispatch` for
exactly this pattern.
- No dry-run guard needed: `publish_canary` runs only on `push` events,
so the `dry_run` dispatch input cannot reach it.
## Verification
- YAML lints clean; the step is a line-for-line mirror of the proven
nightly-lane step (tag source swapped for the job's existing
`canary_tag` output).
- The concurrency claim is docker.yml's own documented behavior: groups
are per-ref, and a tag ref is distinct from `refs/heads/master`.
- Not run: a live canary publish — the next master merge after this
lands exercises it end to end; the worst case (double build for a canary
whose master-push run also survives) is benign, since both runs push
identical content-addressed tags.
## Risks
- Low. Roughly one additional Docker build per canary on busy days (on
quiet days the master-push run and the dispatched run both execute —
duplicate work, identical images, no conflict since the sha tags are
content-equivalent for the same commit).
- `actions: write` on `publish_canary` matches the grant
`publish_nightly` already carries for the same purpose.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
Claude Fable 5 (Anthropic, model id `claude-fable-5`), extended
thinking, agentic tool use in Claude Code: GHCR manifest probing,
Actions run-list forensics, and workflow-lane comparison.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (workflow-only change; YAML
linted; no test harness targets release.yml)
- [x] I have added or updated tests where applicable (not applicable —
CI wiring mirroring an existing proven lane)
- [x] I have updated relevant documentation to reflect my changes (the
step's inline comment documents the invariant)
- [x] I have considered and documented the risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
Paid cells now reuse the AWS image's system Chrome, but Playwright video
recording still resolves its revision-pinned FFmpeg helper from the
Playwright cache. Run 33875618534 proved Chrome qualification succeeds
and then failed before provider startup because that helper was absent.
The same run also exposed that generic lock repair can churn unrelated
package platform metadata, so the automated repair paths need
resolution-only regeneration rather than lockfile-only metadata refresh.
## What Changed
- install Playwright FFmpeg only on the AWS/system-Chrome path
- retry the small helper installation up to three times before provider
secrets are exposed
- keep the GitHub-hosted Chromium fallback unchanged
- bind static coverage to the exact FFmpeg step block and its pre-secret
ordering
- add pnpm `--resolution-only` to all four automated lock-repair paths
while retaining full transitive resolution
- require resolution-only repair in the shared workflow regression
The actual generated lockfile correction remains bot-owned by PR #12828
and is intentionally not committed here.
## Verification
- `node --test
.github/scripts/tests/lockfile-refresh-workflows.test.mjs`
- `actionlint -ignore SC2012` on all modified workflows
- focused Prettier checks
- `git diff --check`
- prior run 33875618534: system Chrome 151 qualified; missing Playwright
FFmpeg was the sole cell startup failure
## Risks
Low. The new network operation is limited to Playwright's pinned FFmpeg
payload, happens before paid credentials are exposed, and leaves the
hosted-runner path unchanged. Resolution-only is still a full
dependency-resolution pass, unlike lockfile-only, while avoiding
unrelated current-platform metadata churn.
## Model Used
GPT-5
Use full dependency resolution in automated lockfile repair paths, add regression coverage, and refresh the stale Rollup snapshot.
Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: lockfile-bot <lockfile-bot@users.noreply.github.com>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip supports opt-in Sentry error monitoring for server and
browser errors.
> - The hosted image must include the server package when an operator
sets SENTRY_DSN.
> - The server package is an optional peer in the source tree, so the
image did not include it.
> - This pull request installs the declared server package in the hosted
image and checks the result.
> - The benefit is a hosted tenant can send server errors without a
manual package install.
## Linked Issues or Issue Description
No public issue exists for this change.
**What happened?**
The hosted image did not include the declared @sentry/node server
package. A hosted tenant could set SENTRY_DSN, but the server could not
load the package from the image.
**Expected behavior**
The hosted image must include the exact @sentry/node version from
server/package.json. The self-hosted image must remain without this
optional package.
**Steps to reproduce**
1. Build or pull the hosted image.
2. Resolve @sentry/node from the server package path.
3. Compare its version with server/package.json.
4. Confirm that the tsx loader path still resolves.
**Paperclip version or commit**
Commit b6ff556a33ebdbe764b7f495951cd59009776608.
**Deployment mode**
Docker hosted image.
## What Changed
- Add a cloud-server-deps Docker stage that installs the declared
@sentry/node version in isolation.
- Copy the isolated package into the cloud image without changing the
production image.
- Add a probe that checks the tsx loader and the resolved Sentry
version.
- Run the probe after the hosted image push in the Docker workflow.
- Add server tests and update the observability documentation.
## Verification
- Run `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/cloud-image-sentry.test.ts`.
- Confirm that the changed test passes in CI.
- Confirm that all pull request checks pass.
- Note that the Docker workflow does not run for pull requests. It runs
after a push to master, for configured tags, or after manual dispatch.
## Risks
- Low risk. The production image body stays unchanged.
- The cloud image adds the declared Sentry package and a small
dependency tree.
- The workflow probe fails if the image loses the tsx loader or resolves
a different Sentry version.
## Model Used
OpenAI GPT-5; exact model version supplied by the execution service;
tool use and code execution; context window not specified.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute inside the server container, and they spawn many
short-lived descendants: git, the adapter CLI, esbuild, sh
> - The server image sets `ENTRYPOINT ["docker-entrypoint.sh"]`, and
that entrypoint ends in `exec`, so node becomes PID 1
> - Node reaps only the children it spawned itself. It installs no
`SIGCHLD`/`waitpid` handler for orphans that the kernel re-parents onto
PID 1, so those orphans stay as zombies forever
> - Zombies accumulate monotonically. When the cgroup pid limit is
reached, every `fork()` in the container fails and the instance is dead
> - This pull request installs `tini` and makes it PID 1 in front of the
existing entrypoint, adds a behavioural test that proves reaping, and
adds a `pids_limit` backstop to both compose files
> - The benefit is that a long-running container no longer degrades into
total fork failure, and a future regression is caught by CI instead of
by an outage
Depends-on: none — this change is self-contained in the image build and
its tests, and it touches no other in-flight branch
## Linked Issues or Issue Description
No public GitHub issue exists for this defect. It was found on a live
long-running instance. Description follows the bug report template.
**What happened?**
The server container ran for 22 hours and reached 2039 of 2048 pids in
its cgroup. Of 1760 processes, 1731 were zombies, and all 1731 had PID 1
as their parent. PID 1 was `node --import
./server/node_modules/tsx/dist/loader.mjs server/dist/index.js`. Zombies
accrued at about 79 per hour and were never reaped. The oldest zombie
was 20.8 hours old against a container uptime of 22.0 hours, so nothing
had been reaped since boot. Once the pid limit was reached, `git` and
`gh` failed with `pthread_create failed: Resource temporarily
unavailable`.
**Expected behavior**
PID 1 reaps orphaned processes that the kernel re-parents onto it. The
pid count of a long-running container stays flat instead of growing
without bound.
**Steps to reproduce**
1. Start the server image without `docker run --init` and without `init:
true`.
2. Run agent work that spawns descendants which outlive their immediate
parent.
3. Read `/sys/fs/cgroup/pids.current` and count processes in `Z` state
over several hours.
4. The zombie count grows monotonically and every zombie has PPID 1.
**Relevant logs or output**
```
cgroup pids.current / pids.max : 2039 / 2048
total processes : 1760
zombies : 1731 (98.4%)
parent of every zombie : PID 1 (1731/1731)
PID 1 cmdline : node --import .../tsx/dist/loader.mjs server/dist/index.js
container uptime : 22.0 h
oldest zombie : 20.8 h median: 14.4 h
zombie names : git 717, claude 280, MainThread 167, sleep 141,
esbuild 138, postgres 76, sh 65, sccache 50
```
**Additional context**
The fix pattern is already in this repository.
`docker/agent-runtime/Dockerfile.base` installs `tini` and sets
`ENTRYPOINT ["/usr/bin/tini", "--"]`. It was never applied to the server
image.
## What Changed
- `Dockerfile`: install `tini` in the `base` stage and set `ENTRYPOINT
["/usr/bin/tini", "--", "docker-entrypoint.sh"]`. The entrypoint stays
in the exec chain, so UID/GID remapping, `gosu`, and graceful shutdown
are unchanged.
- `scripts/assert-orphan-reaping.sh` (new): a behavioural probe. It
spawns a leader that forks a grandchild, exits the leader, and asserts
that the orphaned grandchild leaves `Z` state instead of persisting. It
fails closed if the grandchild is not re-parented onto PID 1, so a pass
cannot mean the check ran too early.
- `.github/workflows/docker.yml`: run that probe against the pushed
image after the publish step. The publish step is multi-arch with `push:
true`, so nothing is loaded into the runner daemon and the pushed tag is
the only thing to test. The cloud variant is `FROM production` and
inherits the same `ENTRYPOINT`.
- `scripts/docker-build-test.sh`: run the same probe against a local
build.
- `docker/docker-compose.yml` and
`docker/docker-compose.quickstart.yml`: add `pids_limit: 2048` as a
backstop, so a future leak dies visibly at its own ceiling instead of
starving the host of pids.
- `server/src/__tests__/container-init-reaping.test.ts` (new): 13
assertions that guard the configuration the probe depends on.
No per-orchestrator init lever was added. The image owning PID 1 covers
compose, plain `docker run`, the quadlet units, and the ECS task
definition in one place. Adding `init: true` in compose or
`initProcessEnabled` on the ECS task would nest a second init around
`tini`, and `tini` then warns on every boot that it is not PID 1. The
new test asserts the absence of both levers across all three manifests,
so the decision survives the next edit.
## Verification
| Check | Result |
|---|---|
| `scripts/assert-orphan-reaping.sh` against a real init | Grandchild
re-parented to PPID 1, then reaped. Exit 0. |
| Same probe forced against a genuine zombie | Reports `Z` and fails.
The failure branch is not vacuous. |
| Config guard against the pre-fix files | Exactly the 3 relevant
assertions turn red. |
| Config guard with `tini` removed from `apt-get` but the comments kept
| Red. It checks the install, not a mention of the name. |
| `cd server && npx vitest run
src/__tests__/container-init-reaping.test.ts` | 13 passed |
| `npx tsc --noEmit -p server` | Clean |
| `node scripts/check-docker-deps-stage.mjs` | PASS |
| `node --test scripts/release-verify-workflow.test.mjs` | 8 passed |
Not verified locally: no container runtime is available in the authoring
environment, so the probe has not run against a build of this image. The
new `docker.yml` step runs it against the pushed image on this PR.
## Risks
Low risk, but it is an image and entrypoint change, so it affects
deployments.
- `tini` adds one small package to the `base` stage.
`docker/agent-runtime/Dockerfile.base` already installs it from the same
Debian archive.
- Signal handling changes shape: `tini` receives `SIGTERM` and forwards
it to the entrypoint, which `exec`s node. `tini` forwards signals to its
direct child by default, and the exec chain keeps node as that child, so
graceful shutdown is preserved. A reviewer should confirm this on a real
stop.
- `pids_limit: 2048` is new for compose users. A deployment that
legitimately needs more than 2048 processes would now hit the ceiling.
The measured steady state on a busy instance was under 400.
- If a deployment already passes `--init` or `init: true`, `tini` runs
under another init and prints a warning that it is not PID 1. Reaping
still works because the outer init handles it. The compose files in this
repository do not set `init: true`.
## Model Used
Claude Opus 5 (`claude-opus-5`), extended thinking, with tool use and
code execution in an agent harness.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local issues or links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: zannis <1011451+zannis@users.noreply.github.com>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip runs across the CLI, server, adapters, plugins, CI, and
container images.
> - These surfaces declared different Node.js versions from 20 through
24.
> - A newer `@types/node` major can expose APIs that the supported
runtime does not provide.
> - Node.js 20 is no longer a suitable project baseline, and Node.js 24
is the current LTS line.
> - This pull request sets Node.js 24.11.0 as one repository-wide
baseline, adds a drift check, and gives users actionable startup
guidance when their runtime is too old.
> - The benefit is one clear runtime contract for development, release,
installation, and published packages.
## Linked Issues or Issue Description
Refs #2734
Refs #11727
Refs #739
## What Changed
- Require Node.js 24.11.0 or newer in all 42 package manifests and
runtime checks.
- Use Node.js 24 in GitHub Actions, Docker images, smoke images, sandbox
setup, portable installs, and esbuild targets.
- Align every direct `@types/node` declaration on `^24.0.0`.
- Prevent Dependabot from opening major `@types/node` upgrades without a
matching runtime decision.
- Add `.nvmrc` and a CI policy check for Node version drift.
- Update ACP version gates, tests, and user documentation for the new
minimum.
- Print a non-blocking warning on CLI and server startup when Node is
unsupported, with remediation through a version manager or the
documented downloaded `install.sh` workflow.
- Deduplicate that warning when `paperclipai run` boots the CLI and
server in the same process.
## Verification
- `node scripts/check-node-version-policy.mjs`
- `node --check scripts/check-node-version-policy.mjs`
- `node --check cli/esbuild.config.mjs`
- `node --check scripts/generate-npm-package-json.mjs`
- `bash -n scripts/install.sh scripts/test-install-sh-docker.sh
scripts/e2e-install-lifecycle.sh`
- Parsed all 42 package manifests and confirmed `engines.node` is
`>=24.11.0`.
- `git diff --check`
- `vitest run
packages/adapter-utils/src/sandbox-install-command.test.ts` passed with
3 tests.
- `vitest run cli/src/node-version.test.ts` passed with 4 tests.
- Directly exercised the shared warning helper for unsupported-version
messaging and same-process deduplication.
- The focused exe.dev suite could not resolve the locally unbuilt plugin
SDK from this isolated worktree. A full offline workspace install was
also blocked because the package-manager signature verifier requires
registry access. The full suite was not run locally; draft CI performs a
clean install and evaluates the wider impact.
## Risks
- This is a breaking runtime change for users, plugins, and deployments
that still use Node.js 20 or 22.
- Published workspace packages will now produce an engine warning or
failure in strict package managers on older Node.js releases.
- Node.js 24 can reveal dependency, native module, Playwright, or agent
CLI compatibility issues in CI.
- The bootstrap installer now installs Node.js 24 when the current
runtime is older than 24.11.0.
- The portable sandbox fallback is pinned to Node.js 24.11.0 and depends
on that upstream tarball remaining available.
- Unsupported runtimes continue booting after a warning, so a later
incompatibility can still fail at its point of use.
- The CLI and server share the warning policy through the published
`@paperclipai/shared` package; packaging checks must keep that subpath
export available.
- This PR does not commit `pnpm-lock.yaml` because repository policy
assigns lockfile generation to CI.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex based on GPT-5. The exact deployment ID and context
window are not exposed in this session. Reasoning, repository tools,
shell execution, and GitHub tools were enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
> Follow-up to #11006 (merged): rebased onto master and ready for
review.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The release subsystem now publishes canary (every master push),
nightly (scheduled, smoke-gated, added in #11006), and stable (manual)
> - There is still no human-approved release-candidate lane between
nightly and stable, and nothing enforces that a stable actually soaked
anywhere before shipping
> - Betas need a real approval gate, and stables need a soak policy that
is data, not prose
> - This pull request adds the beta channel: a manual promotion of a
chosen nightly behind the `npm-beta` environment gate, re-smoked after
publish, plus a stable preflight that enforces a 3-day beta soak with a
written-justification bypass
> - The benefit is a complete canary → nightly → beta → stable train
where every stable shipped as a beta first, and emergencies leave a
written trace
## Linked Issues or Issue Description
**Subsystem affected**
Release automation: `scripts/release.sh`, `scripts/release-lib.sh`,
`.github/workflows/release.yml`, `.github/workflows/docker.yml`,
`.github/workflows/release-smoke.yml`.
**Problem or motivation**
After #11006 the project has canary and nightly prerelease lanes, but no
release-candidate lane. Stable promotion has no enforced soak: any ref
can ship as stable directly. There is no approval boundary for a
broader-audience prerelease, and no structured way to record why an
emergency release skipped validation.
**Proposed solution**
Add a `beta` channel: a manual dispatch that promotes a chosen nightly's
source commit, publishes behind the `npm-beta` GitHub environment
(required reviewers are the gate), re-smokes the published beta, and
tags `beta/vX`. Enforce in the stable path that the source commit
shipped as a beta at least 3 days earlier (measured from the beta's npm
publish time), with a `skip_soak_justification` input as the recorded
emergency bypass.
**Alternatives considered**
Codifying the soak policy in docs only. Rejected: an unenforced policy
decays; the preflight makes the policy executable while the
justification input keeps the emergency path usable and auditable.
## What Changed
- `scripts/release.sh` + `scripts/release-lib.sh`: `beta` channel —
requires HEAD to carry a `nightly/v*` tag, publishes the package set as
`YYYY.MDD.P-beta.N` under dist-tag `beta`, tags
`beta/vYYYY.MDD.P-beta.N`
- `.github/workflows/release.yml`:
- `channel: beta` dispatch path: `select_beta` resolves the newest (or
an explicit `source_version`) nightly and fails loudly on selection
problems; `publish_beta` runs behind the `npm-beta` environment, pushes
the tag, and dispatches `docker.yml`; `smoke_beta` re-runs the release
smoke suite against the exact published beta version
- stable path: new `preflight_stable` job enforces the 3-day beta soak
from the beta's npm publish time; `skip_soak_justification` bypasses
with the reason echoed into the job summary; dry runs report without
blocking
- `.github/workflows/docker.yml`: `beta/v*` tags publish `:beta` on both
images, with exact version stamping
- `.github/workflows/release-smoke.yml`: `beta` added to the dispatch
choice list
- Docs: `CHANNELS.md` beta entries; `RELEASING.md` beta lane, soak gate,
and failure playbook; `RELEASE-AUTOMATION-SETUP.md` `npm-beta`
environment setup, including the warning to create the environment
before the first beta dispatch (GitHub auto-creates unprotected
environments on first reference)
- Tests: beta version-counting coverage in
`scripts/release-registry-versions.test.mjs`; beta identity and
nightly-tag guard coverage in
`scripts/__tests__/release-dry-run-notes.test.mjs`
## Verification
- `node --test` on the two touched suites: 17 pass, including the 3 new
beta tests
- `bash -n` on both shell scripts and YAML parse of all three workflows
- After merge, in order: create the `npm-beta` environment, dispatch
`channel: beta` with `dry_run: true` to preview, then a real promotion
of a published nightly through the approval gate, then a stable dry-run
against a young beta to see the soak gate report
## Risks
- If the `npm-beta` environment does not exist when the first beta
dispatch runs, GitHub creates it with no protection rules and the beta
publishes without approval. Mitigated by documentation and by creating
the environment before merge (operator step)
- Until the first beta exists, every stable dispatch requires
`skip_soak_justification`. This is deliberate — the first beta ships
immediately after this merges — but it is a behavior change to the
stable dispatch
- The soak clock reads the beta's npm publish time from the registry; a
registry outage makes the preflight fall back to requiring justification
(fail-closed)
## Model Used
Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with
extended thinking and full tool use (repository exploration, local test
execution, live registry and git verification). All code, tests, and
docs in this PR were model-authored under human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending — will confirm before
merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending — will confirm before merge)
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The release subsystem publishes the `paperclipai` npm package set
and the Docker images on two lanes: canary on every master push, and
stable on manual promotion
> - There is no middle ground between those lanes. Users must track
every merge or wait weeks for a stable. Docker `:latest` also tracks
master, so Docker users have no stable image at all
> - A calm prerelease lane needs to exist, and it must never ship a
build that failed its checks
> - This pull request adds the nightly channel: a scheduled job that
selects the newest master commit with a green canary publish, runs the
full release smoke suite against that exact published canary, and only
then republishes it as the nightly. It also separates Docker tags by
lane, so `:latest` finally means stable
> - The benefit is that users can follow prereleases at a nightly
cadence with a smoke-tested guarantee, and Docker users get real
`:canary`, `:nightly`, and stable image tags
## Linked Issues or Issue Description
**Subsystem affected**
Release automation: `scripts/release.sh`, `scripts/release-lib.sh`,
`.github/workflows/release.yml`, `.github/workflows/docker.yml`,
`.github/workflows/release-smoke.yml`.
**Problem or motivation**
The project publishes only `canary` (every master push) and `latest`
(manual stable). Users who want prereleases without per-merge churn have
no option. Docker has a second problem: master builds overwrite
`:latest`, and CI-published stables never produced Docker images,
because tags pushed with `GITHUB_TOKEN` do not fire the `v*` tag trigger
in `docker.yml`. No stable-versioned image exists in ghcr today.
**Proposed solution**
Add a `nightly` channel. A scheduled job selects the newest
canary-tagged master commit, smoke-tests that exact published canary,
and republishes the same commit as `YYYY.MDD.P-nightly.N` under the
`nightly` dist-tag. Separate Docker tags by lane (`:canary` for master,
`:nightly` for nightly tags, `:latest` plus version tags for stable tags
only), and have the release jobs dispatch `docker.yml` at the new tag so
lane images actually build.
**Alternatives considered**
Moving the `nightly` dist-tag to the existing canary version without a
republish. Rejected: the version string would say `canary` while the
user is on nightly, which breaks at-a-glance lane identification in bug
reports and `--version` output.
## What Changed
- `scripts/release-lib.sh`: channel-parameterized
`next_prerelease_version` and `prerelease_tag_name` helpers (canary
helpers delegate to them), a `require_channel_tag_at_head` guard, and
the no-provenance retry for Sigstore transparency-log duplicates now
covers the `nightly` dist-tag as well as `canary`
- `scripts/release.sh`: new `nightly` channel. It requires HEAD to carry
a `canary/v*` tag, publishes the full public package set as
`YYYY.MDD.P-nightly.N` under dist-tag `nightly`, and tags the source
commit `nightly/vYYYY.MDD.P-nightly.N`
- `.github/workflows/release.yml`: scheduled nightly chain (09:00 UTC) —
select candidate, smoke it via `release-smoke.yml`, publish on green
under the existing `npm-canary` environment, push the tag, dispatch
`docker.yml`. New `channel` dispatch input (default `stable`, so
existing stable dispatches are unchanged) with `nightly_source_version`
and `dry_run` support for forced runs. The stable path now also
dispatches `docker.yml` at the new `v*` tag
- `.github/workflows/docker.yml`: lane tag mapping for both image jobs —
master pushes publish `:canary` and no longer move `:latest`;
`nightly/v*` tags publish `:nightly`; only stable `v*` tags publish
`:latest` and the versioned tags. New `workflow_dispatch` trigger for
the release-job dispatches. Build-version stamping uses the exact
nightly version on nightly tag builds
- `.github/workflows/release-smoke.yml`: `nightly` added to the dispatch
choice list
- `doc/CHANNELS.md` (new): user-facing guide to the channels
- `doc/RELEASING.md`: nightly lane documentation, Docker tag mapping
table, and a nightly failure playbook
- `doc/RELEASE-AUTOMATION-SETUP.md`: note that nightly reuses
`npm-canary` and needs no npm trusted-publisher changes
- Tests: channel-parameterized version helper coverage in
`scripts/release-registry-versions.test.mjs`, and nightly flow coverage
(publish identity, notes not required, canary-tag guard) in
`scripts/__tests__/release-dry-run-notes.test.mjs`
## Verification
- `node --test` on the release script suites: 68 pass, including 6 new
tests. The only failure, `acpx-patch-packaging.test.mjs`, needs
installed `node_modules` and fails identically on a pristine checkout of
master in the same environment
- `bash -n` on both shell scripts and YAML parse of all three workflows
- Live fail-path check: `./scripts/release.sh nightly --print-version`
from a master tip with no canary tag fails with `HEAD has no canary/v*
tag`
- Live success-path check: the same command from the
`canary/v2026.806.0-canary.7` commit prints `2026.806.0-nightly.0`
- Live selection check: the candidate-selection shell logic run against
the real repository selects the commit of `canary/v2026.806.0-canary.7`,
which matches the current npm `canary` dist-tag exactly
- After merge: dispatch `release.yml` with `channel: nightly` and
`dry_run: true` to preview, then a real forced run to validate end to
end before the first scheduled run
## Risks
- Docker `:latest` changes meaning from "latest master build" to "latest
stable release". This is deliberate and will be announced. Users who
want the old behavior pull `:canary`. Until the first stable release
after this change, `:latest` stays at its current (master-built) image
- The nightly is a rebuild of the same source commit, not the
byte-identical canary artifact that was smoked. The lockfile pins
dependencies, and the publish path's registry-visibility and
clean-prefix install gates still run on the nightly artifacts
- All npm publishing must stay inside `release.yml` because npm trusted
publishing pins that workflow file per package. The nightly jobs were
added to `release.yml` for exactly that reason; this constraint is now
documented in `RELEASING.md`
- The stable-lane Docker dispatch fails gracefully (a warning with
manual instructions) when the source ref predates `docker.yml`'s
`workflow_dispatch` trigger
## Model Used
Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with
extended thinking and full tool use (repository exploration, local test
execution, live registry and git verification). All code, tests, and
docs in this PR were model-authored under human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending — will confirm before
merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending — will confirm before merge)
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Published container images are the deployable unit for self-hosted
and managed instances, so merge-to-image latency bounds every deploy
iteration
> - The docker workflow configures BuildKit caching, but builds still
ran ~12+ minutes essentially cold
> - Two causes: the most expensive layer (four CLI toolchains + apt) is
ordered after the always-changing app copy so it can never cache, and
the type=gha cache's 10GB repo cap means the two multi-arch mode=max
jobs evict each other
> - This pull request reorders the tool layer above the app copy (with a
weekly epoch so @latest tools keep advancing) and switches both jobs to
registry-backed cache in ghcr
> - The benefit is that warm builds shrink to roughly the app build +
push, targeting the sub-5-minute range together with the amd64-only
cloud variant
## Linked Issues or Issue Description
No existing public issue — inline description following the feature
request template:
**Subsystem affected**
CI / release publishing (docker workflow, Dockerfile)
**Problem or motivation**
Despite `cache-from/cache-to` being configured, image builds run
effectively cold: (1) the production stage installs four CLI toolchains
+ apt packages *after* `COPY --from=build /app /app`, and since the app
copy changes every commit, that most-expensive layer rebuilds every
build, per arch; (2) the `type=gha` BuildKit cache is capped at 10GB per
repository, and two multi-arch `mode=max` jobs overflow and evict each
other's entries.
**Proposed solution**
Order the tool/OS layer before the app copy (it references nothing from
`/app`), refresh it weekly via a `CLI_TOOLS_CACHE_EPOCH` build arg so
the `@latest` tools don't freeze in the cache, and move both jobs to
registry-backed BuildKit cache (`:buildcache` / `:buildcache-cloud` refs
in ghcr, no size cap, separate refs so the parallel jobs don't clobber
each other).
**Alternatives considered**
Pinning CLI tool versions instead of the weekly epoch — more
deterministic, but adds a version-bump chore; the weekly epoch preserves
current freshness semantics with bounded staleness. Keeping type=gha
with `mode=min` — smaller cache but loses intermediate-stage reuse,
which is where most of the win is.
**Roadmap alignment**
Not on ROADMAP.md; CI/publishing speed improvement only.
## What Changed
- `Dockerfile`: the production stage's tool/OS `RUN` (npm --global CLIs,
apt, `/paperclip` setup) moves above `COPY --from=build /app /app`; new
`CLI_TOOLS_CACHE_EPOCH` arg consumed by that layer. The `cloud` stage is
unaffected — it only layers plugin dists on top of the finished
production stage.
- `.github/workflows/docker.yml`: both jobs stamp the ISO week into
`CLI_TOOLS_CACHE_EPOCH`, and both switch `cache-from/cache-to` from
`type=gha` to `type=registry` with per-job refs.
- Includes the one-line amd64-only cloud-variant commit from #10570 so
the two PRs can't conflict; if #10570 merges first, this PR rebases down
to a single commit automatically.
## Verification
- Image content is unchanged by layer reordering: the moved `RUN`
references nothing from `/app`, and Docker layer ordering only affects
caching, not the final filesystem (tool installs and app copy touch
disjoint paths).
- The cache ref is written only by this workflow — `docker.yml` runs on
master/tag pushes, never on PRs — so the workflow's existing "no shared
caches into build inputs" supply-chain stance is unchanged (BuildKit
layer cache was already accepted via type=gha; the registry backend has
the same writer trust).
- Runtime proof lands with the first two master builds after merge: the
first warms the cache, the second should show the tool layer and
deps/build stages as CACHED in the build log, with wall clock dropping
accordingly. I'll be watching those as part of managed-deploy work.
## Risks
- Low. Worst case the registry cache misses (cold-build behavior, same
as today). The weekly epoch means CLI tools update at most a week late
inside images; a release built mid-week ships the tools from that week's
first build. Cache refs add two small artifacts to ghcr.
## Model Used
Claude Fable 5 (`claude-fable-5`, extended thinking, via Claude Code
with tool use and code execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no test-affecting changes)
- [x] I have added or updated tests where applicable (n/a — build config
and layer ordering only)
- [x] I have updated relevant documentation to reflect my changes
(in-file comments document both mechanisms)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Published container images are how both self-hosted users and
managed-deployment hosts run it
> - The repository publishes two variants: the self-hosted image and a
cloud variant for managed deployments
> - The cloud variant was built for amd64+arm64, but its only consumers
are managed-deployment hosts, which run amd64
> - The QEMU-emulated arm64 half dominates the build's wall clock,
delaying every merge-to-deployable-image cycle
> - This pull request drops arm64 from the cloud variant only, keeping
the self-hosted image multi-arch
> - The benefit is roughly halving the time from merge to a deployable
cloud image, with no change for any actual consumer
## Linked Issues or Issue Description
No existing public issue — inline description following the feature
request template:
**Subsystem affected**
CI / release publishing (docker workflow)
**Problem or motivation**
The cloud image variant builds for `linux/amd64,linux/arm64`, but the
arm64 half runs under QEMU emulation and dominates the job's wall clock
— while no consumer of the cloud variant runs arm64 (managed-deployment
hosts are amd64). Every deploy iteration pays ~double the necessary
build time.
**Proposed solution**
Build the cloud variant amd64-only. The self-hosted image keeps
`amd64+arm64` so ARM users (Apple Silicon, ARM servers) are unaffected.
**Alternatives considered**
Keeping multi-arch but building arm64 on native arm64 runners with a
manifest merge — faster than QEMU and worth doing for the self-hosted
image if its build time becomes a pain point, but unnecessary complexity
for a variant with no arm64 consumers.
**Roadmap alignment**
Not on ROADMAP.md; CI/publishing speed improvement only.
## What Changed
- `.github/workflows/docker.yml`: the `build-and-push-cloud` job's
`platforms` is now `linux/amd64` (with a comment explaining why). The
self-hosted `build-and-push` job is untouched.
## Verification
- Build-config-only change; the workflow runs on merge to master. The
published `-cloud` manifest will be amd64-only, which its consumers
already pull.
- No test changes: nothing at runtime differs on any platform that
actually runs the image.
## Risks
- Low. If an arm64 consumer of the cloud variant ever appears (e.g.
local `docker run` on Apple Silicon for debugging), it would fall back
to emulation on the consumer's machine or need this reverted — a
one-line change. The self-hosted image's platform matrix is unchanged.
## Model Used
Claude Fable 5 (`claude-fable-5`, extended thinking, via Claude Code
with tool use and code execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no test-affecting changes)
- [x] I have added or updated tests where applicable (n/a — CI platform
matrix only)
- [x] I have updated relevant documentation to reflect my changes
(in-workflow comment documents the rationale)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Instances commonly run from published container images, and
operators need to observe which build a container actually serves
> - `/api/health` now reports the running build commit, and server-info
already falls back to `PAPERCLIP_BUILD_COMMIT` when git is unavailable
> - But published images carry no `.git` and never received
`PAPERCLIP_BUILD_COMMIT`, so containers report `commit: null` — verified
live against a current image
> - That leaves the new deployment-verification field inert exactly
where it matters most: containerized deploys
> - This pull request bakes the exact build commit into both image
variants at build time, mirroring how `PAPERCLIP_BUILD_VERSION` is
already stamped
> - The benefit is that containers report their true commit on
`/api/health`, so deploy tooling can verify a rollout actually shipped
## Linked Issues or Issue Description
Companion to #10563 (which exposed the `commit` field on `/api/health`).
Inline description following the bug report template:
**What happened?**
A container from a published image responds to `GET /api/health` with
`"commit": null`. The image has no `.git` directory and the
`PAPERCLIP_BUILD_COMMIT` fallback that `server-info` supports is never
provided at build time, so git metadata resolves as unavailable.
**Expected behavior**
A container reports the commit it was built from, the same way it
already reports its build version via the baked
`PAPERCLIP_BUILD_VERSION`.
**Steps to reproduce**
Run any published image (e.g.
`ghcr.io/paperclipai/paperclip:sha-c4f6264-cloud`) and `curl
/api/health` — `commit` is `null` even though the build commit is known
at image-build time.
**Paperclip version or commit**
`sha-c4f6264-cloud` (first image containing #10563).
## What Changed
- `Dockerfile`: new `PAPERCLIP_BUILD_COMMIT` build arg, exported as an
ENV in the production stage (the `cloud` stage inherits it), directly
parallel to `PAPERCLIP_BUILD_VERSION`. Empty for local `docker build`,
which keeps the normal fallbacks.
- `.github/workflows/docker.yml`: both build jobs pass
`PAPERCLIP_BUILD_COMMIT=${{ github.sha }}`.
## Verification
- Reviewed the plumbing end-to-end: `build-commit.ts` reads
`PAPERCLIP_BUILD_COMMIT` (validated as a full SHA), `server-info.ts`
`readGitInfo` falls back to it when the git CLI fails, producing
`available: true, fullSha` — which `/api/health` surfaces as `commit`.
- Verified live that a current published image reports `commit: null`;
this change repairs that on the next build. Post-merge, the first master
image should report its commit — I'll be verifying that as part of
managed-deploy validation.
- No test changes: the fallback path is already covered by existing
server-info tests; this PR only supplies the env at image build.
## Risks
- Low. Two build-time stamps; no runtime code changes. A wrong SHA would
only mislabel the build (same failure mode `PAPERCLIP_BUILD_VERSION`
already carries), and `${{ github.sha }}` is the exact commit the
workflow builds.
## Model Used
Claude Fable 5 (`claude-fable-5`, extended thinking, via Claude Code
with tool use and code execution); diagnosis included live probes of a
running container's `/api/health`.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no test-affecting changes;
server suites unaffected)
- [x] I have added or updated tests where applicable (n/a — build-time
stamps only)
- [x] I have updated relevant documentation to reflect my changes
(Dockerfile comments document the arg)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The CI workflow is responsible for producing and publishing the
Docker images that power Paperclip deployments
> - The cloud image publish was previously coupled to the stock publish
job, so a failure or delay in one path could gate the other
> - That coupling makes the release pipeline less resilient than it
needs to be
> - This pull request gives the cloud publish its own top-level job so
both publishes can run in parallel without a `needs:` dependency
> - The benefit is better failure isolation and less wasted time when
one publish path is slow or broken
## Linked Issues or Issue Description
No public GitHub issue was found for this change.
Problem statement:
- The cloud image publish was implemented as trailing work inside the
stock publish job.
- That setup meant the cloud publish could be delayed or skipped if the
stock job failed early.
- The desired behavior is for the cloud publish to run independently so
a failure in one publish path does not gate the other.
Proposed solution:
- Split the cloud publish into its own top-level workflow job.
- Keep the same cloud-specific build settings and cache behavior.
- Preserve the existing top-level concurrency behavior.
Alternatives considered:
- Keeping both publishes in one job with conditionals or later steps.
Rejected because it still couples success and runtime between the two
publish paths.
## What Changed
- Split the cloud image publish into a separate top-level Docker
workflow job.
- Removed the dependency coupling so the cloud job does not need the
stock job.
- Expanded the drift-guard test to assert the two-job structure and the
absence of `needs:` on the cloud job.
## Verification
- The workflow YAML was parsed successfully and confirmed to contain two
jobs: `build-and-push` and `build-and-push-cloud`.
- The cloud job was confirmed to have no `needs:` entry.
- The drift-guard assertions were reproduced in a dependency-free
harness and passed.
- PR #10408 completed GitHub Actions with all required checks green,
including the e2e shards.
- Greptile review completed at 5/5 with no unresolved comments.
- No documentation files changed because this is a workflow/test-only
change.
## Risks
- The workflow now duplicates the prep steps across two runners, so any
shared setup change must be kept in sync between both jobs.
- The new job increases workflow surface area slightly, which can make
future maintenance more verbose.
- Overall risk is low because the change is limited to CI orchestration
and test coverage.
## Model Used
OpenAI Codex (GPT-5, tool-using code assistant)
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The repository uses GitHub Actions workflows to build and publish
Docker images
> - A workflow-level concurrency policy controls whether newer pushes
cancel in-flight builds
> - The current job-level setting can cancel a running image build
before it finishes publishing
> - That leaves the Docker image pipeline brittle when a new push
arrives during an active publish
> - This pull request moves concurrency to the workflow level and
disables cancel-in-progress so running builds finish
> - The benefit is that only pending work is superseded, while a build
already publishing is allowed to complete
## Linked Issues or Issue Description
No public GitHub issue exists for this change. This PR addresses the
Docker workflow concurrency behavior directly: it ensures in-flight
image builds are not cancelled by newer pushes, while still serializing
builds per ref.
## What Changed
- Moved the Docker workflow concurrency block from the job level to the
workflow level.
- Set `cancel-in-progress: false` so an active build can finish
publishing.
- Added a drift-guard test that parses `.github/workflows/docker.yml`
and asserts the workflow-level concurrency policy remains `false`.
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/cloud-image-bundled-plugins.test.ts`
- `python3 -c "import yaml;
yaml.safe_load(open('.github/workflows/docker.yml'))"`
- Verified the fetched remote branch contains a single commit on top of
`origin/master`.
- Searched GitHub for duplicate or related PRs and issues; none found.
- Checked `ROADMAP.md` and did not find overlapping planned core work.
## Risks
- Low risk: the change is limited to workflow concurrency behavior and a
targeted test assertion.
- If the workflow concurrency key is changed later, the drift-guard test
will fail and require an update.
## Model Used
OpenAI Codex (GPT-5, tool use; context window not surfaced in this
environment)
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work; it ships as a Docker image that self-hosters and managed
deployments run.
> - The server resolves its own version at runtime in
`server/src/version.ts` (`resolveServerVersion()`), which feeds
analytics and the server debug panel.
> - That resolver derives the real version from `git describe`, and
falls back to `server/package.json`'s `version` when git isn't
available.
> - But `server/package.json`'s version is a static placeholder — CI
only stamps the real CalVer at publish, so in source it is never the
real version (currently `0.3.1`).
> - A Docker image has no `.git` (it's dockerignored), so `git describe`
can't run inside it. Every image therefore falls back to the placeholder
and reports `0.3.1` in analytics and the debug panel, regardless of
which commit it was built from.
> - This PR computes the real version once on the CI build runner (where
`.git` and tags exist), bakes it into the image, and has
`resolveServerVersion()` prefer that stamp when `git describe` is
unavailable.
> - The benefit: self-hosted and cloud images report their true version
instead of a misleading placeholder, with no change to dev checkouts,
`git describe`-based resolution, or local `docker build`.
## Linked Issues or Issue Description
No public issue exists — describing the bug inline (per the bug report
template).
**What happened?**
Docker images built from `master` (and release tags) report the server
version as the `0.3.1` placeholder in analytics and the server debug
panel, instead of the real version of the commit the image was built
from.
**Expected behavior**
An image reports the real version of its build commit (e.g.
`2026.722.0+51.git.<sha>`), so operators can tell which build is
running.
**Steps to reproduce**
1. Build the server Docker image from any `master` commit (the `Docker`
workflow, `production` target).
2. Run the image and open the server debug panel (or inspect the version
reported to analytics).
3. Observe the version is `0.3.1` rather than the commit's real version.
**Root cause**
`resolveServerVersion()` derives the real version from `git describe`,
but the image has no `.git` (dockerignored), so it falls back to
`server/package.json`'s `version` — a static placeholder CI only
replaces with the real CalVer at publish time. Nothing bakes the real
version into the image.
**Paperclip version or commit:** reproduces on `master` (`4c55f0d8`) and
any published image.
**Deployment mode:** self-hosted and managed (both the `production` and
`-cloud` images).
**Installation method:** Docker image (`ghcr.io/paperclipai/paperclip`).
**Related PRs (dedup search):** #9103 (merged — added the `git
describe`-based source-install resolution this builds on) and #9637
(closed). Neither bakes a version into the image; this PR closes that
gap. No duplicate found.
## What Changed
- **`.github/workflows/docker.yml`** — checkout with full history + tags
(`fetch-depth: 0`), and a new `Compute build version` step that runs
`git describe --tags --match 'v*' --long --dirty` on the pristine runner
checkout. The result is passed as a `PAPERCLIP_BUILD_VERSION` build-arg
to both the `production` and `-cloud` image builds.
- **`Dockerfile`** — the `production` stage takes an `ARG
PAPERCLIP_BUILD_VERSION` (default empty) and bakes it into the runtime
`ENV`; the `cloud` stage inherits it via `FROM production`.
- **`server/src/build-version.ts`** (new) — `readBuildVersion()` /
`parseBuildVersion()`, mirroring `build-commit.ts`: reads
`PAPERCLIP_BUILD_VERSION` (or a `.paperclip-build-version` file) as a
single-token stamp.
- **`server/src/version.ts`** — `resolveServerVersion()` prefers the
baked build version when `git describe` is unavailable, parsing it with
the same rules as a live checkout (`parseGitDescribeVersion`), and
falling through to the existing `build-commit` stamp and package version
when unset. A live checkout's `git describe` still wins over any stamp.
- Tests for the new behavior and the precedence.
## Verification
- `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && tsc
--noEmit` in `server/` — clean.
- `vitest run server/src/__tests__/version.test.ts
server/src/__tests__/build-version.test.ts` — **23 tests pass**,
covering: stamped version used when git describe fails, stamp parsed to
real CalVer, stamp preferred over the build-commit fallback, on-tag
stamp collapses to the release version, a pre-resolved stamp used
verbatim, and a live git describe still winning over a stamp.
- `git describe --tags --match 'v*' --long` for this commit →
`v2026.722.0-51-g<sha>`, which `resolveServerVersion()` reports as
`2026.722.0+51.git.<sha>` — no longer `0.3.1`.
- Not run locally: the full multi-arch image build (CI-only). The
workflow change is verified by inspection; the version is computed on
the pristine checkout before any lockfile refresh, so it carries no
spurious `-dirty`.
## Risks
Low. Additive and image-only:
- No runtime behavior changes for dev checkouts (git describe still
primary and wins over any stamp) or for local `docker build` (empty arg
→ server keeps its existing fallbacks).
- Not a breaking change; no schema or API surface. The stamp is
informational (version reporting only).
- `fetch-depth: 0` makes the release-image checkout fetch full
history/tags — a modest cost on a workflow that already runs at release
cadence with a 60-minute budget.
- Rollback: revert the commit; images simply return to reporting the
placeholder.
## Model Used
Claude Opus 4.8 (`claude-opus-4-8`, 1M-context variant), extended
thinking, with tool use / code execution — agentic edits, `tsc` +
`vitest` runs, and a `git describe` resolution check.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work (bugfix, not core feature work)
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#9103, #9637 — related, not duplicates)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (`fix/build-version-stamp`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user-facing docs affected; behavior is documented inline in `version.ts`
/ `build-version.ts` and the workflow/Dockerfile)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Managed (cloud-hosted) deployments configure instances through
`PAPERCLIP_MANAGED_CONFIG`, including a `plugins.autoInstall` key list
that the boot-time installer resolves against the bundled plugin catalog
> - The installer requires each bundled plugin's `dist/manifest.js`
(`server/src/services/bundled-plugins.ts`), but the published image only
ships the sandbox providers' *source* — they are intentionally excluded
from the pnpm workspace, and the Dockerfile never builds them
> - Every managed auto-install therefore logs `bundled plugin bundle not
present; skipping auto-install` and no sandbox provider can be
provisioned through managed config
> - Baking built plugins into the single published image would fix it
but makes every self-hosted pull carry the providers' `node_modules` for
a managed-only mechanism
> - This pull request adds a `cloud` Dockerfile target extending
`production` with built bundled plugins — parameterized by build arg and
currently just `daytona` — published alongside the default image with a
`-cloud` tag suffix
> - The benefit is working plugin auto-provisioning for managed
deployments while the self-hosted image stays byte-identical and the
cloud variant only carries what is actually deployed
## Linked Issues or Issue Description
Fixes#10158 (filed for this problem; no prior issue existed — searched
for duplicate/related PRs and issues around bundled plugins, docker
image variants, and auto-install). Summary: **What happened:** on a
managed instance with `plugins.autoInstall: ["daytona"]` delivered via
`PAPERCLIP_MANAGED_CONFIG`, boot logs `bundled plugin bundle not
present; skipping auto-install` with `pluginPath:
/app/packages/plugins/sandbox-providers/daytona`, and the plugin is
never installed. **Expected:** the advertised bundled-catalog keys are
installable from the published image. **Why:** the image ships plugin
source without `dist/` — nothing in the Dockerfile builds the
workspace-excluded sandbox providers.
## What Changed
- `Dockerfile`: new `cloud-plugins` stage (based on `build`, so
devDependencies are available for `tsc`) that installs and builds each
provider named in the `CLOUD_BUNDLED_PLUGINS` build arg standalone
(`pnpm install --ignore-workspace --no-lockfile && pnpm build`, exactly
as the providers' READMEs prescribe), asserting `dist/manifest.js`
exists per plugin and failing loudly on unknown names; new `cloud` stage
= `production` + the built plugin tree. The arg defaults to `daytona` —
the only provider managed deployments auto-install today; every entry
adds its `node_modules` to the image, so the list grows only with actual
need (a one-line workflow change).
- `.github/workflows/docker.yml`: the existing build step is pinned to
`target: production` (without this, the new trailing stage would
silently become the default build target — this pin is what keeps the
self-hosted image identical); new metadata + build-push steps publish
the `cloud` target (with `CLOUD_BUNDLED_PLUGINS=daytona`) under the same
tag set with a `-cloud` suffix (`sha-<short>-cloud`, `latest-cloud`,
`<version>-cloud`), same schema labels, reusing the GHA layer cache
## Verification
- All seven sandbox providers build standalone from a clean checkout
with the exact commands the new stage runs, each producing
`dist/manifest.js` — so the current `daytona` default works and future
list additions are known-good
- The stage's shell loop was dry-run against the checkout (directory
existence + per-plugin assertion logic)
- Workflow YAML lints clean
- **Not run:** a full multi-arch `docker build` (no local docker
daemon). The `cloud` stage is additive and the default target is pinned,
so the risk is contained to the new build step; the first master build
after merge proves it end-to-end
## Risks
- Self-hosted behavior: unchanged. The default image build is pinned to
the `production` target, which produces the same layers as before this
change; the `cloud` stages run only for the new build step.
- The plugin installs in the `cloud-plugins` stage use `--no-lockfile`
(the providers are workspace-excluded and lockfile-less by design), so
plugin dependency resolution is not pinned at image-build time. This
mirrors the existing Plugins-page install path, which resolves from npm
at install time.
- CI cost: one additional build-push per master push. It reuses the
layer cache from the production build, so the marginal work is the
single plugin's build layers.
- An unknown name in `CLOUD_BUNDLED_PLUGINS`, or a provider that stops
producing `dist/manifest.js`, fails the cloud build loudly rather than
publishing a broken variant.
## Model Used
Claude (Anthropic), model ID `claude-fable-5[1m]` via Claude Code CLI —
extended thinking and tool use (code edits, standalone plugin build
verification, workflow lint).
## Checklist
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] Self-hosted behavior unchanged (default build target pinned to
`production`)
- [x] One clear change: publish a cloud image variant with built bundled
plugins
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Docker image publish workflow (`.github/workflows/docker.yml`)
builds and pushes the multi-arch `ghcr.io` image on every master push,
so users pulling the container get the latest code
> - The two newest master runs of that workflow failed, so no images
have been published past a recent master commit
> - The failures had two distinct causes: run
[30054330748](https://github.com/paperclipai/paperclip/actions/runs/30054330748)
hit `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` (committed `pnpm-lock.yaml`
drifted from `patchedDependencies` in package metadata), and run
[30050197392](https://github.com/paperclipai/paperclip/actions/runs/30050197392)
hit `no space left on device` during the multi-arch buildx export
> - This pull request hardens the publish job against both failure
modes: it refreshes the lockfile (lockfile-only, guarded) before the
build, and frees runner disk space before buildx setup
> - The benefit is that image publishing keeps working through routine
lockfile drift and the growing multi-arch build footprint, so `ghcr.io`
images stay current with master
## Linked Issues or Issue Description
- Refs #8286 — same class of Docker-build lockfile mismatch failure
- Refs #8827 — pnpm 9.15.x pin / lockfile regeneration discussion
- Note: the immediate lockfile drift on master was fixed by #10132; the
refresh step here prevents the *next* drift from breaking image
publishing again
## What Changed
- Added a pnpm + Node setup and a **"Refresh lockfile for Docker build
context"** step to the image job in `.github/workflows/docker.yml`: runs
`pnpm install --lockfile-only --ignore-scripts --no-frozen-lockfile`,
exits cleanly if nothing changed, and **fails the job if anything other
than `pnpm-lock.yaml` was modified** by the refresh
- Added a **"Free runner disk"** step (before buildx setup) that prunes
the pnpm store, apt caches, preinstalled toolchains
(`/usr/share/dotnet`, Android SDK, Swift, Boost, PowerShell, GHC,
CodeQL/PyPy/Ruby toolcache), and dangling Docker state, logging `df -h`
before/after
- No changes outside the workflow file (54 added lines, nothing removed)
## Verification
- Pulled the logs of both failed master runs and matched each failure to
the step that addresses it:
[30054330748](https://github.com/paperclipai/paperclip/actions/runs/30054330748)
failed with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`,
[30050197392](https://github.com/paperclipai/paperclip/actions/runs/30050197392)
failed with `no space left on device` during the buildx export
- Confirmed pnpm `9.15.4` in the new setup step matches the repo
`packageManager` field and the version used in the Dockerfile, so the
refreshed lockfile is generated by the same pnpm the image build
consumes
- Validated the workflow YAML parses cleanly
- The workflow triggers on master pushes / manual dispatch; the
definitive check is the first master run after merge — reviewers can
also `workflow_dispatch` it from this branch if desired
## Risks
- The lockfile refresh runs with `--ignore-scripts` and a guard that
aborts on any non-lockfile change, so it cannot silently pull unexpected
code into the image; worst case it fails the job with a clear diff
- The published image could be built from a refreshed lockfile that
differs from the committed one when drift exists — that keeps publishing
alive but can mask drift on master, which still needs the committed
lockfile fixed (as #10132 did)
- Disk cleanup removes preinstalled toolchains only on the ephemeral
runner for this job; other jobs/workflows are unaffected
- Low risk overall: additive steps in a single workflow file
## Model Used
- Claude (Anthropic) — `claude-fable-5` (Claude Code agent harness,
extended thinking, tool use). Used to diagnose the failing CI runs from
logs, author the workflow changes, and prepare this PR.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no runtime code touched;
workflow YAML validated — see Verification)
- [ ] I have added or updated tests where applicable (n/a — CI workflow
change)
- [x] I have updated relevant documentation to reflect my changes (none
needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending — will confirm once
checks run)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review pass)
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip publishes Docker images from this repo that self-hosters
and orchestration tooling deploy; the server refuses to start when its
database is missing any schema migration the build bundles
(`ensureMigrations`)
> - Anything that deploys these images therefore needs to know an
image's schema expectations *before* deploying it — today that requires
pulling the image or checking out the matching commit, both heavyweight
for tooling that just wants to answer "will this image boot against a
database migrated to N?"
> - Getting this wrong is expensive: an image ahead of the applied
schema crash-loops at startup and fails healthchecks after deployment
resources are already created
> - This pull request labels every published image with its bundled
migration set (last migration file and count), computed at build time
from `packages/db/src/migrations` in the same tree the image is built
from
> - The benefit is image/schema compatibility verification with two
cheap registry requests (manifest + config blob), no pull, and no drift
risk between label and image contents
## Linked Issues or Issue Description
No existing public issue; inline description per the feature request
template:
- **Subsystem affected**: Docker image publishing
(`.github/workflows/docker.yml`), `packages/db` migrations
- **Problem or motivation**: deployment tooling cannot cheaply determine
which schema migrations a published image expects; the only options are
pulling the image or checking out the matching commit. Deploying an
image whose bundled migrations exceed the applied schema makes the
server refuse to start, so this check is needed *before* resources are
created.
- **Proposed solution**: OCI labels
(`io.github.paperclipai.schema.last-migration`,
`io.github.paperclipai.schema.migration-count`), computed from the
migrations directory at build time via the existing
`docker/metadata-action` step. Keys use org-based reverse-DNS (the
GitHub org) so the label contract survives product-domain migrations.
- **Alternatives considered**: a schema manifest published beside the
image (second artifact to keep in sync — rejected); encoding schema info
in tags (tags already carry semver/sha meaning — rejected).
- **Roadmap alignment**: checked `ROADMAP.md` — no overlap with planned
core work; this is build metadata only.
## What Changed
- `.github/workflows/docker.yml`: a `Compute schema migration labels`
step (`ls` + `sort` over `packages/db/src/migrations/*.sql`) feeding two
custom labels into the existing `docker/metadata-action` step.
## Verification
- Workflow YAML validated locally.
- The label-computation commands run against the current tree produce
`last=0181_decision_training_retention_policy.sql`, `count=180`.
- After merge, verify with: fetch the image config blob for a fresh
`sha-*` tag from ghcr and confirm both `io.github.paperclipai.schema.*`
labels are present.
## Risks
- Low. Labels are metadata only; no change to image contents. If the
migrations directory ever moves, the label step fails the workflow
loudly (`ls` exits non-zero) rather than publishing wrong labels.
## Model Used
- Claude (Anthropic) — model id `claude-fable-5`, via the Claude Code
CLI harness with tool use (shell, file edits). Change authored and
verified agent-assisted, human-directed.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (YAML validation; label
commands — no app code changed)
- [x] I have added or updated tests where applicable (n/a — CI metadata
only)
- [x] I have updated relevant documentation to reflect my changes (n/a —
workflow comment documents the labels)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Devin Foley
Tonio
Dotta
github-actions[bot]
Nicky Leach
Zannis Kalampoukis
dependabot[bot]
Dotta
albttx