build: make image layer caching actually hit (#10571)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Published container images are the deployable unit for self-hosted
and managed instances, so merge-to-image latency bounds every deploy
iteration
> - The docker workflow configures BuildKit caching, but builds still
ran ~12+ minutes essentially cold
> - Two causes: the most expensive layer (four CLI toolchains + apt) is
ordered after the always-changing app copy so it can never cache, and
the type=gha cache's 10GB repo cap means the two multi-arch mode=max
jobs evict each other
> - This pull request reorders the tool layer above the app copy (with a
weekly epoch so @latest tools keep advancing) and switches both jobs to
registry-backed cache in ghcr
> - The benefit is that warm builds shrink to roughly the app build +
push, targeting the sub-5-minute range together with the amd64-only
cloud variant

## Linked Issues or Issue Description

No existing public issue — inline description following the feature
request template:

**Subsystem affected**

CI / release publishing (docker workflow, Dockerfile)

**Problem or motivation**

Despite `cache-from/cache-to` being configured, image builds run
effectively cold: (1) the production stage installs four CLI toolchains
+ apt packages *after* `COPY --from=build /app /app`, and since the app
copy changes every commit, that most-expensive layer rebuilds every
build, per arch; (2) the `type=gha` BuildKit cache is capped at 10GB per
repository, and two multi-arch `mode=max` jobs overflow and evict each
other's entries.

**Proposed solution**

Order the tool/OS layer before the app copy (it references nothing from
`/app`), refresh it weekly via a `CLI_TOOLS_CACHE_EPOCH` build arg so
the `@latest` tools don't freeze in the cache, and move both jobs to
registry-backed BuildKit cache (`:buildcache` / `:buildcache-cloud` refs
in ghcr, no size cap, separate refs so the parallel jobs don't clobber
each other).

**Alternatives considered**

Pinning CLI tool versions instead of the weekly epoch — more
deterministic, but adds a version-bump chore; the weekly epoch preserves
current freshness semantics with bounded staleness. Keeping type=gha
with `mode=min` — smaller cache but loses intermediate-stage reuse,
which is where most of the win is.

**Roadmap alignment**

Not on ROADMAP.md; CI/publishing speed improvement only.

## What Changed

- `Dockerfile`: the production stage's tool/OS `RUN` (npm --global CLIs,
apt, `/paperclip` setup) moves above `COPY --from=build /app /app`; new
`CLI_TOOLS_CACHE_EPOCH` arg consumed by that layer. The `cloud` stage is
unaffected — it only layers plugin dists on top of the finished
production stage.
- `.github/workflows/docker.yml`: both jobs stamp the ISO week into
`CLI_TOOLS_CACHE_EPOCH`, and both switch `cache-from/cache-to` from
`type=gha` to `type=registry` with per-job refs.
- Includes the one-line amd64-only cloud-variant commit from #10570 so
the two PRs can't conflict; if #10570 merges first, this PR rebases down
to a single commit automatically.

## Verification

- Image content is unchanged by layer reordering: the moved `RUN`
references nothing from `/app`, and Docker layer ordering only affects
caching, not the final filesystem (tool installs and app copy touch
disjoint paths).
- The cache ref is written only by this workflow — `docker.yml` runs on
master/tag pushes, never on PRs — so the workflow's existing "no shared
caches into build inputs" supply-chain stance is unchanged (BuildKit
layer cache was already accepted via type=gha; the registry backend has
the same writer trust).
- Runtime proof lands with the first two master builds after merge: the
first warms the cache, the second should show the tool layer and
deps/build stages as CACHED in the build log, with wall clock dropping
accordingly. I'll be watching those as part of managed-deploy work.

## Risks

- Low. Worst case the registry cache misses (cold-build behavior, same
as today). The weekly epoch means CLI tools update at most a week late
inside images; a release built mid-week ships the tools from that week's
first build. Cache refs add two small artifacts to ghcr.

## Model Used

Claude Fable 5 (`claude-fable-5`, extended thinking, via Claude Code
with tool use and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (no test-affecting changes)
- [x] I have added or updated tests where applicable (n/a — build config
and layer ordering only)
- [x] I have updated relevant documentation to reflect my changes
(in-file comments document both mechanisms)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

.github/workflows/docker.yml

@@ -44,6 +44,13 @@ jobs:
           echo "version=${version}" >> "$GITHUB_OUTPUT"
           echo "Stamping build version: ${version:-<none>}"
 
+      # ISO week stamp for the Dockerfile's tool layer: the layer caches
+      # across commits and re-pulls the @latest CLI tools when the week rolls
+      # over, instead of on every build.
+      - name: Compute tool cache epoch
+        id: tools-epoch
+        run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
+
       - name: Setup pnpm
         uses: pnpm/action-setup@v6
         with:
@@ -149,10 +156,16 @@ jobs:
           build-args: |
             PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }}
             PAPERCLIP_BUILD_COMMIT=${{ github.sha }}
+            CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }}
           platforms: linux/amd64,linux/arm64
           push: true
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+          # Registry-backed BuildKit cache instead of type=gha: the Actions
+          # cache is capped at 10GB per repo, and two multi-arch mode=max jobs
+          # evict each other, so most builds ran effectively cold. The cache
+          # ref lives in ghcr next to the image and is written only by this
+          # workflow (docker.yml runs on master/tag pushes, never on PRs).
+          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
+          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
 
@@ -188,6 +201,13 @@ jobs:
           echo "version=${version}" >> "$GITHUB_OUTPUT"
           echo "Stamping build version: ${version:-<none>}"
 
+      # ISO week stamp for the Dockerfile's tool layer: the layer caches
+      # across commits and re-pulls the @latest CLI tools when the week rolls
+      # over, instead of on every build.
+      - name: Compute tool cache epoch
+        id: tools-epoch
+        run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
+
       - name: Setup pnpm
         uses: pnpm/action-setup@v6
         with:
@@ -297,13 +317,17 @@ jobs:
             CLOUD_BUNDLED_PLUGINS=daytona
             PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }}
             PAPERCLIP_BUILD_COMMIT=${{ github.sha }}
+            CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }}
           # amd64 only, unlike the self-hosted image above: the cloud variant
           # is consumed exclusively by managed-deployment hosts, which run
           # amd64. The QEMU-emulated arm64 half dominated this job's wall
           # clock, and dropping it roughly halves time-to-deployable-image.
           platforms: linux/amd64
           push: true
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+          # Registry-backed BuildKit cache, separate ref from the self-hosted
+          # job so the two parallel builds never clobber each other's cache
+          # manifest (see the rationale on the job above).
+          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-cloud
+          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-cloud,mode=max
           tags: ${{ steps.meta-cloud.outputs.tags }}
           labels: ${{ steps.meta-cloud.outputs.labels }}

Dockerfile

@@ -67,9 +67,17 @@ ARG PAPERCLIP_BUILD_VERSION=""
 # falls back to PAPERCLIP_BUILD_COMMIT when git is unavailable, which feeds the
 # /api/health `commit` field that deploy tooling verifies. Empty locally.
 ARG PAPERCLIP_BUILD_COMMIT=""
+# Refreshes the tool layer below when it changes (CI stamps an ISO week, so
+# the @latest CLI tools advance weekly). Without it the cached layer would
+# freeze the tools until an unrelated cache bust.
+ARG CLI_TOOLS_CACHE_EPOCH=""
 WORKDIR /app
-COPY --chown=node:node --from=build /app /app
-RUN npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest \
+# Tool and OS layer BEFORE the app copy: it references nothing from /app, and
+# the app copy changes on every commit — ordered the other way around, this
+# (the single most expensive layer: four CLI toolchains + apt, per arch) can
+# never hit the layer cache and rebuilds on every build.
+RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \
+  && npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest \
   && apt-get update \
   && apt-get install -y --no-install-recommends openssh-client jq \
   && rm -rf /var/lib/apt/lists/* \
@@ -79,6 +87,8 @@ RUN npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/cod
 COPY scripts/docker-entrypoint.sh /usr/local/bin/
 RUN chmod +x /usr/local/bin/docker-entrypoint.sh
 
+COPY --chown=node:node --from=build /app /app
+
 ENV NODE_ENV=production \
   HOME=/paperclip \
   HOST=0.0.0.0 \