ci: cache compiled Docker Rust dependencies separately from source (#13329)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images that contain the native Rust Runner.
> - The image already builds that Runner before copying ordinary app
source.
> - A Rust source change still invalidates its entire compiled
dependency layer.
> - Compiled dependencies can survive source changes when their recipe
is unchanged.
> - This PR adds a separate locked dependency build before compiling the
real workspace.

## Linked Issues or Issue Description

Refs #13195. A search of related Docker and Cargo cache PRs found no
duplicate dependency-recipe change.

**What existing behavior does this improve?**

Docker image build time after Rust source or embedded protocol changes.

**Current behavior**

The `runner-build` stage compiles dependencies and workspace code in one
layer. In Cloud readiness run 34698143548, that stage took about 3m48s
when its cache was unavailable.

**Proposed behavior**

Generate a recipe with pinned cargo-chef 0.1.73. Build locked release
dependencies in `runner-deps`, then copy and compile real Rust source
and embedded protocol inputs in `runner-build`. Source edits can reuse
the dependency layer from the existing registry cache.

**Reason and benefit**

Reduce dependency recompilation during source changes and merge bursts.
Expected savings are roughly 2–4 minutes when the old native layer would
miss but dependency layers are available. Full cold builds also pay for
the recipe tool installation. Ordinary app-only cache hits gain little
from this change.

**Breaking changes**

None to the shipped application or image tags. The recipe tool and
compiled dependencies remain in build stages.

## What Changed

- Install a pinned recipe generator with its locked dependencies and the
existing package-owned compiler.
- Add recipe planning and compiled dependency stages. Use the same
release profile, package, binary, and lockfile enforcement as the real
native build.
- Remove generated source stubs before copying actual source. Preserve
protocol inputs, timestamp normalization, binary staging, and
application checks.
- Add Docker cache wiring regressions and update the Docker cache
documentation.
- Run a two-build probe in Docker Runner check. It requires dependency
reuse, changed real binary metadata after a source edit, and a changed
recipe after a dependency declaration edit. It uses a disposable
tracked-source context and exports only small metadata files.

## Verification

- Passed all five Docker build-stamp and dependency-cache tests with
`pnpm exec vitest run server/src/__tests__/docker-build-stamp.test.ts`.
- Passed the local ARM64 `docker buildx build --target runner-build
--progress plain`. Local Docker then hit storage errors during a runtime
probe; cache invalidation verification continues on GitHub-hosted Linux.
- Passed `bash -n scripts/check-docker-runner-cache.sh`, `actionlint`,
and `git diff --check`.
- Passed a [Linux AMD64 cache
probe](https://github.com/paperclipai/paperclip/actions/runs/34711042199)
against the PR source: dependencies compiled in 3m49s for the baseline
and were `CACHED` after a source edit; real source compilation took
about 37 seconds. Binary metadata changed and dependency declaration
changes altered the recipe. The permanent probe is also running in
latest-head Docker Runner check.
- Passed latest-head [Docker Runner
check](https://github.com/paperclipai/paperclip/actions/runs/34711145160),
including the permanent source/dependency invalidation probe.
- Passed full [PR
verification](https://github.com/paperclipai/paperclip/actions/runs/34711145352/attempts/2):
typecheck, all grouped tests, native verification, build, release dry
run, and browser checks. One unrelated signoff-policy browser test
failed waiting for a heartbeat run on attempt 1; only that failed shard
and dependent checks were retried, and passed.
- Latest-head Greptile is 5/5 with no unresolved findings. Full local
tests/build were limited by local disk exhaustion; Linux CI completed
those checks.

## Risks

- The two-build CI probe has a 20-minute job limit to cover the cold
build and source rebuild. It adds no AWS routing.
- A fully cold build must install cargo-chef and populate the dependency
layer. Both become reusable registry layers; no Actions cache is added.
- The recipe and final build must keep the same compiler, build profile,
package, binary, and directory layout. A source-change rebuild probe
checks real cache reuse and binary invalidation.
- Dependency or compiler changes still require rebuilding dependencies.
Existing image verification and full-SHA publication gates remain
unchanged.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

.github/workflows/docker-runner-check.yml

@@ -6,6 +6,7 @@ on:
       - .github/workflows/docker-runner-check.yml
       - Dockerfile
       - .dockerignore
+      - scripts/check-docker-runner-cache.sh
       - packages/paperclip-runner/rust-toolchain.toml
       - packages/paperclip-runner/runner/**
       - packages/paperclip-runner/protocol/**
@@ -20,7 +21,7 @@ jobs:
   runner:
     name: Compile isolated native Runner
     runs-on: ubuntu-latest
-    timeout-minutes: 15
+    timeout-minutes: 20
     permissions:
       contents: read
     steps:
@@ -30,8 +31,8 @@ jobs:
           persist-credentials: false
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
-      # Compile the real target with the real .dockerignore. This catches new
-      # Cargo or embedded protocol inputs that the isolated COPY set omits.
-      # No registry credentials, cache imports/exports, or image publication.
-      - name: Compile the Runner from its isolated Docker context
-        run: docker buildx build --target runner-build --progress plain .
+      # Compile the real target, then change source in a disposable context.
+      # Require dependency-layer reuse and changed metadata from the real binary.
+      # No registry credentials, external cache, or image publication.
+      - name: Verify native build and dependency cache reuse
+        run: bash scripts/check-docker-runner-cache.sh

Dockerfile

@@ -88,7 +88,27 @@ RUN set -eux; \
 COPY packages/paperclip-runner/rust-toolchain.toml /tmp/runner-toolchain/rust-toolchain.toml
 RUN cd /tmp/runner-toolchain && rustup show
 
-FROM rust-toolchain AS runner-build
+# Pin the recipe generator and its dependency lockfile. It is a build-only tool
+# and uses the same package-owned compiler as both native build stages.
+FROM rust-toolchain AS rust-chef
+RUN cd /tmp/runner-toolchain && cargo install cargo-chef --version 0.1.73 --locked
+
+FROM rust-chef AS runner-plan
+WORKDIR /app/packages/paperclip-runner
+COPY packages/paperclip-runner/rust-toolchain.toml ./
+COPY packages/paperclip-runner/runner ./runner
+RUN cd runner && cargo chef prepare --recipe-path /tmp/runner-recipe.json
+
+FROM rust-chef AS runner-deps
+WORKDIR /app/packages/paperclip-runner/runner
+COPY packages/paperclip-runner/rust-toolchain.toml ../
+# The recipe changes only when dependency manifests, the lockfile, or target
+# metadata change. Source edits can reuse this compiled dependency layer.
+COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json
+RUN cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd --recipe-path /tmp/runner-recipe.json \
+  && find . -mindepth 1 -maxdepth 1 ! -name target -exec rm -rf {} +
+
+FROM runner-deps AS runner-build
 WORKDIR /app/packages/paperclip-runner
 # Rust embeds protocol schemas and fixtures with include_str!. Keep those
 # alongside the complete Cargo workspace so every compile-time input keys

doc/DOCKER.md

@@ -342,11 +342,20 @@ Notes:
 ## Native Runner build cache
 
 The image compiles the native Runner in `runner-build`, before copying the
-application source. That stage includes the pinned Rust compiler, the complete
-Cargo workspace and lockfile, and the protocol schemas and fixtures embedded
-by Rust. Changes to those inputs rebuild the native binary. Ordinary server or
-UI changes can reuse it through the existing registry cache (`mode=max`). Each
-platform gets its own native build; no cross-architecture binary is reused.
+application source. A pinned `cargo-chef` generates a dependency recipe in
+`runner-plan`. The separate `runner-deps` stage compiles that recipe with the
+package-owned Rust compiler. Both the dependency build and the real binary use
+the release profile and locked Cargo dependencies. The recipe stage never
+modifies source in the checkout.
+
+Changes to Rust source or embedded protocol inputs rebuild the real binary but
+can reuse compiled dependencies when the recipe is unchanged. Dependency
+manifests, the Cargo lockfile, target metadata, or compiler changes invalidate
+the relevant cache. Ordinary server or UI changes can reuse the entire native
+build through the existing registry cache (`mode=max`). Each platform gets its
+own native build; no cross-architecture binary is reused. No additional GitHub
+Actions cache is created. A cold build also installs the recipe generator and
+compiles dependencies, so the savings apply after those layers are available.
 
 The application build inherits that stage and still runs the normal server
 build, including Cargo, binary staging, and generated-contract checks. Rust
@@ -357,6 +366,13 @@ directory as before. Cache misses only cost compilation time.
 
 Pull requests that change the Dockerfile, Docker ignore rules, or Runner native
 inputs also build the isolated `runner-build` target in `Docker Runner check`.
-This compiles against the actual reduced context and catches missing embedded
-inputs before the post-merge image build. It uses a GitHub-hosted runner with
-read-only repository access and does not publish images or cache artifacts.
+The check runs `bash scripts/check-docker-runner-cache.sh` against a disposable
+copy of tracked source and the actual Docker ignore rules. It compiles a baseline,
+changes a Rust metadata constant, and rebuilds. It requires a cached dependency
+build, an unchanged dependency recipe, and changed metadata from the real binary.
+It also verifies that a dependency declaration change alters the recipe. The
+probe exports only small metadata files, avoiding a large image import into the
+Docker daemon. It catches missing embedded inputs before the post-merge build.
+It uses a GitHub-hosted runner with read-only repository access and does not
+publish images or cache artifacts. Allow up to 20 minutes for its cold build and
+source rebuild.

scripts/check-docker-runner-cache.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Build the real Docker target twice in a disposable copy of tracked source.
+# Export only metadata, avoiding a multi-gigabyte test image in the daemon.
+set -euo pipefail
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")"
+trap 'rm -rf "$probe_dir"' EXIT
+mkdir "$probe_dir/context"
+cd "$repo_root"
+git ls-files -z | tar -cf - --null -T - | tar -xf - -C "$probe_dir/context"
+cd "$probe_dir/context"
+export PROBE_DIR="$probe_dir"
+cp Dockerfile "$probe_dir/cache-probe.Dockerfile"
+cat >> "$probe_dir/cache-probe.Dockerfile" <<'DOCKER'
+FROM runner-build AS cache-proof
+RUN ./runner/target/release/paperclip-runnerd --build-metadata > /metadata.json
+FROM scratch AS cache-proof-export
+COPY --from=cache-proof /metadata.json /metadata.json
+COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json
+FROM scratch AS recipe-proof-export
+COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json
+DOCKER
+build_proof() {
+  docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$1" --progress plain . 2>&1 | tee "$probe_dir/$1.log"
+}
+build_proof baseline
+python3 - <<'CHECK'
+from pathlib import Path
+p=Path('packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs')
+s=p.read_text(); needle='paperclip-runner/runnerd-build-metadata/v1'
+assert s.count(needle)==1
+p.write_text(s.replace(needle,needle+'-cache-probe'))
+CHECK
+build_proof source-change
+python3 - <<'CHECK'
+import os,json,re
+from pathlib import Path
+root=Path(os.environ['PROBE_DIR'])
+before=json.loads((root/'baseline/metadata.json').read_text())
+after=json.loads((root/'source-change/metadata.json').read_text())
+assert before['schema']=='paperclip-runner/runnerd-build-metadata/v1'
+assert after['schema']==before['schema']+'-cache-probe'
+assert (root/'baseline/recipe.json').read_bytes()==(root/'source-change/recipe.json').read_bytes()
+log=(root/'source-change.log').read_text()
+step=re.search(r'#(\d+) \[runner-deps[^\n]+ RUN cargo chef cook',log)[1]
+assert f'#{step} CACHED' in log
+assert 'Compiling paperclip-runner-core' in log
+print('PASS: unchanged dependency recipe and cached cook layer; real binary changed.')
+p=Path('packages/paperclip-runner/runner/Cargo.toml')
+s=p.read_text(); assert 'serde_json = "1.0"' in s
+p.write_text(s.replace('serde_json = "1.0"','serde_json = ">=1.0.0, <2.0.0"'))
+CHECK
+docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target recipe-proof-export --output "type=local,dest=$probe_dir/manifest-change" --progress plain .
+python3 - <<'CHECK'
+from pathlib import Path
+import os
+root=Path(os.environ['PROBE_DIR'])
+assert (root/'source-change/recipe.json').read_bytes()!=(root/'manifest-change/recipe.json').read_bytes()
+print('PASS: dependency declaration change invalidates the recipe.')
+CHECK

server/src/__tests__/docker-build-stamp.test.ts

@@ -76,3 +76,32 @@ describe("docker build-stamp wiring", () => {
     ).toBeGreaterThanOrEqual(2);
   });
 });
+
+
+describe("Docker Rust dependency cache", () => {
+  it("caches the locked dependency recipe separately from source and per-build metadata", () => {
+    const chef = stageBody(dockerfile, "rust-chef");
+    const planner = stageBody(dockerfile, "runner-plan");
+    const dependencies = stageBody(dockerfile, "runner-deps");
+    expect(chef).toContain("FROM rust-toolchain AS rust-chef");
+    expect(chef).toMatch(/cargo install cargo-chef --version \d+\.\d+\.\d+ --locked/);
+    expect(planner).toContain("COPY packages/paperclip-runner/runner ./runner");
+    expect(planner).toContain("cargo chef prepare --recipe-path /tmp/runner-recipe.json");
+    expect(dependencies).toContain("FROM rust-chef AS runner-deps");
+    expect(dependencies).toContain("COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json");
+    expect(dependencies).toContain("cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd");
+    expect(dependencies).not.toMatch(/COPY .*\.\/runner|COPY .*\.\/protocol|COPY \. \.|PAPERCLIP_BUILD_COMMIT/);
+  });
+
+  it("rebuilds real workspace code and embedded protocol inputs after cooking dependencies", () => {
+    const native = stageBody(dockerfile, "runner-build");
+    expect(native).toContain("FROM runner-deps AS runner-build");
+    for (const source of ["runner", "protocol"]) {
+      expect(native.indexOf(`COPY packages/paperclip-runner/${source} ./${source}`))
+        .toBeLessThan(native.indexOf("cargo build --release"));
+      expect(native).toContain(`COPY packages/paperclip-runner/${source} ./${source}`);
+    }
+    expect(native).toContain("cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd");
+    expect(stageBody(dockerfile, "build")).toContain("FROM runner-build AS build");
+  });
+});