ci: reuse trusted cache for Daytona images (#12862)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The full-stack runner campaign checks local and Daytona runner behavior. > - A Daytona image content miss starts a cold multi-stage Docker build. > - Stable dependency and agent CLI layers take most of the image build time. > - Development targets must not write shared cache state. > - This pull request adds a registry cache with a default-branch write gate. > - It also puts volatile source inputs after stable install layers. > - The benefit is a shorter Daytona image build without weaker secret isolation. ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves the Daytona runner image stage in the full-stack E2E workflow. **Subsystem affected** The GitHub Actions runner E2E workflow and its Daytona Docker image are affected. **Current behavior** Each new Daytona image content ID starts with an empty BuildKit cache. A runner source change also invalidates dependency and agent CLI install layers because volatile inputs occur before those layers. **Proposed behavior** All authorized campaigns can read one GHCR BuildKit cache. Only a campaign whose target ref is the repository default branch can update that cache. The Dockerfile installs dependencies and agent CLIs before it consumes volatile runner source or revision metadata. **Reason and benefit** The paid runner matrix spends several minutes building the image before any selected cell can start. Cache reuse removes repeated stable setup work and makes focused Daytona iterations faster. **Breaking changes** None. The immutable content tag, digest inspection, Cosign signature, image labels, pinned base images, and provider credential boundary stay unchanged. ## What Changed - Read a registry-backed BuildKit cache for Daytona image content misses. - Export the cache only when the resolved target ref is the default branch. - Keep provider credentials outside the image build and cache. - Install provider-pack dependencies before runner source is copied. - Keep expensive agent CLI installs before source revision metadata. - Add workflow and Docker layer-order contract checks. ## Verification - `prettier --write .github/workflows/runner-full-stack-e2e.yml tests/runner-e2e/daytona-image.test.ts tests/runner-e2e/workflow-security.test.ts` - `actionlint .github/workflows/runner-full-stack-e2e.yml` - `git diff --check` - I did not run a test suite or Docker image build locally. The requested iteration policy reserves those checks for GitHub Actions. ## Risks Low risk. BuildKit can use a cache record only when its content key matches the build instruction and input. Development targets have read-only cache access. The cache contains public source and build outputs, but it does not receive provider credentials or the GitHub token as Docker build inputs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with GPT-5, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
GitHub
parent
bcc6fe7a44
commit
8430bd897f
|
|
@ -409,7 +409,10 @@ jobs:
|
|||
NEEDS_DAYTONA: ${{ needs.catalog.outputs.needs_daytona }}
|
||||
IMAGE_CONTENT_ID: ${{ needs.catalog.outputs.daytona_image_content_id }}
|
||||
IMAGE_TAG: ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-${{ needs.catalog.outputs.daytona_image_content_id }}
|
||||
IMAGE_CACHE: ghcr.io/paperclipai/paperclip-daytona-runner:e2e-buildcache-amd64
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
TARGET_REF: ${{ needs.authorize.outputs.target_ref }}
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$NEEDS_DAYTONA" != true ]; then
|
||||
|
|
@ -425,12 +428,24 @@ jobs:
|
|||
if docker buildx imagetools inspect "$IMAGE_TAG" >/dev/null 2>&1; then
|
||||
digest="$(docker buildx imagetools inspect "$IMAGE_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')"
|
||||
else
|
||||
cache_args=(
|
||||
--cache-from "type=registry,ref=${IMAGE_CACHE}"
|
||||
)
|
||||
if [ "$TARGET_REF" = "refs/heads/$DEFAULT_BRANCH" ]; then
|
||||
cache_args+=(
|
||||
--cache-to "type=registry,ref=${IMAGE_CACHE},mode=max"
|
||||
)
|
||||
echo '::notice title=Daytona image cache::Publishing cache from the trusted default-branch target'
|
||||
else
|
||||
echo '::notice title=Daytona image cache::Using the default-branch cache without publishing development-branch layers'
|
||||
fi
|
||||
docker buildx build \
|
||||
--platform linux/amd64 \
|
||||
--build-arg "PAPERCLIP_RUNNER_CONTENT_ID=${IMAGE_CONTENT_ID}" \
|
||||
--build-arg "PAPERCLIP_RUNNER_SOURCE_REVISION=${TARGET_SHA}" \
|
||||
--file docker/daytona-runner/Dockerfile \
|
||||
--tag "$IMAGE_TAG" \
|
||||
"${cache_args[@]}" \
|
||||
--push \
|
||||
.
|
||||
digest="$(docker buildx imagetools inspect "$IMAGE_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')"
|
||||
|
|
|
|||
|
|
@ -13,24 +13,17 @@ RUN cargo build --locked --release -p paperclip-runner-core --bin paperclip-runn
|
|||
&& strip /workspace/packages/paperclip-runner/runner/target/release/paperclip-runnerd
|
||||
|
||||
FROM node:24-bookworm@sha256:9137a20e25879e0b557227b57e3ee4e9af4bde29eb3db66134cd1723e84f830b AS provider-pack-build
|
||||
ARG PAPERCLIP_RUNNER_SOURCE_REVISION
|
||||
RUN test -n "${PAPERCLIP_RUNNER_SOURCE_REVISION}"
|
||||
RUN corepack enable && corepack prepare pnpm@9.15.4 --activate
|
||||
WORKDIR /workspace
|
||||
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml .npmrc tsconfig.base.json ./
|
||||
COPY patches ./patches
|
||||
COPY scripts/link-plugin-dev-sdk.mjs ./scripts/link-plugin-dev-sdk.mjs
|
||||
COPY \
|
||||
packages/paperclip-eval-kernel/package.json \
|
||||
packages/paperclip-eval-kernel/tsconfig.json \
|
||||
./packages/paperclip-eval-kernel/
|
||||
COPY packages/paperclip-eval-kernel/package.json ./packages/paperclip-eval-kernel/package.json
|
||||
COPY packages/paperclip-runner/package.json ./packages/paperclip-runner/package.json
|
||||
RUN pnpm install --frozen-lockfile --filter '@paperclipai/paperclip-runner...'
|
||||
COPY packages/paperclip-eval-kernel/tsconfig.json ./packages/paperclip-eval-kernel/tsconfig.json
|
||||
COPY packages/paperclip-eval-kernel/src ./packages/paperclip-eval-kernel/src
|
||||
COPY \
|
||||
packages/paperclip-runner/package.json \
|
||||
packages/paperclip-runner/styles.css \
|
||||
packages/paperclip-runner/tsconfig.json \
|
||||
packages/paperclip-runner/tsconfig.surfaces.json \
|
||||
./packages/paperclip-runner/
|
||||
COPY packages/paperclip-runner/styles.css packages/paperclip-runner/tsconfig.json packages/paperclip-runner/tsconfig.surfaces.json ./packages/paperclip-runner/
|
||||
COPY packages/paperclip-runner/protocol ./packages/paperclip-runner/protocol
|
||||
COPY packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs ./packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs
|
||||
COPY \
|
||||
|
|
@ -41,7 +34,8 @@ COPY \
|
|||
packages/paperclip-runner/scripts/generate-protocol-schema-module.mjs \
|
||||
./packages/paperclip-runner/scripts/
|
||||
COPY packages/paperclip-runner/src ./packages/paperclip-runner/src
|
||||
RUN pnpm install --frozen-lockfile --filter '@paperclipai/paperclip-runner...'
|
||||
ARG PAPERCLIP_RUNNER_SOURCE_REVISION
|
||||
RUN test -n "${PAPERCLIP_RUNNER_SOURCE_REVISION}"
|
||||
RUN pnpm --filter @paperclipai/paperclip-runner build:typescript \
|
||||
&& PAPERCLIP_RUNNER_SOURCE_REVISION="${PAPERCLIP_RUNNER_SOURCE_REVISION}" \
|
||||
node packages/paperclip-runner/scripts/build-provider-pack.mjs /provider-pack
|
||||
|
|
@ -51,13 +45,6 @@ RUN pnpm --filter @paperclipai/paperclip-runner build:typescript \
|
|||
# runner-specific addition is /usr/local/bin/paperclip-runnerd below.
|
||||
FROM daytonaio/sandbox:0.8.0@sha256:eadf88e4391072b7ad4bed27d9cadfc9fe9d8ed375d9219d34c2ccb518f213e3
|
||||
|
||||
ARG PAPERCLIP_RUNNER_CONTENT_ID
|
||||
ARG PAPERCLIP_RUNNER_SOURCE_REVISION
|
||||
RUN test -n "${PAPERCLIP_RUNNER_CONTENT_ID}" \
|
||||
&& test -n "${PAPERCLIP_RUNNER_SOURCE_REVISION}"
|
||||
LABEL io.paperclip.runner.content-id="${PAPERCLIP_RUNNER_CONTENT_ID}" \
|
||||
org.opencontainers.image.revision="${PAPERCLIP_RUNNER_SOURCE_REVISION}"
|
||||
|
||||
USER root
|
||||
|
||||
ENV PAPERCLIP_RUNNER_PROVIDER_PACK_ROOT=/opt/paperclip-runner/provider-pack
|
||||
|
|
@ -104,6 +91,14 @@ RUN set -eu; \
|
|||
COPY --from=runnerd-build /workspace/packages/paperclip-runner/runner/target/release/paperclip-runnerd /usr/local/bin/paperclip-runnerd
|
||||
COPY --from=provider-pack-build /provider-pack /opt/paperclip-runner/provider-pack
|
||||
|
||||
# Keep revision-dependent metadata below the stable agent CLI installation
|
||||
# layers. A source-only image miss can then reuse those expensive layers from
|
||||
# the trusted registry cache.
|
||||
ARG PAPERCLIP_RUNNER_CONTENT_ID
|
||||
ARG PAPERCLIP_RUNNER_SOURCE_REVISION
|
||||
RUN test -n "${PAPERCLIP_RUNNER_CONTENT_ID}" \
|
||||
&& test -n "${PAPERCLIP_RUNNER_SOURCE_REVISION}"
|
||||
|
||||
RUN set -eu; \
|
||||
chmod -R a+rX /opt/paperclip-runner/provider-pack; \
|
||||
printf '%s\n' 'export PATH=/opt/paperclip-runner/provider-pack/node_modules/.bin:$PATH' \
|
||||
|
|
@ -132,3 +127,6 @@ RUN /bin/sh -lc 'set -eu; \
|
|||
test "$(acpx --version)" = "0.13.1"; \
|
||||
test "$(claude-agent-acp --version)" = "0.70.0"; \
|
||||
test "$(codex-acp --version)" = "@agentclientprotocol/codex-acp 1.6.2"'
|
||||
|
||||
LABEL io.paperclip.runner.content-id="${PAPERCLIP_RUNNER_CONTENT_ID}" \
|
||||
org.opencontainers.image.revision="${PAPERCLIP_RUNNER_SOURCE_REVISION}"
|
||||
|
|
|
|||
|
|
@ -96,6 +96,18 @@ describe("runner E2E Daytona image contract", () => {
|
|||
expect(workflow).toContain(
|
||||
'--build-arg "PAPERCLIP_RUNNER_CONTENT_ID=${IMAGE_CONTENT_ID}"',
|
||||
);
|
||||
expect(workflow).toContain(
|
||||
"IMAGE_CACHE: ghcr.io/paperclipai/paperclip-daytona-runner:e2e-buildcache-amd64",
|
||||
);
|
||||
expect(workflow).toContain(
|
||||
'--cache-from "type=registry,ref=${IMAGE_CACHE}"',
|
||||
);
|
||||
expect(workflow).toContain(
|
||||
'--cache-to "type=registry,ref=${IMAGE_CACHE},mode=max"',
|
||||
);
|
||||
expect(workflow).toContain(
|
||||
'if [ "$TARGET_REF" = "refs/heads/$DEFAULT_BRANCH" ]; then',
|
||||
);
|
||||
expect(workflow).not.toContain("e2e-git-${{ github.sha }}");
|
||||
expect(workflow).toContain("cosign sign --yes");
|
||||
expect(workflow).toContain("docker logout ghcr.io");
|
||||
|
|
@ -124,6 +136,24 @@ describe("runner E2E Daytona image contract", () => {
|
|||
expect(workflow.indexOf("docker logout ghcr.io")).toBeLessThan(
|
||||
workflow.indexOf(`--format '{{json .Image}}'`),
|
||||
);
|
||||
const providerInstall = dockerfile.indexOf(
|
||||
"RUN pnpm install --frozen-lockfile --filter '@paperclipai/paperclip-runner...'",
|
||||
);
|
||||
const runnerSourceCopy = dockerfile.indexOf(
|
||||
"COPY packages/paperclip-runner/src ./packages/paperclip-runner/src",
|
||||
);
|
||||
const providerRevisionArg = dockerfile.indexOf(
|
||||
"ARG PAPERCLIP_RUNNER_SOURCE_REVISION",
|
||||
);
|
||||
const cliInstall = dockerfile.indexOf("RUN npm install -g");
|
||||
const finalMetadataArgs = dockerfile.lastIndexOf(
|
||||
"ARG PAPERCLIP_RUNNER_CONTENT_ID",
|
||||
);
|
||||
expect(providerInstall).toBeGreaterThan(0);
|
||||
expect(providerInstall).toBeLessThan(runnerSourceCopy);
|
||||
expect(providerInstall).toBeLessThan(providerRevisionArg);
|
||||
expect(cliInstall).toBeGreaterThan(0);
|
||||
expect(cliInstall).toBeLessThan(finalMetadataArgs);
|
||||
});
|
||||
|
||||
it("hashes the audited image dependency closure rather than the repository revision", async () => {
|
||||
|
|
@ -190,7 +220,10 @@ describe("runner E2E Daytona image contract", () => {
|
|||
"FROM pinned\n",
|
||||
);
|
||||
await writeFile(path.join(root, "package.json"), '{"private":true}\n');
|
||||
await writeFile(path.join(root, "pnpm-lock.yaml"), "lockfileVersion: 9\n");
|
||||
await writeFile(
|
||||
path.join(root, "pnpm-lock.yaml"),
|
||||
"lockfileVersion: 9\n",
|
||||
);
|
||||
await writeFile(
|
||||
path.join(root, "packages/paperclip-runner/package.json"),
|
||||
'{"name":"@paperclipai/paperclip-runner"}\n',
|
||||
|
|
@ -225,9 +258,7 @@ describe("runner E2E Daytona image contract", () => {
|
|||
path.join(root, "unrelated.txt"),
|
||||
"does not enter the image\n",
|
||||
);
|
||||
expect(
|
||||
await computeDaytonaImageContentId(options),
|
||||
).toBe(baseline);
|
||||
expect(await computeDaytonaImageContentId(options)).toBe(baseline);
|
||||
|
||||
for (const relativePath of [
|
||||
"docker/daytona-runner/Dockerfile",
|
||||
|
|
|
|||
|
|
@ -352,6 +352,30 @@ describe("public repository paid workflow security", () => {
|
|||
);
|
||||
expect(daytonaImageJob).toContain('echo "source_revision="');
|
||||
expect(daytonaImageJob).toContain('echo "content_id="');
|
||||
expect(daytonaImageJob).toContain(
|
||||
"IMAGE_CACHE: ghcr.io/paperclipai/paperclip-daytona-runner:e2e-buildcache-amd64",
|
||||
);
|
||||
expect(daytonaImageJob).toContain(
|
||||
"TARGET_REF: ${{ needs.authorize.outputs.target_ref }}",
|
||||
);
|
||||
expect(daytonaImageJob).toContain(
|
||||
"DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}",
|
||||
);
|
||||
const cacheRead = daytonaImageJob.indexOf(
|
||||
'--cache-from "type=registry,ref=${IMAGE_CACHE}"',
|
||||
);
|
||||
const trustedTargetCheck = daytonaImageJob.indexOf(
|
||||
'if [ "$TARGET_REF" = "refs/heads/$DEFAULT_BRANCH" ]; then',
|
||||
);
|
||||
const cacheWrite = daytonaImageJob.indexOf(
|
||||
'--cache-to "type=registry,ref=${IMAGE_CACHE},mode=max"',
|
||||
);
|
||||
expect(cacheRead).toBeGreaterThan(0);
|
||||
expect(trustedTargetCheck).toBeGreaterThan(cacheRead);
|
||||
expect(cacheWrite).toBeGreaterThan(trustedTargetCheck);
|
||||
expect(daytonaImageJob.slice(trustedTargetCheck, cacheWrite)).not.toContain(
|
||||
"secrets.",
|
||||
);
|
||||
const targetCodeJobs = [
|
||||
fullStack.slice(
|
||||
fullStack.indexOf(" catalog:"),
|
||||
|
|
|
|||
Loading…
Reference in New Issue