From 965a827ee75ada8ccb642ff199ad22a3b2f1834b Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 24 Jul 2026 08:22:34 -0700 Subject: [PATCH] feat(docker): publish a cloud image variant with built bundled plugins (#10157) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Managed (cloud-hosted) deployments configure instances through `PAPERCLIP_MANAGED_CONFIG`, including a `plugins.autoInstall` key list that the boot-time installer resolves against the bundled plugin catalog > - The installer requires each bundled plugin's `dist/manifest.js` (`server/src/services/bundled-plugins.ts`), but the published image only ships the sandbox providers' *source* — they are intentionally excluded from the pnpm workspace, and the Dockerfile never builds them > - Every managed auto-install therefore logs `bundled plugin bundle not present; skipping auto-install` and no sandbox provider can be provisioned through managed config > - Baking built plugins into the single published image would fix it but makes every self-hosted pull carry the providers' `node_modules` for a managed-only mechanism > - This pull request adds a `cloud` Dockerfile target extending `production` with built bundled plugins — parameterized by build arg and currently just `daytona` — published alongside the default image with a `-cloud` tag suffix > - The benefit is working plugin auto-provisioning for managed deployments while the self-hosted image stays byte-identical and the cloud variant only carries what is actually deployed ## Linked Issues or Issue Description Fixes #10158 (filed for this problem; no prior issue existed — searched for duplicate/related PRs and issues around bundled plugins, docker image variants, and auto-install). Summary: **What happened:** on a managed instance with `plugins.autoInstall: ["daytona"]` delivered via `PAPERCLIP_MANAGED_CONFIG`, boot logs `bundled plugin bundle not present; skipping auto-install` with `pluginPath: /app/packages/plugins/sandbox-providers/daytona`, and the plugin is never installed. **Expected:** the advertised bundled-catalog keys are installable from the published image. **Why:** the image ships plugin source without `dist/` — nothing in the Dockerfile builds the workspace-excluded sandbox providers. ## What Changed - `Dockerfile`: new `cloud-plugins` stage (based on `build`, so devDependencies are available for `tsc`) that installs and builds each provider named in the `CLOUD_BUNDLED_PLUGINS` build arg standalone (`pnpm install --ignore-workspace --no-lockfile && pnpm build`, exactly as the providers' READMEs prescribe), asserting `dist/manifest.js` exists per plugin and failing loudly on unknown names; new `cloud` stage = `production` + the built plugin tree. The arg defaults to `daytona` — the only provider managed deployments auto-install today; every entry adds its `node_modules` to the image, so the list grows only with actual need (a one-line workflow change). - `.github/workflows/docker.yml`: the existing build step is pinned to `target: production` (without this, the new trailing stage would silently become the default build target — this pin is what keeps the self-hosted image identical); new metadata + build-push steps publish the `cloud` target (with `CLOUD_BUNDLED_PLUGINS=daytona`) under the same tag set with a `-cloud` suffix (`sha--cloud`, `latest-cloud`, `-cloud`), same schema labels, reusing the GHA layer cache ## Verification - All seven sandbox providers build standalone from a clean checkout with the exact commands the new stage runs, each producing `dist/manifest.js` — so the current `daytona` default works and future list additions are known-good - The stage's shell loop was dry-run against the checkout (directory existence + per-plugin assertion logic) - Workflow YAML lints clean - **Not run:** a full multi-arch `docker build` (no local docker daemon). The `cloud` stage is additive and the default target is pinned, so the risk is contained to the new build step; the first master build after merge proves it end-to-end ## Risks - Self-hosted behavior: unchanged. The default image build is pinned to the `production` target, which produces the same layers as before this change; the `cloud` stages run only for the new build step. - The plugin installs in the `cloud-plugins` stage use `--no-lockfile` (the providers are workspace-excluded and lockfile-less by design), so plugin dependency resolution is not pinned at image-build time. This mirrors the existing Plugins-page install path, which resolves from npm at install time. - CI cost: one additional build-push per master push. It reuses the layer cache from the production build, so the marginal work is the single plugin's build layers. - An unknown name in `CLOUD_BUNDLED_PLUGINS`, or a provider that stops producing `dist/manifest.js`, fails the cloud build loudly rather than publishing a broken variant. ## Model Used Claude (Anthropic), model ID `claude-fable-5[1m]` via Claude Code CLI — extended thinking and tool use (code edits, standalone plugin build verification, workflow lint). ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] Self-hosted behavior unchanged (default build target pinned to `production`) - [x] One clear change: publish a cloud image variant with built bundled plugins --- .github/workflows/docker.yml | 41 ++++++++++ Dockerfile | 35 +++++++++ .../cloud-image-bundled-plugins.test.ts | 78 +++++++++++++++++++ 3 files changed, 154 insertions(+) create mode 100644 server/src/__tests__/cloud-image-bundled-plugins.test.ts diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index d108ffcfc9..1cf0640c62 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -120,9 +120,50 @@ jobs: uses: docker/build-push-action@v7 with: context: . + # Pin the self-hosted image to the production stage explicitly: + # the Dockerfile now declares a later `cloud` stage, and without a + # target the default would silently become that stage. + target: production platforms: linux/amd64,linux/arm64 push: true cache-from: type=gha cache-to: type=gha,mode=max tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} + + # The cloud variant carries built bundled plugins for managed + # deployments (see the `cloud` stage in the Dockerfile). Published + # under the same tag set with a `-cloud` suffix (sha--cloud, + # latest-cloud, -cloud). Reuses the layer cache from the + # production build, so this mostly adds the plugin-build layers. + - name: Docker meta (cloud) + id: meta-cloud + uses: docker/metadata-action@v6 + with: + images: ghcr.io/${{ github.repository }} + flavor: | + suffix=-cloud,onlatest=true + tags: | + type=raw,value=latest,enable={{is_default_branch}} + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + type=sha + labels: | + io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }} + io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }} + + - name: Build and push (cloud) + uses: docker/build-push-action@v7 + with: + context: . + target: cloud + # Space-separated sandbox-provider directory names to build into + # the variant; add here when managed deployments need another. + build-args: | + CLOUD_BUNDLED_PLUGINS=daytona + platforms: linux/amd64,linux/arm64 + push: true + cache-from: type=gha + cache-to: type=gha,mode=max + tags: ${{ steps.meta-cloud.outputs.tags }} + labels: ${{ steps.meta-cloud.outputs.labels }} diff --git a/Dockerfile b/Dockerfile index f07931cab9..e6a3cba9d1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -90,3 +90,38 @@ EXPOSE 3100 ENTRYPOINT ["docker-entrypoint.sh"] CMD ["node", "--import", "./server/node_modules/tsx/dist/loader.mjs", "server/dist/index.js"] + +# Cloud image variant (build with `--target cloud`): the production image +# plus built bundled sandbox-provider plugins. Managed instances receive a +# `plugins.autoInstall` key list through PAPERCLIP_MANAGED_CONFIG and +# install those plugins from the bundled catalog at boot +# (server/src/services/bundled-plugins.ts), which requires each plugin's +# dist/ to exist in the image — the default image ships only their source, +# so auto-install logs "bundle not present" and skips. The plugins are +# built in this separate target so the default (self-hosted) image stays +# lean; CI pins the default build to `--target production`, which is +# byte-identical to before this stage existed. +# +# The sandbox providers are intentionally excluded from the pnpm workspace +# (see pnpm-workspace.yaml), so each installs standalone exactly as its +# README prescribes. Installing in a `build`-based stage (not `production`) +# keeps devDependencies available for tsc: `production` sets +# NODE_ENV=production, which would make pnpm skip them. +# +# CLOUD_BUNDLED_PLUGINS is the space-separated list of sandbox-provider +# directory names to build into the variant. Only what managed deployments +# actually auto-install belongs here — every entry adds its node_modules +# to the image. Growing the list is a one-line workflow change. +FROM build AS cloud-plugins +ARG CLOUD_BUNDLED_PLUGINS="daytona" +RUN set -eu; \ + for name in $CLOUD_BUNDLED_PLUGINS; do \ + dir="packages/plugins/sandbox-providers/$name"; \ + test -d "$dir" || { echo "ERROR: unknown sandbox provider '$name'" >&2; exit 1; }; \ + pnpm -C "$dir" install --ignore-workspace --no-lockfile; \ + pnpm -C "$dir" build; \ + test -f "$dir/dist/manifest.js" || { echo "ERROR: $dir is missing dist/manifest.js after build" >&2; exit 1; }; \ + done + +FROM production AS cloud +COPY --chown=node:node --from=cloud-plugins /app/packages/plugins/sandbox-providers /app/packages/plugins/sandbox-providers diff --git a/server/src/__tests__/cloud-image-bundled-plugins.test.ts b/server/src/__tests__/cloud-image-bundled-plugins.test.ts new file mode 100644 index 0000000000..ad944fd0cd --- /dev/null +++ b/server/src/__tests__/cloud-image-bundled-plugins.test.ts @@ -0,0 +1,78 @@ +import { existsSync, readFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import { BUNDLED_PLUGIN_CATALOG } from "../services/bundled-plugins.js"; + +/** + * Drift guard for the cloud image variant (Dockerfile `cloud` target). + * + * The cloud image builds the sandbox-provider plugins named in the + * CLOUD_BUNDLED_PLUGINS build arg so managed instances can auto-install + * them from the bundled catalog at boot. That contract spans three places + * that nothing else ties together: the Dockerfile ARG default, the docker + * workflow's build-arg, and BUNDLED_PLUGIN_CATALOG. A rename or removal in + * any one of them would otherwise surface only when the image build fails + * on master — or worse, as a silent "bundle not present" skip at instance + * boot. + */ + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", ".."); +const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8"); +const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker.yml"), "utf8"); + +function parseList(source: string, pattern: RegExp, label: string): string[] { + const match = source.match(pattern); + expect(match, `${label} must declare CLOUD_BUNDLED_PLUGINS`).toBeTruthy(); + const names = (match?.[1] ?? "").trim().split(/\s+/).filter(Boolean); + expect(names.length, `${label} CLOUD_BUNDLED_PLUGINS must not be empty`).toBeGreaterThan(0); + return names; +} + +const dockerfileDefault = parseList( + dockerfile, + /^ARG CLOUD_BUNDLED_PLUGINS="([^"]*)"/m, + "Dockerfile", +); +const workflowArg = parseList( + workflow, + /^\s*CLOUD_BUNDLED_PLUGINS=(.*)$/m, + "docker workflow", +); + +describe("cloud image bundled plugins", () => { + it("keeps the Dockerfile default and the workflow build-arg in sync", () => { + expect(workflowArg).toEqual(dockerfileDefault); + }); + + it.each([...new Set([...dockerfileDefault, ...workflowArg])])( + "plugin %s is buildable and resolvable by the auto-installer", + (name) => { + const dir = path.join(repoRoot, "packages", "plugins", "sandbox-providers", name); + expect(existsSync(dir), `${dir} must exist`).toBe(true); + expect( + existsSync(path.join(dir, "src", "manifest.ts")), + `${name} must have src/manifest.ts so the build produces dist/manifest.js`, + ).toBe(true); + const packageJson = JSON.parse(readFileSync(path.join(dir, "package.json"), "utf8")) as { + scripts?: Record; + }; + expect(packageJson.scripts?.build, `${name} must have a build script`).toBeTruthy(); + + // The auto-installer resolves catalog keys to relative paths; a plugin + // baked into the image but absent from the catalog (or vice versa) + // can never be auto-installed. + const catalogEntry = BUNDLED_PLUGIN_CATALOG.find( + (entry) => entry.relativePath === `sandbox-providers/${name}`, + ); + expect(catalogEntry, `${name} must be listed in BUNDLED_PLUGIN_CATALOG`).toBeTruthy(); + }, + ); + + it("pins the default image build to the production target", () => { + // The Dockerfile's final stage is `cloud`; without an explicit target + // the workflow's main build would silently publish the cloud variant + // to the self-hosted tags. + expect(workflow).toMatch(/^\s*target: production$/m); + }); +});