feat(docker): publish a cloud image variant with built bundled plugins (#10157)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Managed (cloud-hosted) deployments configure instances through `PAPERCLIP_MANAGED_CONFIG`, including a `plugins.autoInstall` key list that the boot-time installer resolves against the bundled plugin catalog > - The installer requires each bundled plugin's `dist/manifest.js` (`server/src/services/bundled-plugins.ts`), but the published image only ships the sandbox providers' *source* — they are intentionally excluded from the pnpm workspace, and the Dockerfile never builds them > - Every managed auto-install therefore logs `bundled plugin bundle not present; skipping auto-install` and no sandbox provider can be provisioned through managed config > - Baking built plugins into the single published image would fix it but makes every self-hosted pull carry the providers' `node_modules` for a managed-only mechanism > - This pull request adds a `cloud` Dockerfile target extending `production` with built bundled plugins — parameterized by build arg and currently just `daytona` — published alongside the default image with a `-cloud` tag suffix > - The benefit is working plugin auto-provisioning for managed deployments while the self-hosted image stays byte-identical and the cloud variant only carries what is actually deployed ## Linked Issues or Issue Description Fixes #10158 (filed for this problem; no prior issue existed — searched for duplicate/related PRs and issues around bundled plugins, docker image variants, and auto-install). Summary: **What happened:** on a managed instance with `plugins.autoInstall: ["daytona"]` delivered via `PAPERCLIP_MANAGED_CONFIG`, boot logs `bundled plugin bundle not present; skipping auto-install` with `pluginPath: /app/packages/plugins/sandbox-providers/daytona`, and the plugin is never installed. **Expected:** the advertised bundled-catalog keys are installable from the published image. **Why:** the image ships plugin source without `dist/` — nothing in the Dockerfile builds the workspace-excluded sandbox providers. ## What Changed - `Dockerfile`: new `cloud-plugins` stage (based on `build`, so devDependencies are available for `tsc`) that installs and builds each provider named in the `CLOUD_BUNDLED_PLUGINS` build arg standalone (`pnpm install --ignore-workspace --no-lockfile && pnpm build`, exactly as the providers' READMEs prescribe), asserting `dist/manifest.js` exists per plugin and failing loudly on unknown names; new `cloud` stage = `production` + the built plugin tree. The arg defaults to `daytona` — the only provider managed deployments auto-install today; every entry adds its `node_modules` to the image, so the list grows only with actual need (a one-line workflow change). - `.github/workflows/docker.yml`: the existing build step is pinned to `target: production` (without this, the new trailing stage would silently become the default build target — this pin is what keeps the self-hosted image identical); new metadata + build-push steps publish the `cloud` target (with `CLOUD_BUNDLED_PLUGINS=daytona`) under the same tag set with a `-cloud` suffix (`sha-<short>-cloud`, `latest-cloud`, `<version>-cloud`), same schema labels, reusing the GHA layer cache ## Verification - All seven sandbox providers build standalone from a clean checkout with the exact commands the new stage runs, each producing `dist/manifest.js` — so the current `daytona` default works and future list additions are known-good - The stage's shell loop was dry-run against the checkout (directory existence + per-plugin assertion logic) - Workflow YAML lints clean - **Not run:** a full multi-arch `docker build` (no local docker daemon). The `cloud` stage is additive and the default target is pinned, so the risk is contained to the new build step; the first master build after merge proves it end-to-end ## Risks - Self-hosted behavior: unchanged. The default image build is pinned to the `production` target, which produces the same layers as before this change; the `cloud` stages run only for the new build step. - The plugin installs in the `cloud-plugins` stage use `--no-lockfile` (the providers are workspace-excluded and lockfile-less by design), so plugin dependency resolution is not pinned at image-build time. This mirrors the existing Plugins-page install path, which resolves from npm at install time. - CI cost: one additional build-push per master push. It reuses the layer cache from the production build, so the marginal work is the single plugin's build layers. - An unknown name in `CLOUD_BUNDLED_PLUGINS`, or a provider that stops producing `dist/manifest.js`, fails the cloud build loudly rather than publishing a broken variant. ## Model Used Claude (Anthropic), model ID `claude-fable-5[1m]` via Claude Code CLI — extended thinking and tool use (code edits, standalone plugin build verification, workflow lint). ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] Self-hosted behavior unchanged (default build target pinned to `production`) - [x] One clear change: publish a cloud image variant with built bundled plugins
This commit is contained in:
parent
ee3ed0117e
commit
965a827ee7
|
|
@ -120,9 +120,50 @@ jobs:
|
|||
uses: docker/build-push-action@v7
|
||||
with:
|
||||
context: .
|
||||
# Pin the self-hosted image to the production stage explicitly:
|
||||
# the Dockerfile now declares a later `cloud` stage, and without a
|
||||
# target the default would silently become that stage.
|
||||
target: production
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
# The cloud variant carries built bundled plugins for managed
|
||||
# deployments (see the `cloud` stage in the Dockerfile). Published
|
||||
# under the same tag set with a `-cloud` suffix (sha-<short>-cloud,
|
||||
# latest-cloud, <version>-cloud). Reuses the layer cache from the
|
||||
# production build, so this mostly adds the plugin-build layers.
|
||||
- name: Docker meta (cloud)
|
||||
id: meta-cloud
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: ghcr.io/${{ github.repository }}
|
||||
flavor: |
|
||||
suffix=-cloud,onlatest=true
|
||||
tags: |
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}}
|
||||
type=sha
|
||||
labels: |
|
||||
io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }}
|
||||
io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }}
|
||||
|
||||
- name: Build and push (cloud)
|
||||
uses: docker/build-push-action@v7
|
||||
with:
|
||||
context: .
|
||||
target: cloud
|
||||
# Space-separated sandbox-provider directory names to build into
|
||||
# the variant; add here when managed deployments need another.
|
||||
build-args: |
|
||||
CLOUD_BUNDLED_PLUGINS=daytona
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
tags: ${{ steps.meta-cloud.outputs.tags }}
|
||||
labels: ${{ steps.meta-cloud.outputs.labels }}
|
||||
|
|
|
|||
35
Dockerfile
35
Dockerfile
|
|
@ -90,3 +90,38 @@ EXPOSE 3100
|
|||
|
||||
ENTRYPOINT ["docker-entrypoint.sh"]
|
||||
CMD ["node", "--import", "./server/node_modules/tsx/dist/loader.mjs", "server/dist/index.js"]
|
||||
|
||||
# Cloud image variant (build with `--target cloud`): the production image
|
||||
# plus built bundled sandbox-provider plugins. Managed instances receive a
|
||||
# `plugins.autoInstall` key list through PAPERCLIP_MANAGED_CONFIG and
|
||||
# install those plugins from the bundled catalog at boot
|
||||
# (server/src/services/bundled-plugins.ts), which requires each plugin's
|
||||
# dist/ to exist in the image — the default image ships only their source,
|
||||
# so auto-install logs "bundle not present" and skips. The plugins are
|
||||
# built in this separate target so the default (self-hosted) image stays
|
||||
# lean; CI pins the default build to `--target production`, which is
|
||||
# byte-identical to before this stage existed.
|
||||
#
|
||||
# The sandbox providers are intentionally excluded from the pnpm workspace
|
||||
# (see pnpm-workspace.yaml), so each installs standalone exactly as its
|
||||
# README prescribes. Installing in a `build`-based stage (not `production`)
|
||||
# keeps devDependencies available for tsc: `production` sets
|
||||
# NODE_ENV=production, which would make pnpm skip them.
|
||||
#
|
||||
# CLOUD_BUNDLED_PLUGINS is the space-separated list of sandbox-provider
|
||||
# directory names to build into the variant. Only what managed deployments
|
||||
# actually auto-install belongs here — every entry adds its node_modules
|
||||
# to the image. Growing the list is a one-line workflow change.
|
||||
FROM build AS cloud-plugins
|
||||
ARG CLOUD_BUNDLED_PLUGINS="daytona"
|
||||
RUN set -eu; \
|
||||
for name in $CLOUD_BUNDLED_PLUGINS; do \
|
||||
dir="packages/plugins/sandbox-providers/$name"; \
|
||||
test -d "$dir" || { echo "ERROR: unknown sandbox provider '$name'" >&2; exit 1; }; \
|
||||
pnpm -C "$dir" install --ignore-workspace --no-lockfile; \
|
||||
pnpm -C "$dir" build; \
|
||||
test -f "$dir/dist/manifest.js" || { echo "ERROR: $dir is missing dist/manifest.js after build" >&2; exit 1; }; \
|
||||
done
|
||||
|
||||
FROM production AS cloud
|
||||
COPY --chown=node:node --from=cloud-plugins /app/packages/plugins/sandbox-providers /app/packages/plugins/sandbox-providers
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
import { existsSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { BUNDLED_PLUGIN_CATALOG } from "../services/bundled-plugins.js";
|
||||
|
||||
/**
|
||||
* Drift guard for the cloud image variant (Dockerfile `cloud` target).
|
||||
*
|
||||
* The cloud image builds the sandbox-provider plugins named in the
|
||||
* CLOUD_BUNDLED_PLUGINS build arg so managed instances can auto-install
|
||||
* them from the bundled catalog at boot. That contract spans three places
|
||||
* that nothing else ties together: the Dockerfile ARG default, the docker
|
||||
* workflow's build-arg, and BUNDLED_PLUGIN_CATALOG. A rename or removal in
|
||||
* any one of them would otherwise surface only when the image build fails
|
||||
* on master — or worse, as a silent "bundle not present" skip at instance
|
||||
* boot.
|
||||
*/
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
|
||||
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
|
||||
const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker.yml"), "utf8");
|
||||
|
||||
function parseList(source: string, pattern: RegExp, label: string): string[] {
|
||||
const match = source.match(pattern);
|
||||
expect(match, `${label} must declare CLOUD_BUNDLED_PLUGINS`).toBeTruthy();
|
||||
const names = (match?.[1] ?? "").trim().split(/\s+/).filter(Boolean);
|
||||
expect(names.length, `${label} CLOUD_BUNDLED_PLUGINS must not be empty`).toBeGreaterThan(0);
|
||||
return names;
|
||||
}
|
||||
|
||||
const dockerfileDefault = parseList(
|
||||
dockerfile,
|
||||
/^ARG CLOUD_BUNDLED_PLUGINS="([^"]*)"/m,
|
||||
"Dockerfile",
|
||||
);
|
||||
const workflowArg = parseList(
|
||||
workflow,
|
||||
/^\s*CLOUD_BUNDLED_PLUGINS=(.*)$/m,
|
||||
"docker workflow",
|
||||
);
|
||||
|
||||
describe("cloud image bundled plugins", () => {
|
||||
it("keeps the Dockerfile default and the workflow build-arg in sync", () => {
|
||||
expect(workflowArg).toEqual(dockerfileDefault);
|
||||
});
|
||||
|
||||
it.each([...new Set([...dockerfileDefault, ...workflowArg])])(
|
||||
"plugin %s is buildable and resolvable by the auto-installer",
|
||||
(name) => {
|
||||
const dir = path.join(repoRoot, "packages", "plugins", "sandbox-providers", name);
|
||||
expect(existsSync(dir), `${dir} must exist`).toBe(true);
|
||||
expect(
|
||||
existsSync(path.join(dir, "src", "manifest.ts")),
|
||||
`${name} must have src/manifest.ts so the build produces dist/manifest.js`,
|
||||
).toBe(true);
|
||||
const packageJson = JSON.parse(readFileSync(path.join(dir, "package.json"), "utf8")) as {
|
||||
scripts?: Record<string, string>;
|
||||
};
|
||||
expect(packageJson.scripts?.build, `${name} must have a build script`).toBeTruthy();
|
||||
|
||||
// The auto-installer resolves catalog keys to relative paths; a plugin
|
||||
// baked into the image but absent from the catalog (or vice versa)
|
||||
// can never be auto-installed.
|
||||
const catalogEntry = BUNDLED_PLUGIN_CATALOG.find(
|
||||
(entry) => entry.relativePath === `sandbox-providers/${name}`,
|
||||
);
|
||||
expect(catalogEntry, `${name} must be listed in BUNDLED_PLUGIN_CATALOG`).toBeTruthy();
|
||||
},
|
||||
);
|
||||
|
||||
it("pins the default image build to the production target", () => {
|
||||
// The Dockerfile's final stage is `cloud`; without an explicit target
|
||||
// the workflow's main build would silently publish the cloud variant
|
||||
// to the self-hosted tags.
|
||||
expect(workflow).toMatch(/^\s*target: production$/m);
|
||||
});
|
||||
});
|
||||
Loading…
Reference in New Issue