diff --git a/doc/plugins/PLUGIN_SPEC.md b/doc/plugins/PLUGIN_SPEC.md index 9b0bcc0d25..45fba6be86 100644 --- a/doc/plugins/PLUGIN_SPEC.md +++ b/doc/plugins/PLUGIN_SPEC.md @@ -395,6 +395,11 @@ Rules: - `projects` → `projects.managed` - `routines` → `routines.managed` - `skills` → `skills.managed` +- an `environmentDrivers` entry with `kind: "sandbox_provider"` declares sandbox + capabilities through `sandboxCapabilities`. See the + [sandbox provider capability contract](./SANDBOX_PROVIDER_CAPABILITIES.md) for + the supported keys, the worker-method prerequisites, and the narrowing and + failure rules. ## 11. Agent Tools diff --git a/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md b/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md new file mode 100644 index 0000000000..c3a0071c80 --- /dev/null +++ b/doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md @@ -0,0 +1,160 @@ +# Sandbox provider capability contract + +A sandbox provider plugin declares an environment driver with +`kind: "sandbox_provider"`. Each driver can declare a set of optional sandbox +capabilities. This document is the contract for a third-party provider author. +It states what to declare, which worker methods each capability needs, what an +omitted key means, and when the host narrows or denies a capability. + +Read [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md) for the +native file-transfer hooks. Read the driver declaration shape in +[the plugin specification](./PLUGIN_SPEC.md). + +## How the host resolves an effective capability + +The host never trusts a declaration alone. For every run it resolves each +capability as the intersection of three inputs: + +``` +effective = verified ∩ declared ∩ narrowing +``` + +- **verified** — the methods the live worker advertised in + `InitializeResult.supportedMethods`. The host maps each capability to the + worker methods it needs (see the table below). A capability is verified only + when the worker advertises every required method. An empty or missing method + list verifies nothing, so every capability resolves `false`. +- **declared** — the values in the driver declaration. A declared key is + optional (see the next section). +- **narrowing** — a per-run restriction from the lease policy or the resolved + provider config. A narrowing can only remove a capability, never add one. + +A capability is effective only when all three allow it. A declaration therefore +never grants a capability that the worker did not verify. + +## The declaration is optional and partial + +Declare capabilities through the nested `sandboxCapabilities` object on the +driver declaration: + +```ts +environmentDrivers: [ + { + driverKey: "my-provider", + kind: "sandbox_provider", + displayName: "My Provider", + configSchema: { type: "object", properties: {} }, + sandboxCapabilities: { + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: true, + persistentProcessSessions: false, + independentControlCommands: false, + }, + }, +] +``` + +Every key is optional. For a valid, identified provider each key has one of three +states: + +- **Omitted** — the host defers to verified worker discovery. The capability is + effective when the worker advertises the required methods and no narrowing + removes it. Omission is the correct default for a provider that follows the + standard method contract. `reusableLeases` is the one exception: an omitted + `reusableLeases` key never grants reusable leases (see the next section). +- **`false`** — the host narrows the capability to off. The capability is never + effective, even when the worker advertises the required methods. +- **`true`** — the host still requires the verified prerequisites. A `true` + value never grants a capability without them. It documents intent and lets the + host present the capability, but the worker must still advertise the required + methods. + +## Reusable leases need an explicit opt-in + +Reusable leases are the exception to the omission rule above. The host grants +reusable-lease acquisition only when the declaration sets `reusableLeases` to +`true`. The provider opts in through one of two fields: + +- the nested `sandboxCapabilities.reusableLeases: true`, or +- the legacy `supportsReusableLeases: true`. + +An omitted key leaves `reusableLeases` unset. An unset key does not make the +provider eligible for reusable-lease acquisition, and it does not advertise +provider-level reusable support. The host then always creates an ephemeral lease. + +The opt-in never removes the other prerequisites. The worker must still verify +all three lifecycle methods, `environmentResumeLease`, `environmentReleaseLease`, +and `environmentDestroyLease`, and per-run narrowing still applies. + +The two opt-in fields have a fixed precedence. The host keeps the legacy +`supportsReusableLeases` field for backward compatibility, and it folds the field +into `sandboxCapabilities.reusableLeases`. + +- When only `supportsReusableLeases` is present, the host reads it as + `reusableLeases`. +- When both `supportsReusableLeases` and `sandboxCapabilities.reusableLeases` are + present, the nested value wins. + +A manifest with legacy `true` and nested `false` therefore resolves to `false`. +Prefer the nested `sandboxCapabilities.reusableLeases` in a new manifest. + +## The capabilities and their worker-method prerequisites + +| Capability | Required worker methods | Meaning | +| --- | --- | --- | +| `reusableLeases` | `environmentResumeLease`, `environmentReleaseLease`, **and** `environmentDestroyLease` | The host retains a provider lease and resumes it across runs. | +| `nativeSyncIn` | `environmentSyncIn` | The host transfers files into the sandbox through the native inbound hook. | +| `nativeSyncOut` | `environmentSyncOut` | The host transfers files out of the sandbox through the native outbound hook. | +| `persistentProcessSessions` | `environmentExecute` | The provider keeps a persistent process session open across commands. | +| `independentControlCommands` | `environmentExecute` | The provider runs a one-shot control command beside a long-lived command. | + +Reusable leases need all three lifecycle methods. The host resumes a lease with +`environmentResumeLease`, ends it with `environmentReleaseLease`, and tears down a +stale lease with `environmentDestroyLease`. The reuse path destroys a stale lease +when a resume fails, so a provider that cannot destroy a lease would strand it. A +worker that omits any of the three methods never gets reusable leases, even with a +positive declaration. The host then always creates an ephemeral lease. + +The host advertises and consumes the two native sync methods as a pair. Define +both or neither. See [Sandbox file-sync lifecycle hooks](./SANDBOX_FILE_SYNC_HOOKS.md). + +## Target and config narrowing + +A narrowing removes a capability that the provider verified and declared but that +this run cannot use. + +- **Ephemeral lease policy.** A lease that the host does not retain never reuses. + The host narrows `reusableLeases` to off for an ephemeral lease and keeps it on + only for a reuse-by-environment lease. +- **Kubernetes Job backend.** A Job-backed lease, or a lease the host marked as + unable to run native file sync, disables native sync. The host narrows + `nativeSyncIn` and `nativeSyncOut` to off and keeps the base64-over-exec + fallback. +- **`useSessions` provider config.** A session-based provider follows its + `useSessions` config value for `persistentProcessSessions`. Sessions default to + off. A config that omits the key adds no narrowing. + +## Failure behavior + +The host fails closed on two failure states. It never grants a capability from an +unknown state. + +- **Config-resolution failure.** When the host cannot resolve the provider + config, it cannot read `useSessions`. It narrows `persistentProcessSessions` to + off instead of allowing it through an empty config. +- **Exact-plugin identity failure.** A retained lease pins the exact plugin that + acquired it. When that plugin is absent, or when it no longer declares this + provider key with the `sandbox_provider` kind, the host cannot establish the + declaration. It resolves every effective capability to `false`, no matter what + methods a stale worker still advertises. An omitted `sandboxCapabilities` + object on a valid, identified plugin is a different state; the host defers to + verified discovery for that case. + +## Concurrency capabilities are not part of the contract + +Earlier drafts listed two concurrency keys, `concurrentSyncAndExec` and +`concurrentSyncOperations`. The runtime never exposed a scheduling choice that +read either key, so a declaration had no effect. The host removed both keys. The +strict capability validator now rejects them as unknown keys. The host can +reintroduce a concurrency capability when a runtime path enforces it. diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index fab8ce9227..9d53709f39 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -89,10 +89,31 @@ export interface AdapterSshExecutionTarget extends AdapterExecutionTargetWorkspa spec: SshRemoteExecutionSpec; } +/** + * Read-only snapshot of the effective sandbox capabilities for one execution + * target. Each flag is the resolved result of the provider's declaration, the + * live worker's verified methods, and any narrowing from the config or lease. + * The host computes it once and attaches it to the target; a consumer reads it + * but never changes it, so every field is `readonly`. + */ +export interface EffectiveSandboxCapabilities { + readonly reusableLeases: boolean; + readonly nativeSyncIn: boolean; + readonly nativeSyncOut: boolean; + readonly persistentProcessSessions: boolean; + readonly independentControlCommands: boolean; +} + export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWorkspaceMetadata { kind: "remote"; transport: "sandbox"; providerKey?: string | null; + /** + * Read-only effective capability snapshot for this sandbox target. The host + * resolves it from the provider declaration ∩ the verified worker methods ∩ + * narrowing, then attaches it here. Absent when no snapshot was resolved. + */ + readonly effectiveCapabilities?: EffectiveSandboxCapabilities | null; shellCommand?: "bash" | "sh" | null; environmentId?: string | null; leaseId?: string | null; @@ -213,6 +234,22 @@ function readString(value: unknown): string | null { return typeof value === "string" && value.trim().length > 0 ? value.trim() : null; } +// Read a serialized effective-capability snapshot back into a full record. A +// missing or non-boolean field reads as `false`, so a round-tripped target +// never grants a capability that the snapshot did not carry. Returns null when +// there is no object to read. +function parseEffectiveSandboxCapabilities(value: unknown): EffectiveSandboxCapabilities | null { + const parsed = parseObject(value); + if (Object.keys(parsed).length === 0) return null; + return { + reusableLeases: parsed.reusableLeases === true, + nativeSyncIn: parsed.nativeSyncIn === true, + nativeSyncOut: parsed.nativeSyncOut === true, + persistentProcessSessions: parsed.persistentProcessSessions === true, + independentControlCommands: parsed.independentControlCommands === true, + }; +} + function readStringMeta(parsed: Record, key: string): string | null { return readString(parsed[key]); } @@ -1085,6 +1122,7 @@ export function parseAdapterExecutionTarget(value: unknown): AdapterExecutionTar if (kind === "remote" && readStringMeta(parsed, "transport") === "sandbox") { const remoteCwd = readStringMeta(parsed, "remoteCwd"); if (!remoteCwd) return null; + const effectiveCapabilities = parseEffectiveSandboxCapabilities(parsed.effectiveCapabilities); return { kind: "remote", transport: "sandbox", @@ -1094,6 +1132,7 @@ export function parseAdapterExecutionTarget(value: unknown): AdapterExecutionTar remoteCwd, timeoutMs: typeof parsed.timeoutMs === "number" ? parsed.timeoutMs : null, streamRunLogs: typeof parsed.streamRunLogs === "boolean" ? parsed.streamRunLogs : null, + ...(effectiveCapabilities ? { effectiveCapabilities } : {}), }; } diff --git a/packages/shared/src/environment-support.ts b/packages/shared/src/environment-support.ts index f19fb95355..2f0c808470 100644 --- a/packages/shared/src/environment-support.ts +++ b/packages/shared/src/environment-support.ts @@ -1,6 +1,33 @@ import type { AgentAdapterType, EnvironmentDriver } from "./constants.js"; import type { SandboxEnvironmentProvider } from "./types/environment.js"; -import type { JsonSchema, PluginEnvironmentTemplateConfigBinding } from "./types/plugin.js"; +import type { + JsonSchema, + PluginEnvironmentTemplateConfigBinding, + SandboxProviderCapabilities, +} from "./types/plugin.js"; + +/** + * Resolve the DECLARED sandbox capabilities of a provider driver, with the + * legacy `supportsReusableLeases` flag folded in for compatibility. + * + * The result is a partial: a key is present only when the driver declared it, + * so a caller can tell a declared `false` from an absent key. The nested + * `sandboxCapabilities.reusableLeases` wins over the legacy flag when both are + * present. This is the DECLARATION only; the runtime still intersects it with + * the verified worker methods and any narrowing before it grants a capability. + */ +export function resolveDeclaredSandboxCapabilities( + driver: { + supportsReusableLeases?: boolean; + sandboxCapabilities?: SandboxProviderCapabilities; + }, +): SandboxProviderCapabilities { + const declared: SandboxProviderCapabilities = { ...(driver.sandboxCapabilities ?? {}) }; + if (declared.reusableLeases === undefined && driver.supportsReusableLeases !== undefined) { + declared.reusableLeases = driver.supportsReusableLeases; + } + return declared; +} export type EnvironmentSupportStatus = "supported" | "unsupported"; diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index da182e227b..51f8032ebf 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1381,6 +1381,7 @@ export type { PluginWebhookDeclaration, PluginToolDeclaration, PluginEnvironmentDriverDeclaration, + SandboxProviderCapabilities, PluginEnvironmentTemplateConfigBinding, PluginManagedAgentDeclaration, PluginManagedProjectDeclaration, @@ -2216,6 +2217,7 @@ export { pluginWebhookDeclarationSchema, pluginToolDeclarationSchema, pluginEnvironmentDriverDeclarationSchema, + sandboxProviderCapabilitiesSchema, pluginUiSlotDeclarationSchema, pluginLauncherActionDeclarationSchema, pluginLauncherRenderDeclarationSchema, @@ -2235,6 +2237,7 @@ export { type PluginWebhookDeclarationInput, type PluginToolDeclarationInput, type PluginEnvironmentDriverDeclarationInput, + type SandboxProviderCapabilitiesInput, type PluginUiSlotDeclarationInput, type PluginLauncherActionDeclarationInput, type PluginLauncherRenderDeclarationInput, @@ -2343,6 +2346,7 @@ export { getAdapterEnvironmentSupport, isEnvironmentDriverSupportedForAdapter, isSandboxProviderSupportedForAdapter, + resolveDeclaredSandboxCapabilities, supportedEnvironmentDriversForAdapter, supportedSandboxProvidersForAdapter, } from "./environment-support.js"; diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 1f4971b940..82f1da7400 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -935,6 +935,7 @@ export type { PluginWebhookDeclaration, PluginToolDeclaration, PluginEnvironmentDriverDeclaration, + SandboxProviderCapabilities, PluginEnvironmentTemplateConfigBinding, PluginManagedAgentDeclaration, PluginManagedProjectDeclaration, diff --git a/packages/shared/src/types/plugin.ts b/packages/shared/src/types/plugin.ts index 2ff9334382..7a8e9081a2 100644 --- a/packages/shared/src/types/plugin.ts +++ b/packages/shared/src/types/plugin.ts @@ -132,6 +132,29 @@ export interface PluginEnvironmentTemplateConfigBinding { unsetFields?: string[]; } +/** + * Optional capability declaration for a sandbox provider driver. + * + * Each flag states that the provider intends to support one behavior. The + * declaration is a request, not a grant: the host resolves the effective + * capability as the intersection of the declaration, the live worker's verified + * methods, and any narrowing from the provider config or lease. A declared flag + * never grants a capability the live worker did not verify. Every flag is + * optional; an absent flag defers to the verified discovery baseline. + */ +export interface SandboxProviderCapabilities { + /** Provider can retain and resume a provider lease across runs. */ + reusableLeases?: boolean; + /** Provider can transfer files into the sandbox through a native inbound hook. */ + nativeSyncIn?: boolean; + /** Provider can transfer files out of the sandbox through a native outbound hook. */ + nativeSyncOut?: boolean; + /** Provider can keep a persistent process session open across commands. */ + persistentProcessSessions?: boolean; + /** Provider can run a control command that does not wait for the main command. */ + independentControlCommands?: boolean; +} + export interface PluginEnvironmentDriverDeclaration { /** Stable driver key, unique within the plugin. Namespaced by plugin ID at runtime. */ driverKey: string; @@ -153,6 +176,13 @@ export interface PluginEnvironmentDriverDeclaration { * behavior even if their config schema exposes a reuse-like setting. */ supportsReusableLeases?: boolean; + /** + * Fine-grained sandbox capability declaration. Optional and partial. The host + * resolves the effective capability as declaration ∩ verified ∩ narrowing; + * see {@link SandboxProviderCapabilities}. When both `supportsReusableLeases` + * and `sandboxCapabilities.reusableLeases` are present, the nested value wins. + */ + sandboxCapabilities?: SandboxProviderCapabilities; /** Provider can keep a temporary setup sandbox alive for user-driven sandbox customization and capture. */ supportsInteractiveSetup?: boolean; /** Connection types the setup sandbox can expose. Initially `ssh`; providers may add custom values. */ diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index 67b8244990..3fc569eaf3 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -762,6 +762,7 @@ export { pluginWebhookDeclarationSchema, pluginToolDeclarationSchema, pluginEnvironmentDriverDeclarationSchema, + sandboxProviderCapabilitiesSchema, pluginUiSlotDeclarationSchema, pluginLauncherActionDeclarationSchema, pluginLauncherRenderDeclarationSchema, @@ -783,6 +784,7 @@ export { type PluginWebhookDeclarationInput, type PluginToolDeclarationInput, type PluginEnvironmentDriverDeclarationInput, + type SandboxProviderCapabilitiesInput, type PluginUiSlotDeclarationInput, type PluginLauncherActionDeclarationInput, type PluginLauncherRenderDeclarationInput, diff --git a/packages/shared/src/validators/plugin.test.ts b/packages/shared/src/validators/plugin.test.ts index a560ab25e5..95f7b27c8f 100644 --- a/packages/shared/src/validators/plugin.test.ts +++ b/packages/shared/src/validators/plugin.test.ts @@ -1,7 +1,31 @@ import { describe, expect, it } from "vitest"; import { PLUGIN_CAPABILITIES } from "../constants.js"; +import { resolveDeclaredSandboxCapabilities } from "../environment-support.js"; import { pluginManagedRoutineDeclarationSchema, pluginManifestV1Schema, pluginUiSlotDeclarationSchema } from "./plugin.js"; +function buildSandboxProviderManifest(driver: Record) { + return { + id: "paperclip.capability-provider", + apiVersion: 1, + version: "0.1.0", + displayName: "Capability Provider", + description: "Sandbox provider that declares fine-grained capabilities.", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "./dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "capability-provider", + kind: "sandbox_provider", + displayName: "Capability Provider", + configSchema: { type: "object" }, + ...driver, + }, + ], + }; +} + describe("plugin capability constants", () => { it("exposes each capability once", () => { expect(new Set(PLUGIN_CAPABILITIES).size).toBe(PLUGIN_CAPABILITIES.length); @@ -250,3 +274,77 @@ describe("plugin UI slot validators", () => { expect(parsed.error.issues.some((issue) => issue.message.includes("reserved by the host"))).toBe(true); }); }); + +describe("sandbox provider capability declaration validators", () => { + it("test_manifest_accepts_sandbox_capabilities_and_rejects_unknown_capability_keys", () => { + const parsed = pluginManifestV1Schema.parse( + buildSandboxProviderManifest({ + sandboxCapabilities: { + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: false, + persistentProcessSessions: true, + independentControlCommands: false, + }, + }), + ); + + expect(parsed.environmentDrivers?.[0]?.sandboxCapabilities).toEqual({ + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: false, + persistentProcessSessions: true, + independentControlCommands: false, + }); + + const rejected = pluginManifestV1Schema.safeParse( + buildSandboxProviderManifest({ + sandboxCapabilities: { + reusableLeases: true, + // A typo or unknown capability name must fail validation, not drop + // silently. The nested schema is `.strict()`. + nativeSync: true, + }, + }), + ); + + expect(rejected.success).toBe(false); + }); + + it("test_removed_concurrency_capabilities_are_rejected_as_unknown_keys", () => { + // The concurrency flags left the public contract because no runtime path + // enforced them. The strict schema now rejects them, so a manifest cannot + // declare a capability the host does not honor. + for (const key of ["concurrentSyncAndExec", "concurrentSyncOperations"]) { + const rejected = pluginManifestV1Schema.safeParse( + buildSandboxProviderManifest({ + sandboxCapabilities: { [key]: true }, + }), + ); + expect(rejected.success).toBe(false); + } + }); + + it("test_supports_reusable_leases_compat_maps_to_reusable_leases", () => { + const parsed = pluginManifestV1Schema.parse( + buildSandboxProviderManifest({ supportsReusableLeases: true }), + ); + const driver = parsed.environmentDrivers?.[0]; + + expect(driver?.sandboxCapabilities).toBeUndefined(); + expect(resolveDeclaredSandboxCapabilities(driver!).reusableLeases).toBe(true); + }); + + it("test_sandbox_capabilities_reusable_leases_wins_over_compat_field", () => { + const parsed = pluginManifestV1Schema.parse( + buildSandboxProviderManifest({ + supportsReusableLeases: true, + sandboxCapabilities: { reusableLeases: false }, + }), + ); + const driver = parsed.environmentDrivers?.[0]; + + // The nested declaration wins over the legacy compat flag when both exist. + expect(resolveDeclaredSandboxCapabilities(driver!).reusableLeases).toBe(false); + }); +}); diff --git a/packages/shared/src/validators/plugin.ts b/packages/shared/src/validators/plugin.ts index c2671cc6f0..f8b5a937da 100644 --- a/packages/shared/src/validators/plugin.ts +++ b/packages/shared/src/validators/plugin.ts @@ -155,6 +155,20 @@ export const pluginEnvironmentTemplateConfigBindingSchema = z.object({ } }); +// The nested sandbox capability declaration is `.strict()` so an unknown +// capability key is a validation error, not a silently dropped field. The outer +// driver schema below is non-strict and drops unknown top-level keys, so the +// declaration itself is the gate that a typo in a capability name cannot pass. +export const sandboxProviderCapabilitiesSchema = z.object({ + reusableLeases: z.boolean().optional(), + nativeSyncIn: z.boolean().optional(), + nativeSyncOut: z.boolean().optional(), + persistentProcessSessions: z.boolean().optional(), + independentControlCommands: z.boolean().optional(), +}).strict(); + +export type SandboxProviderCapabilitiesInput = z.infer; + export const pluginEnvironmentDriverDeclarationSchema = z.object({ driverKey: z.string().min(1).regex( /^[a-z0-9][a-z0-9._-]*$/, @@ -164,6 +178,7 @@ export const pluginEnvironmentDriverDeclarationSchema = z.object({ displayName: z.string().min(1).max(100), description: z.string().max(500).optional(), supportsReusableLeases: z.boolean().optional(), + sandboxCapabilities: sandboxProviderCapabilitiesSchema.optional(), supportsInteractiveSetup: z.boolean().optional(), interactiveSetupConnectionTypes: z.array(z.string().min(1).max(100)).max(10).optional(), supportsTemplateCapture: z.boolean().optional(), diff --git a/server/src/__tests__/environment-custom-image-routes.test.ts b/server/src/__tests__/environment-custom-image-routes.test.ts index ae20e42526..649055adeb 100644 --- a/server/src/__tests__/environment-custom-image-routes.test.ts +++ b/server/src/__tests__/environment-custom-image-routes.test.ts @@ -85,6 +85,9 @@ vi.mock("../services/environment-probe.js", () => ({ })); vi.mock("../services/plugin-environment-driver.js", () => ({ + // The runtime reads this published constant at import time. Mirror the real + // value so the mocked module keeps the same reusable-lease method contract. + REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: vi.fn(async () => []), resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null), validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config), diff --git a/server/src/__tests__/environment-execution-target-capabilities.test.ts b/server/src/__tests__/environment-execution-target-capabilities.test.ts new file mode 100644 index 0000000000..deb3ba3c20 --- /dev/null +++ b/server/src/__tests__/environment-execution-target-capabilities.test.ts @@ -0,0 +1,297 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const { mockResolveEnvironmentDriverConfigForRuntime } = vi.hoisted(() => ({ + mockResolveEnvironmentDriverConfigForRuntime: vi.fn(), +})); + +vi.mock("../services/environment-config.js", () => ({ + resolveEnvironmentDriverConfigForRuntime: mockResolveEnvironmentDriverConfigForRuntime, +})); + +import type { EffectiveSandboxCapabilities } from "@paperclipai/adapter-utils/execution-target"; +import { resolveEnvironmentExecutionTarget } from "../services/environment-execution-target.js"; +import type { EnvironmentRuntimeService } from "../services/environment-runtime.js"; + +const SNAPSHOT: EffectiveSandboxCapabilities = { + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: false, + persistentProcessSessions: true, + independentControlCommands: false, +}; + +// A snapshot that grants every capability. A test overrides one flag to prove +// that the removed capability alone changes the runtime decision. +const FULL_GRANT: EffectiveSandboxCapabilities = { + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: true, + persistentProcessSessions: true, + independentControlCommands: true, +}; + +// Build a sandbox execution target with a fixed snapshot and a fixed +// `supportsSync` result. The helper returns the sandbox target so a test reads +// the runner and the streaming flag the snapshot gates. +async function buildSandboxTarget(input: { + snapshot: EffectiveSandboxCapabilities | null; + supportsSync: boolean; + config?: Record; + // Reject the capability resolution to exercise the fail-closed error path. + rejectResolution?: boolean; +}) { + mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({ + driver: "sandbox", + config: { provider: "daytona", timeoutMs: 30_000, ...(input.config ?? {}) }, + }); + + const execute = vi.fn().mockResolvedValue({ + exitCode: 0, + signal: null, + timedOut: false, + stdout: "ok", + stderr: "", + metadata: { durationMs: 600, getDurationMs: 15 }, + }); + const environmentRuntime = { + execute, + supportsSync: () => input.supportsSync, + syncIn: vi.fn(), + syncOut: vi.fn(), + effectiveSandboxCapabilities: vi.fn(async () => { + if (input.rejectResolution) { + throw new Error("capability resolution failed"); + } + return input.snapshot ? Object.freeze({ ...input.snapshot }) : null; + }), + } as unknown as EnvironmentRuntimeService; + + const target = await resolveEnvironmentExecutionTarget({ + db: {} as never, + companyId: "company-1", + adapterType: "codex_local", + environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } }, + leaseId: "lease-1", + leaseMetadata: { remoteCwd: "/work" }, + lease: { id: "lease-1", leasePolicy: "reuse_by_environment" } as never, + environmentRuntime, + }); + + if (target?.kind !== "remote" || target.transport !== "sandbox") { + throw new Error("expected a sandbox target"); + } + return { target, execute }; +} + +describe("resolveEnvironmentExecutionTarget effective capability snapshot", () => { + beforeEach(() => { + mockResolveEnvironmentDriverConfigForRuntime.mockReset(); + }); + + it("test_execution_target_carries_read_only_effective_snapshot", async () => { + mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({ + driver: "sandbox", + config: { provider: "daytona", reuseLease: true, timeoutMs: 30_000 }, + }); + + const effectiveSandboxCapabilities = vi.fn(async () => Object.freeze({ ...SNAPSHOT })); + const environmentRuntime = { + supportsSync: () => false, + effectiveSandboxCapabilities, + } as unknown as EnvironmentRuntimeService; + + const target = await resolveEnvironmentExecutionTarget({ + db: {} as never, + companyId: "company-1", + adapterType: "codex_local", + environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } }, + leaseId: "lease-1", + leaseMetadata: { remoteCwd: "/work" }, + lease: { id: "lease-1", leasePolicy: "reuse_by_environment" } as never, + environmentRuntime, + }); + + expect(target?.kind).toBe("remote"); + if (target?.kind !== "remote" || target.transport !== "sandbox") { + throw new Error("expected a sandbox target"); + } + expect(effectiveSandboxCapabilities).toHaveBeenCalledTimes(1); + expect(target.effectiveCapabilities).toEqual(SNAPSHOT); + + // The snapshot is read-only: it is frozen, so a write does not change it. + expect(Object.isFrozen(target.effectiveCapabilities)).toBe(true); + const snapshot = target.effectiveCapabilities as EffectiveSandboxCapabilities; + try { + (snapshot as { reusableLeases: boolean }).reusableLeases = false; + } catch { + // A strict-mode assignment throws; a non-strict one is a silent no-op. + } + expect(snapshot.reusableLeases).toBe(true); + }); + + it("omits the snapshot when no environment runtime resolves it", async () => { + mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({ + driver: "sandbox", + config: { provider: "daytona", reuseLease: false, timeoutMs: 30_000 }, + }); + + const target = await resolveEnvironmentExecutionTarget({ + db: {} as never, + companyId: "company-1", + adapterType: "codex_local", + environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } }, + leaseId: "lease-1", + leaseMetadata: {}, + lease: null, + environmentRuntime: null, + }); + + expect(target?.kind).toBe("remote"); + if (target?.kind !== "remote" || target.transport !== "sandbox") { + throw new Error("expected a sandbox target"); + } + expect(target.effectiveCapabilities).toBeUndefined(); + }); +}); + +describe("effective snapshot gates the sync decision", () => { + beforeEach(() => { + mockResolveEnvironmentDriverConfigForRuntime.mockReset(); + }); + + it("exposes the native sync hooks when the snapshot grants both sync verbs", async () => { + const { target } = await buildSandboxTarget({ snapshot: FULL_GRANT, supportsSync: true }); + expect(target.runner?.syncIn).toBeTypeOf("function"); + expect(target.runner?.syncOut).toBeTypeOf("function"); + }); + + it("omits the native sync hooks when the snapshot removes a sync verb", async () => { + // The snapshot verified inbound sync but not outbound sync. The runner + // exposes the sync hooks both-or-neither, so it keeps the base64 fallback. + const { target } = await buildSandboxTarget({ + snapshot: { ...FULL_GRANT, nativeSyncOut: false }, + supportsSync: true, + }); + expect(target.runner?.syncIn).toBeUndefined(); + expect(target.runner?.syncOut).toBeUndefined(); + }); + + it("still requires supportsSync even when the snapshot grants native sync", async () => { + const { target } = await buildSandboxTarget({ snapshot: FULL_GRANT, supportsSync: false }); + expect(target.runner?.syncIn).toBeUndefined(); + expect(target.runner?.syncOut).toBeUndefined(); + }); +}); + +describe("effective snapshot gates the session-output-streaming decision", () => { + beforeEach(() => { + mockResolveEnvironmentDriverConfigForRuntime.mockReset(); + }); + + it("keeps session output streaming on when the snapshot grants both session capabilities", async () => { + const { target } = await buildSandboxTarget({ + snapshot: FULL_GRANT, + supportsSync: false, + config: { streamAgentSessionOutput: true }, + }); + expect(target.streamAgentSessionOutput).toBe(true); + }); + + it("drops session output streaming when the snapshot removes persistent process sessions", async () => { + const { target } = await buildSandboxTarget({ + snapshot: { ...FULL_GRANT, persistentProcessSessions: false }, + supportsSync: false, + config: { streamAgentSessionOutput: true }, + }); + expect(target.streamAgentSessionOutput).toBe(false); + }); + + it("drops session output streaming when the snapshot removes independent control commands", async () => { + const { target } = await buildSandboxTarget({ + snapshot: { ...FULL_GRANT, independentControlCommands: false }, + supportsSync: false, + config: { streamAgentSessionOutput: true }, + }); + expect(target.streamAgentSessionOutput).toBe(false); + }); +}); + +describe("effective snapshot gates the persistent-session execution decision", () => { + beforeEach(() => { + mockResolveEnvironmentDriverConfigForRuntime.mockReset(); + }); + + it("forces the persistent session when the snapshot grants persistent process sessions", async () => { + const { target, execute } = await buildSandboxTarget({ snapshot: FULL_GRANT, supportsSync: false }); + await target.runner?.execute({ command: "node", args: ["agent.js"], useSession: true }); + const call = execute.mock.calls[0]![0] as { forceSession?: boolean }; + expect(call.forceSession).toBe(true); + }); + + it("never forces the persistent session when the snapshot removes persistent process sessions", async () => { + // The bridge still asks for the session with `useSession`, but the provider + // cannot keep one, so the seam runs the command one-shot instead. + const { target, execute } = await buildSandboxTarget({ + snapshot: { ...FULL_GRANT, persistentProcessSessions: false }, + supportsSync: false, + }); + await target.runner?.execute({ command: "node", args: ["agent.js"], useSession: true }); + const call = execute.mock.calls[0]![0] as { forceSession?: boolean }; + expect(call.forceSession).toBe(false); + }); +}); + +describe("a rejected capability resolution fails closed for persistent-session behavior", () => { + beforeEach(() => { + mockResolveEnvironmentDriverConfigForRuntime.mockReset(); + }); + + it("omits the snapshot when the resolution rejects", async () => { + // A rejected resolution carries no snapshot; the target never publishes a + // guessed capability set. + const { target } = await buildSandboxTarget({ + snapshot: null, + supportsSync: false, + rejectResolution: true, + }); + expect(target.effectiveCapabilities).toBeUndefined(); + }); + + it("drops session output streaming when the resolution rejects", async () => { + // A rejected resolution must not read as an open grant that enables + // streaming; keep the host output-file poll path. + const { target } = await buildSandboxTarget({ + snapshot: null, + supportsSync: false, + config: { streamAgentSessionOutput: true }, + rejectResolution: true, + }); + expect(target.streamAgentSessionOutput).toBe(false); + }); + + it("never forces the persistent session when the resolution rejects", async () => { + // The bridge asks for the session with `useSession`, but a rejected + // resolution fails closed, so the seam runs the command one-shot instead. + const { target, execute } = await buildSandboxTarget({ + snapshot: null, + supportsSync: false, + rejectResolution: true, + }); + await target.runner?.execute({ command: "node", args: ["agent.js"], useSession: true }); + const call = execute.mock.calls[0]![0] as { forceSession?: boolean }; + expect(call.forceSession).toBe(false); + }); + + it("omits the native sync hooks when the resolution rejects", async () => { + // The worker advertises both sync verbs, but a rejected resolution must not + // read as an open grant. Fail closed and keep the base64 fallback, so an + // unverified provider never gets the native sync path. + const { target } = await buildSandboxTarget({ + snapshot: null, + supportsSync: true, + rejectResolution: true, + }); + expect(target.runner?.syncIn).toBeUndefined(); + expect(target.runner?.syncOut).toBeUndefined(); + }); +}); diff --git a/server/src/__tests__/environment-instance-routes.test.ts b/server/src/__tests__/environment-instance-routes.test.ts index b7dc69725f..28f24be2fb 100644 --- a/server/src/__tests__/environment-instance-routes.test.ts +++ b/server/src/__tests__/environment-instance-routes.test.ts @@ -73,6 +73,9 @@ vi.mock("../services/secrets.js", () => ({ })); vi.mock("../services/plugin-environment-driver.js", () => ({ + // The runtime reads this published constant at import time. Mirror the real + // value so the mocked module keeps the same reusable-lease method contract. + REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: vi.fn(async () => []), resolvePluginSandboxProviderDriverByKey: vi.fn(async () => null), validatePluginEnvironmentDriverConfig: vi.fn(async ({ config }) => config), diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index b718e89ab2..e7d02a16e1 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -109,6 +109,9 @@ vi.mock("../services/execution-workspaces.js", () => ({ })); vi.mock("../services/plugin-environment-driver.js", () => ({ + // The runtime reads this published constant at import time. Mirror the real + // value so the mocked module keeps the same reusable-lease method contract. + REUSABLE_LEASE_WORKER_METHODS: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"], listReadyPluginEnvironmentDrivers: mockListReadyPluginEnvironmentDrivers, resolvePluginSandboxProviderDriverByKey: mockResolvePluginSandboxProviderDriverByKey, startPluginEnvironmentInteractiveSetup: mockStartPluginEnvironmentInteractiveSetup, @@ -1192,6 +1195,87 @@ describe("environment routes", () => { .toBe("supported"); }); + it("publishes reusable leases from the nested capability, not the legacy flag, so the API agrees with acquisition", async () => { + // The manifest sets the legacy flag `true` but the nested override `false`. + // Acquisition lets the nested value win and refuses reuse. The published + // API value must derive from the same declaration resolver and present the + // provider as not reusable, even when the worker verified both lifecycle + // methods. + mockListReadyPluginEnvironmentDrivers.mockResolvedValue([ + { + pluginId: "plugin-1", + pluginKey: "acme.legacy-override-provider", + driverKey: "override-plugin", + displayName: "Override Sandbox", + supportsReusableLeases: true, + sandboxCapabilities: { reusableLeases: false }, + reusableLeaseMethodsVerified: true, + configSchema: { type: "object", properties: {} }, + }, + ]); + const app = createApp({ + type: "board", + userId: "user-1", + source: "local_implicit", + }); + + const res = await request(app).get("/api/companies/company-1/environments/capabilities"); + + expect(res.status).toBe(200); + expect(res.body.sandboxProviders["override-plugin"].supportsReusableLeases).toBe(false); + }); + + it("publishes reusable leases from the legacy flag when the manifest omits the nested override and the worker verified both methods", async () => { + mockListReadyPluginEnvironmentDrivers.mockResolvedValue([ + { + pluginId: "plugin-1", + pluginKey: "acme.legacy-only-provider", + driverKey: "legacy-plugin", + displayName: "Legacy Sandbox", + supportsReusableLeases: true, + reusableLeaseMethodsVerified: true, + configSchema: { type: "object", properties: {} }, + }, + ]); + const app = createApp({ + type: "board", + userId: "user-1", + source: "local_implicit", + }); + + const res = await request(app).get("/api/companies/company-1/environments/capabilities"); + + expect(res.status).toBe(200); + expect(res.body.sandboxProviders["legacy-plugin"].supportsReusableLeases).toBe(true); + }); + + it("does not publish reusable leases when the declaration allows them but the worker omits a lifecycle method", async () => { + // A positive declaration alone is not enough. Acquisition verifies both + // reuse lifecycle methods live and falls back to an ephemeral lease when one + // is missing. The published value must agree and present as not reusable. + mockListReadyPluginEnvironmentDrivers.mockResolvedValue([ + { + pluginId: "plugin-1", + pluginKey: "acme.unverified-reuse-provider", + driverKey: "unverified-plugin", + displayName: "Unverified Reuse Sandbox", + sandboxCapabilities: { reusableLeases: true }, + reusableLeaseMethodsVerified: false, + configSchema: { type: "object", properties: {} }, + }, + ]); + const app = createApp({ + type: "board", + userId: "user-1", + source: "local_implicit", + }); + + const res = await request(app).get("/api/companies/company-1/environments/capabilities"); + + expect(res.status).toBe(200); + expect(res.body.sandboxProviders["unverified-plugin"].supportsReusableLeases).toBe(false); + }); + it("rejects agent list reads for instance-scoped environments", async () => { mockEnvironmentService.list.mockResolvedValue([createEnvironment()]); mockAgentService.getById.mockResolvedValue({ diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index e36439ab55..174bd7b60d 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -28,7 +28,7 @@ import { startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; import { resolveEnvironmentDriverConfigForRuntime } from "../services/environment-config.ts"; -import { environmentRuntimeService, findReusableSandboxLeaseId } from "../services/environment-runtime.ts"; +import { SANDBOX_CAPABILITY_KEYS, environmentRuntimeService, findReusableSandboxLeaseId } from "../services/environment-runtime.ts"; import { environmentService } from "../services/environments.ts"; import { secretService } from "../services/secrets.ts"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts"; @@ -673,6 +673,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -799,6 +800,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -871,6 +873,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -919,6 +922,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -994,6 +998,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -1107,6 +1112,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -1352,6 +1358,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -1555,6 +1562,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager, @@ -1631,6 +1639,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { call: vi.fn(async (_pluginId: string, method: string) => { throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager, @@ -1730,6 +1739,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager, @@ -1818,6 +1828,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -1984,6 +1995,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2025,6 +2037,376 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); }); + it("fails closed and does not resume when a worker restart drops the resume method after the capability snapshot", async () => { + const pluginId = randomUUID(); + const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); + const providerConfig = { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + }; + const environment = { + ...baseEnvironment, + name: "Reusable Plugin Sandbox", + driver: "sandbox", + config: providerConfig, + }; + await environmentService(db).update(environment.id, { + driver: "sandbox", + name: environment.name, + config: providerConfig, + }); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: "acme.fake-sandbox-provider", + packageName: "@acme/fake-sandbox-provider", + version: "1.0.0", + apiVersion: 1, + categories: ["automation"], + manifestJson: { + id: "acme.fake-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Fake Sandbox Provider", + description: "Test schema-driven provider", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + supportsReusableLeases: true, + configSchema: { + type: "object", + properties: { + image: { type: "string" }, + timeoutMs: { type: "number" }, + reuseLease: { type: "boolean" }, + }, + }, + }, + ], + }, + status: "ready", + installOrder: 1, + updatedAt: new Date(), + } as any); + const executionWorkspaceId = randomUUID(); + const projectId = randomUUID(); + await db.insert(projects).values({ + id: projectId, + companyId, + name: `Workspace ${projectId.slice(0, 8)}`, + status: "active", + createdAt: new Date(), + updatedAt: new Date(), + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + mode: "shared_workspace", + strategyType: "project_primary", + name: "Reusable workspace", + status: "active", + providerType: "local_fs", + createdAt: new Date(), + updatedAt: new Date(), + }); + const staleLease = await environmentService(db).acquireLease({ + companyId, + environmentId: environment.id, + executionWorkspaceId, + heartbeatRunId: runId, + leasePolicy: "reuse_by_environment", + provider: "fake-plugin", + providerLeaseId: "stale-plugin-lease", + metadata: { + agentId, + driver: "sandbox", + pluginId, + pluginKey: "acme.fake-sandbox-provider", + sandboxProviderPlugin: true, + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + reusableSandboxLease: { + version: 1, + companyId, + environmentId: environment.id, + executionWorkspaceId, + agentId, + adapterType: null, + provider: "fake-plugin", + runtimeFingerprint: reusableRuntimeFingerprint({ + provider: "fake-plugin", + adapterType: null, + config: providerConfig, + }), + }, + }, + }); + + // The runtime reads the worker methods once to decide reuse, then does + // asynchronous database work before the resume dispatch. A worker restart + // in that window drops `environmentResumeLease`. The first read returns the + // reuse verbs, so the runtime treats the lease as resumable. Every later + // read returns the restarted worker's methods, which no longer include + // `environmentResumeLease`. + let getWorkerReads = 0; + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentResumeLease") { + // A regression that reads the stale snapshot dispatches the resume + // RPC and fails here. The live worker cannot serve the method. + throw new Error("worker no longer advertises environmentResumeLease"); + } + if (method === "environmentDestroyLease") { + return undefined; + } + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "fresh-plugin-lease", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + remoteCwd: "/workspace", + }, + }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + getWorker: vi.fn(() => { + getWorkerReads += 1; + return { + supportedMethods: + getWorkerReads === 1 + ? ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] + : ["environmentReleaseLease", "environmentDestroyLease"], + }; + }), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + agentId, + heartbeatRunId: runId, + persistedExecutionWorkspace: { + id: executionWorkspaceId, + mode: "shared_workspace", + }, + }); + + // The runtime re-checks the live worker before the resume dispatch. The + // restarted worker no longer advertises `environmentResumeLease`, so the + // runtime must not dispatch the resume RPC. + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentResumeLease", + expect.anything(), + expect.anything(), + ); + // It destroys the stale reusable lease and acquires a fresh one. + expect(workerManager.call).toHaveBeenCalledWith( + pluginId, + "environmentDestroyLease", + expect.objectContaining({ driverKey: "fake-plugin", providerLeaseId: "stale-plugin-lease" }), + 31234, + ); + expect(workerManager.call).toHaveBeenCalledWith( + pluginId, + "environmentAcquireLease", + expect.objectContaining({ driverKey: "fake-plugin", agentId, executionWorkspaceId, runId }), + 31234, + ); + expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease"); + await expect(environmentService(db).getLeaseById(staleLease.id)).resolves.toMatchObject({ + status: "expired", + cleanupStatus: "success", + }); + }); + + // Seed a reusable plugin sandbox lease that a worker created under an earlier + // capability set. The worker restarts and no longer advertises the lifecycle + // methods. The lease lifecycle paths must verify the live worker before they + // dispatch a lifecycle RPC, so the runtime fails closed instead of a doomed + // dispatch. + async function seedStaleLifecycleReusableLease( + leasePolicy: "reuse_by_environment" | "retain_on_failure" = "reuse_by_environment", + ) { + const pluginId = randomUUID(); + const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); + const providerConfig = { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + }; + const environment = { + ...baseEnvironment, + name: "Reusable Plugin Sandbox", + driver: "sandbox", + config: providerConfig, + }; + await environmentService(db).update(environment.id, { + driver: "sandbox", + name: environment.name, + config: providerConfig, + }); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: "acme.fake-sandbox-provider", + packageName: "@acme/fake-sandbox-provider", + version: "1.0.0", + apiVersion: 1, + categories: ["automation"], + manifestJson: { + id: "acme.fake-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Fake Sandbox Provider", + description: "Test schema-driven provider", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + supportsReusableLeases: true, + configSchema: { type: "object" }, + }, + ], + }, + status: "ready", + installOrder: 1, + updatedAt: new Date(), + } as any); + const executionWorkspaceId = randomUUID(); + const projectId = randomUUID(); + await db.insert(projects).values({ + id: projectId, + companyId, + name: `Workspace ${projectId.slice(0, 8)}`, + status: "active", + createdAt: new Date(), + updatedAt: new Date(), + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + mode: "shared_workspace", + strategyType: "project_primary", + name: "Reusable workspace", + status: "active", + providerType: "local_fs", + createdAt: new Date(), + updatedAt: new Date(), + }); + const lease = await environmentService(db).acquireLease({ + companyId, + environmentId: environment.id, + executionWorkspaceId, + heartbeatRunId: runId, + leasePolicy, + provider: "fake-plugin", + providerLeaseId: "stale-lifecycle-lease", + metadata: { + agentId, + driver: "sandbox", + pluginId, + pluginKey: "acme.fake-sandbox-provider", + sandboxProviderPlugin: true, + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + }, + }); + + // The worker is running, but its discovery list dropped the reusable-lease + // lifecycle methods. `call` throws on any lifecycle RPC so a regression that + // dispatches one fails the test through the `not.toHaveBeenCalledWith` check. + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + throw new Error(`Unexpected plugin method: ${method}`); + }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentAcquireLease", "environmentExecute"] })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + return { pluginId, environment, runId, lease, workerManager, runtimeWithPlugin }; + } + + it("routes release to pending_cleanup when the worker no longer advertises the release lifecycle method", async () => { + const { pluginId, lease, workerManager, runtimeWithPlugin } = await seedStaleLifecycleReusableLease(); + + const released = await runtimeWithPlugin.releaseRunLeases(lease.heartbeatRunId!); + + expect(released).toHaveLength(1); + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentReleaseLease", + expect.anything(), + expect.anything(), + ); + // The failed release verification must enter the pending-cleanup retry flow. + // The reaper sweeps only `pending_cleanup` leases, so a `released` status + // here would strand the still-active provider resource. + await expect(environmentService(db).getLeaseById(lease.id)).resolves.toMatchObject({ + status: "pending_cleanup", + cleanupStatus: "failed", + failureReason: "release_cleanup_failed", + }); + }); + + it("retains a retain_on_failure lease on failed release instead of routing to pending_cleanup", async () => { + const { lease, runtimeWithPlugin } = await seedStaleLifecycleReusableLease("retain_on_failure"); + + const released = await runtimeWithPlugin.releaseRunLeases(lease.heartbeatRunId!, "failed"); + + expect(released).toHaveLength(1); + // A retain_on_failure lease keeps the provider resource for reuse. The + // reaper destroys `pending_cleanup` leases, so the retained lease must not + // enter that flow even when the release verification fails. + await expect(environmentService(db).getLeaseById(lease.id)).resolves.toMatchObject({ + status: "retained", + cleanupStatus: "failed", + }); + }); + + it("fails closed on expiry destruction when the worker no longer advertises the destroy lifecycle method", async () => { + const { pluginId, lease, workerManager, runtimeWithPlugin } = await seedStaleLifecycleReusableLease(); + + await runtimeWithPlugin.releaseRunLeases(lease.heartbeatRunId!, "expired"); + + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentDestroyLease", + expect.anything(), + expect.anything(), + ); + await expect(environmentService(db).getLeaseById(lease.id)).resolves.toMatchObject({ + status: "pending_cleanup", + cleanupStatus: "failed", + }); + }); + it("does not resume released reusable plugin sandbox leases after provider config drift", async () => { const pluginId = randomUUID(); const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); @@ -2126,6 +2508,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2315,6 +2698,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2404,6 +2788,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2552,6 +2937,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2573,6 +2959,336 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.anything(), 31234); }); + it("does not resume a lease when the nested capability disables reusable leases", async () => { + // The provider declares the legacy `supportsReusableLeases: true` flag but + // the nested `sandboxCapabilities.reusableLeases: false`. The nested value + // wins through the capability contract, so acquisition must acquire a fresh + // lease and must never resume the existing reusable lease. + const pluginId = randomUUID(); + const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); + const providerConfig = { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + }; + const environment = { + ...baseEnvironment, + name: "Nested-disabled Plugin Sandbox", + driver: "sandbox", + config: providerConfig, + }; + await environmentService(db).update(environment.id, { + driver: "sandbox", + name: environment.name, + config: providerConfig, + }); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: "acme.nested-disabled-sandbox-provider", + packageName: "@acme/nested-disabled-sandbox-provider", + version: "1.0.0", + apiVersion: 1, + categories: ["automation"], + manifestJson: { + id: "acme.nested-disabled-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Nested-disabled Sandbox Provider", + description: "Test provider with a legacy flag and a disabled nested capability", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + supportsReusableLeases: true, + sandboxCapabilities: { reusableLeases: false }, + configSchema: { + type: "object", + properties: { + image: { type: "string" }, + timeoutMs: { type: "number" }, + reuseLease: { type: "boolean" }, + }, + }, + }, + ], + }, + status: "ready", + installOrder: 1, + updatedAt: new Date(), + } as any); + const executionWorkspaceId = randomUUID(); + const projectId = randomUUID(); + await db.insert(projects).values({ + id: projectId, + companyId, + name: `Workspace ${projectId.slice(0, 8)}`, + status: "active", + createdAt: new Date(), + updatedAt: new Date(), + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + mode: "shared_workspace", + strategyType: "project_primary", + name: "Nested-disabled workspace", + status: "active", + providerType: "local_fs", + createdAt: new Date(), + updatedAt: new Date(), + }); + await environmentService(db).acquireLease({ + companyId, + environmentId: environment.id, + executionWorkspaceId, + heartbeatRunId: runId, + leasePolicy: "reuse_by_environment", + provider: "fake-plugin", + providerLeaseId: "old-plugin-lease", + metadata: { + agentId, + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + reusableSandboxLease: { + version: 1, + companyId, + environmentId: environment.id, + executionWorkspaceId, + agentId, + adapterType: null, + provider: "fake-plugin", + runtimeFingerprint: reusableRuntimeFingerprint({ + provider: "fake-plugin", + adapterType: null, + config: providerConfig, + }), + }, + }, + }); + + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "fresh-plugin-lease", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + remoteCwd: "/workspace", + }, + }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + agentId, + heartbeatRunId: runId, + persistedExecutionWorkspace: { + id: executionWorkspaceId, + mode: "shared_workspace", + }, + }); + + expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease"); + expect(acquired.lease.leasePolicy).toBe("ephemeral"); + expect(workerManager.call).toHaveBeenCalledTimes(1); + expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.anything(), 31234); + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentResumeLease", + expect.anything(), + expect.anything(), + ); + }); + + it("fails closed and does not resume when the worker does not verify the reuse methods", async () => { + // The provider declares `reusableLeases: true`, but its worker advertises + // neither `environmentResumeLease` nor `environmentReleaseLease`. The runtime + // must not resume or reuse the lease, because it cannot dispatch a resume or + // a release the worker does not serve. It acquires a fresh ephemeral lease + // and leaves the old reusable lease untouched. + const pluginId = randomUUID(); + const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); + const providerConfig = { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + }; + const environment = { + ...baseEnvironment, + name: "Unverified-worker Plugin Sandbox", + driver: "sandbox", + config: providerConfig, + }; + await environmentService(db).update(environment.id, { + driver: "sandbox", + name: environment.name, + config: providerConfig, + }); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: "acme.unverified-worker-sandbox-provider", + packageName: "@acme/unverified-worker-sandbox-provider", + version: "1.0.0", + apiVersion: 1, + categories: ["automation"], + manifestJson: { + id: "acme.unverified-worker-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Unverified-worker Sandbox Provider", + description: "Test provider that declares reusable leases but whose worker lacks the reuse methods", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + sandboxCapabilities: { reusableLeases: true }, + configSchema: { + type: "object", + properties: { + image: { type: "string" }, + timeoutMs: { type: "number" }, + reuseLease: { type: "boolean" }, + }, + }, + }, + ], + }, + status: "ready", + installOrder: 1, + updatedAt: new Date(), + } as any); + const executionWorkspaceId = randomUUID(); + const projectId = randomUUID(); + await db.insert(projects).values({ + id: projectId, + companyId, + name: `Workspace ${projectId.slice(0, 8)}`, + status: "active", + createdAt: new Date(), + updatedAt: new Date(), + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + mode: "shared_workspace", + strategyType: "project_primary", + name: "Unverified-worker workspace", + status: "active", + providerType: "local_fs", + createdAt: new Date(), + updatedAt: new Date(), + }); + const existingLease = await environmentService(db).acquireLease({ + companyId, + environmentId: environment.id, + executionWorkspaceId, + heartbeatRunId: runId, + leasePolicy: "reuse_by_environment", + provider: "fake-plugin", + providerLeaseId: "old-plugin-lease", + metadata: { + agentId, + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + reusableSandboxLease: { + version: 1, + companyId, + environmentId: environment.id, + executionWorkspaceId, + agentId, + adapterType: null, + provider: "fake-plugin", + runtimeFingerprint: reusableRuntimeFingerprint({ + provider: "fake-plugin", + adapterType: null, + config: providerConfig, + }), + }, + }, + }); + + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "fresh-plugin-lease", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + remoteCwd: "/workspace", + }, + }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + // The worker verifies only `environmentExecute`. It advertises neither + // reuse verb, so the reusable-lease capability fails closed. + getWorker: vi.fn(() => ({ supportedMethods: ["environmentExecute"] })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + agentId, + heartbeatRunId: runId, + persistedExecutionWorkspace: { + id: executionWorkspaceId, + mode: "shared_workspace", + }, + }); + + expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease"); + expect(acquired.lease.leasePolicy).toBe("ephemeral"); + expect(workerManager.call).toHaveBeenCalledTimes(1); + expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.anything(), 31234); + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentResumeLease", + expect.anything(), + expect.anything(), + ); + // The runtime leaves the old reusable lease untouched: it neither resumes + // nor destroys a lease it cannot serve. + await expect(environmentService(db).getLeaseById(existingLease.id)).resolves.toMatchObject({ + status: "active", + leasePolicy: "reuse_by_environment", + }); + }); + it("destroys scoped reusable plugin-backed sandbox leases", async () => { const { pluginId, companyId, executionWorkspaceId, reusableLease } = await seedReusablePluginSandboxLease(); @@ -2585,6 +3301,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2620,6 +3337,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { const offlineWorkerManager = { isRunning: vi.fn(() => false), call: vi.fn(), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithOfflinePlugin = environmentRuntimeService(db, { pluginWorkerManager: offlineWorkerManager, @@ -2649,6 +3367,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithRecoveredPlugin = environmentRuntimeService(db, { pluginWorkerManager: recoveredWorkerManager, @@ -2679,6 +3398,262 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); }); + it("resolves effective capabilities from the lease's exact plugin, not an earlier plugin that shares the driver key", async () => { + // The helper seeds the plugin that owns the lease. Pin it to `pluginId` + // through the lease metadata and give it a lower-priority sibling. + const { pluginId, environment, reusableLease } = await seedReusablePluginSandboxLease(); + + // Rewrite the owner plugin so it DENIES reusable leases through the nested + // capability declaration. + await db + .update(plugins) + .set({ + manifestJson: { + id: "acme.reusable-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Reusable Sandbox Provider", + description: "Owner plugin that denies reusable leases", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + sandboxCapabilities: { reusableLeases: false }, + configSchema: { type: "object", properties: {} }, + }, + ], + }, + updatedAt: new Date(), + } as any) + .where(eq(plugins.id, pluginId)); + + // Install an EARLIER plugin that shares the driver key and grants reusable + // leases. It is not ready and never acquired this lease. A resolver keyed by + // driver key alone would read this declaration and grant a capability the + // owner plugin denied. + const collidingPluginId = randomUUID(); + await db.insert(plugins).values({ + id: collidingPluginId, + pluginKey: "acme.colliding-sandbox-provider", + packageName: "@acme/colliding-sandbox-provider", + version: "1.0.0", + apiVersion: 1, + categories: ["automation"], + manifestJson: { + id: "acme.colliding-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Colliding Sandbox Provider", + description: "Earlier plugin that shares the driver key", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + supportsReusableLeases: true, + configSchema: { type: "object", properties: {} }, + }, + ], + }, + status: "installed", + installOrder: 0, + updatedAt: new Date(), + } as any); + + // The owner worker verifies the reuse verbs and the sync verbs, so every + // capability is verified. Only the owner's declaration can narrow one. + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(), + getWorker: vi.fn((id: string) => + id === pluginId + ? { + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentSyncIn", + "environmentSyncOut", + ], + } + : undefined, + ), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + expect(reusableLease.metadata?.pluginId).toBe(pluginId); + + const effective = await runtimeWithPlugin.effectiveSandboxCapabilities({ + environment, + lease: reusableLease, + }); + + // The runtime read the owner plugin's declaration, so it denies reusable + // leases even though the earlier plugin grants them. + expect(effective?.reusableLeases).toBe(false); + // The owner plugin does not restrict native sync, and its worker verifies + // the sync verbs, so those stay granted. This proves the resolver read the + // owner declaration and did not fail every capability closed. + expect(effective?.nativeSyncIn).toBe(true); + expect(effective?.nativeSyncOut).toBe(true); + }); + + it("fails every effective capability closed when the pinned plugin id is absent from the registry", async () => { + // The lease pins a plugin id, but that plugin record is gone. A stale worker + // entry still advertises every method. The runtime must not read the stale + // methods; it must fail closed because the exact-plugin identity is gone. + const { pluginId, environment, reusableLease } = await seedReusablePluginSandboxLease(); + await db.delete(plugins).where(eq(plugins.id, pluginId)); + + const workerManager = { + isRunning: vi.fn(() => true), + call: vi.fn(), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentExecute", + "environmentSyncIn", + "environmentSyncOut", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const effective = await runtimeWithPlugin.effectiveSandboxCapabilities({ + environment, + lease: reusableLease, + }); + + for (const key of SANDBOX_CAPABILITY_KEYS) { + expect(effective?.[key]).toBe(false); + } + }); + + it("fails every effective capability closed when the pinned plugin no longer declares this provider key", async () => { + // The pinned plugin still exists, but it no longer declares a + // `sandbox_provider` driver with this key (here it changed the driver kind). + // A running worker still advertises every method. The runtime must fail + // closed because the exact-plugin declaration is gone. + const { pluginId, environment, reusableLease } = await seedReusablePluginSandboxLease(); + await db + .update(plugins) + .set({ + manifestJson: { + id: "acme.reusable-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Reusable Sandbox Provider", + description: "Owner plugin that no longer declares the provider key", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + // The key exists, but the kind is now a plain environment driver, + // not a sandbox provider. The by-id resolver fails closed. + kind: "environment_driver", + displayName: "Fake Plugin", + configSchema: { type: "object", properties: {} }, + }, + ], + }, + updatedAt: new Date(), + } as any) + .where(eq(plugins.id, pluginId)); + + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentExecute", + "environmentSyncIn", + "environmentSyncOut", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const effective = await runtimeWithPlugin.effectiveSandboxCapabilities({ + environment, + lease: reusableLease, + }); + + for (const key of SANDBOX_CAPABILITY_KEYS) { + expect(effective?.[key]).toBe(false); + } + }); + + it("defers to verified worker discovery for a valid pinned plugin that omits sandboxCapabilities", async () => { + // A valid, identified plugin whose manifest declares no `sandboxCapabilities` + // and no legacy reuse flag. Its worker verifies the sync verbs. An omitted + // declaration is NOT an identity failure: the runtime defers to the verified + // baseline, so native sync stays granted while unverified capabilities stay + // false. + const { pluginId, environment, reusableLease } = await seedReusablePluginSandboxLease(); + await db + .update(plugins) + .set({ + manifestJson: { + id: "acme.reusable-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Reusable Sandbox Provider", + description: "Owner plugin that omits the capability declaration", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + configSchema: { type: "object", properties: {} }, + }, + ], + }, + updatedAt: new Date(), + } as any) + .where(eq(plugins.id, pluginId)); + + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(), + getWorker: vi.fn((id: string) => + id === pluginId + ? { supportedMethods: ["environmentSyncIn", "environmentSyncOut"] } + : undefined, + ), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const effective = await runtimeWithPlugin.effectiveSandboxCapabilities({ + environment, + lease: reusableLease, + }); + + // The worker verified the sync verbs and the omitted declaration adds no + // restriction, so native sync stays granted. + expect(effective?.nativeSyncIn).toBe(true); + expect(effective?.nativeSyncOut).toBe(true); + // The worker did not verify the reuse verbs, so reusable leases stay false. + expect(effective?.reusableLeases).toBe(false); + }); + it("releases a sandbox run lease from metadata after the environment config changes", async () => { const { companyId, environment, runId } = await seedEnvironment({ driver: "sandbox", @@ -2810,6 +3785,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } throw new Error(`Unexpected plugin method: ${method}`); }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); @@ -2856,6 +3832,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } return undefined; }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager, @@ -3011,6 +3988,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } return undefined; }), + getWorker: vi.fn(() => ({ supportedMethods: ["environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease"] })), } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager, diff --git a/server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts b/server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts index 718c399153..766a9e4fc5 100644 --- a/server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts +++ b/server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts @@ -513,3 +513,80 @@ describe("listReadyPluginEnvironmentDrivers worker recovery", () => { } }); }); + +describe("listReadyPluginEnvironmentDrivers reusable-lease method verification", () => { + beforeEach(() => { + mockRegistry.getById.mockReset(); + mockRegistry.list.mockReset(); + mockRegistry.listConfigs.mockReset(); + mockRegistry.update.mockReset(); + }); + + function createRunningWorker(supportedMethods: string[]) { + return { + isRunning: vi.fn(() => true), + getWorker: vi.fn(() => ({ supportedMethods })), + } as unknown as PluginWorkerManager; + } + + it("verifies reusable-lease methods when the worker advertises all lifecycle methods", async () => { + mockRegistry.list.mockResolvedValue([createPlugin("ready")]); + const drivers = await listReadyPluginEnvironmentDrivers({ + db: {} as never, + workerManager: createRunningWorker([ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + "environmentExecute", + ]), + }); + + expect(drivers).toHaveLength(1); + expect(drivers[0]?.reusableLeaseMethodsVerified).toBe(true); + }); + + it("does not verify reusable-lease methods when the worker omits the resume method", async () => { + mockRegistry.list.mockResolvedValue([createPlugin("ready")]); + const drivers = await listReadyPluginEnvironmentDrivers({ + db: {} as never, + workerManager: createRunningWorker([ + "environmentReleaseLease", + "environmentDestroyLease", + ]), + }); + + expect(drivers).toHaveLength(1); + expect(drivers[0]?.reusableLeaseMethodsVerified).toBe(false); + }); + + it("does not verify reusable-lease methods when the worker omits the release method", async () => { + mockRegistry.list.mockResolvedValue([createPlugin("ready")]); + const drivers = await listReadyPluginEnvironmentDrivers({ + db: {} as never, + workerManager: createRunningWorker([ + "environmentResumeLease", + "environmentDestroyLease", + ]), + }); + + expect(drivers).toHaveLength(1); + expect(drivers[0]?.reusableLeaseMethodsVerified).toBe(false); + }); + + it("does not verify reusable-lease methods when the worker omits the destroy method", async () => { + // A provider that resumes and releases but cannot destroy a stale lease must + // not present as reusable. The reuse path destroys the stale lease when a + // resume fails, so without destroy the runtime would strand the lease. + mockRegistry.list.mockResolvedValue([createPlugin("ready")]); + const drivers = await listReadyPluginEnvironmentDrivers({ + db: {} as never, + workerManager: createRunningWorker([ + "environmentResumeLease", + "environmentReleaseLease", + ]), + }); + + expect(drivers).toHaveLength(1); + expect(drivers[0]?.reusableLeaseMethodsVerified).toBe(false); + }); +}); diff --git a/server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts b/server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts new file mode 100644 index 0000000000..d1716928e4 --- /dev/null +++ b/server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts @@ -0,0 +1,80 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import type { PaperclipPluginManifestV1 } from "@paperclipai/shared"; +import { listReadyPluginEnvironmentDrivers } from "../services/plugin-environment-driver.js"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; + +const mockRegistry = vi.hoisted(() => ({ + getById: vi.fn(), + list: vi.fn(), + listConfigs: vi.fn(), + update: vi.fn(), +})); + +vi.mock("../services/plugin-registry.js", () => ({ + pluginRegistryService: () => mockRegistry, +})); + +const PLUGIN_ID = "plugin-capability"; +const PLUGIN_KEY = "paperclip.capability-sandbox-provider"; + +const manifest: PaperclipPluginManifestV1 = { + id: PLUGIN_KEY, + apiVersion: 1, + version: "1.0.0", + displayName: "Capability Sandbox Provider", + description: "Sandbox provider that declares fine-grained capabilities.", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "capability-provider", + kind: "sandbox_provider", + displayName: "Capability Provider", + supportsReusableLeases: true, + sandboxCapabilities: { + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: true, + persistentProcessSessions: false, + }, + configSchema: { type: "object", properties: {} }, + }, + ], +}; + +describe("listReadyPluginEnvironmentDrivers sandbox capability projection", () => { + beforeEach(() => { + mockRegistry.getById.mockReset(); + mockRegistry.list.mockReset(); + mockRegistry.listConfigs.mockReset(); + mockRegistry.update.mockReset(); + }); + + it("test_capabilities_propagate_from_manifest_to_ready_driver_projection", async () => { + mockRegistry.list.mockResolvedValue([ + { id: PLUGIN_ID, pluginKey: PLUGIN_KEY, status: "ready", manifestJson: manifest }, + ]); + const workerManager = { + isRunning: vi.fn(() => true), + getWorker: vi.fn(() => ({ status: "running" })), + } as unknown as PluginWorkerManager; + + const drivers = await listReadyPluginEnvironmentDrivers({ + db: {} as never, + workerManager, + }); + + expect(drivers).toHaveLength(1); + // The projection allowlist carries the raw declaration through unchanged, so + // no reader downstream loses the capability contract. + expect(drivers[0]?.sandboxCapabilities).toEqual({ + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: true, + persistentProcessSessions: false, + }); + expect(drivers[0]?.supportsReusableLeases).toBe(true); + }); +}); diff --git a/server/src/__tests__/sandbox-capability-contract.test.ts b/server/src/__tests__/sandbox-capability-contract.test.ts new file mode 100644 index 0000000000..eb5da33f9a --- /dev/null +++ b/server/src/__tests__/sandbox-capability-contract.test.ts @@ -0,0 +1,260 @@ +import { describe, expect, it } from "vitest"; +import { + SANDBOX_CAPABILITY_KEYS, + buildSandboxCapabilityNarrowing, + builtinSandboxProviderVerifiedMethods, + resolveEffectiveSandboxCapabilities, +} from "../services/environment-runtime.js"; + +// The worker verbs a fully-capable plug-in provider advertises. +const ALL_PLUGIN_METHODS = [ + "environmentAcquireLease", + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + "environmentExecute", + "environmentSyncIn", + "environmentSyncOut", +]; + +describe("sandbox capability contract normalizer", () => { + it("test_absent_declaration_defers_to_worker_supported_methods_discovery", () => { + // No declaration at all. The effective set must fall back to what the worker + // verified, so a third-party provider that implements the sync hooks keeps + // native sync without declaring it. + const effective = resolveEffectiveSandboxCapabilities({ + verifiedMethods: ["environmentSyncIn", "environmentSyncOut"], + declared: null, + }); + + expect(effective.nativeSyncIn).toBe(true); + expect(effective.nativeSyncOut).toBe(true); + // The worker did not verify these verbs, so the baseline is false. + expect(effective.persistentProcessSessions).toBe(false); + expect(effective.reusableLeases).toBe(false); + }); + + it("test_effective_capabilities_are_subset_of_verified_and_declared", () => { + const verifiedMethods = ["environmentExecute"]; + const declared = { + persistentProcessSessions: true, + independentControlCommands: false, + nativeSyncIn: true, + }; + const effective = resolveEffectiveSandboxCapabilities({ verifiedMethods, declared }); + + // Verified + declared true. + expect(effective.persistentProcessSessions).toBe(true); + // Declared false, so removed even though verified. + expect(effective.independentControlCommands).toBe(false); + // Declared true but not verified, so removed. + expect(effective.nativeSyncIn).toBe(false); + + // Every effective capability must be a subset of the verified set and the + // declaration: an effective `true` never appears where the worker did not + // verify or the declaration set `false`. + const verifiedOnly = resolveEffectiveSandboxCapabilities({ verifiedMethods }); + for (const key of SANDBOX_CAPABILITY_KEYS) { + if (effective[key]) { + expect(verifiedOnly[key]).toBe(true); + expect((declared as Record)[key]).not.toBe(false); + } + } + }); + + it("test_kubernetes_job_lease_disables_native_sync", () => { + const narrowing = buildSandboxCapabilityNarrowing({ + leasePolicy: "ephemeral", + leaseMetadata: { backend: "job" }, + config: {}, + }); + const effective = resolveEffectiveSandboxCapabilities({ + verifiedMethods: ALL_PLUGIN_METHODS, + declared: { nativeSyncIn: true, nativeSyncOut: true }, + narrowing, + }); + + expect(effective.nativeSyncIn).toBe(false); + expect(effective.nativeSyncOut).toBe(false); + // A non-sync capability is unaffected by the job-lease narrowing. + expect(effective.persistentProcessSessions).toBe(true); + + // The `nativeFileSyncUnsupported` lease flag narrows the same way. + const flaggedNarrowing = buildSandboxCapabilityNarrowing({ + leasePolicy: "ephemeral", + leaseMetadata: { nativeFileSyncUnsupported: true }, + config: {}, + }); + expect(flaggedNarrowing.nativeSyncIn).toBe(false); + expect(flaggedNarrowing.nativeSyncOut).toBe(false); + }); + + it("test_daytona_persistent_process_sessions_follows_use_sessions_config", () => { + const verifiedMethods = ["environmentExecute"]; + const declared = { persistentProcessSessions: true }; + + const sessionsOff = resolveEffectiveSandboxCapabilities({ + verifiedMethods, + declared, + narrowing: buildSandboxCapabilityNarrowing({ + leasePolicy: "ephemeral", + leaseMetadata: {}, + config: { useSessions: false }, + }), + }); + expect(sessionsOff.persistentProcessSessions).toBe(false); + + const sessionsOn = resolveEffectiveSandboxCapabilities({ + verifiedMethods, + declared, + narrowing: buildSandboxCapabilityNarrowing({ + leasePolicy: "ephemeral", + leaseMetadata: {}, + config: { useSessions: true }, + }), + }); + expect(sessionsOn.persistentProcessSessions).toBe(true); + + // A provider config that omits `useSessions` adds no narrowing here. + const noKey = buildSandboxCapabilityNarrowing({ + leasePolicy: "ephemeral", + leaseMetadata: {}, + config: {}, + }); + expect(noKey.persistentProcessSessions).toBeUndefined(); + }); + + it("test_config_resolution_failure_fails_closed_on_persistent_process_sessions", () => { + const verifiedMethods = ["environmentExecute"]; + const declared = { persistentProcessSessions: true }; + + // Config resolution failed, so the runtime cannot read `useSessions`. The + // narrowing must deny persistent process sessions instead of allowing them + // through an empty config. Without the fail-closed guard this narrowing key + // stays undefined and `persistentProcessSessions` resolves to true. + const narrowing = buildSandboxCapabilityNarrowing({ + leasePolicy: "ephemeral", + leaseMetadata: {}, + config: {}, + configResolutionFailed: true, + }); + expect(narrowing.persistentProcessSessions).toBe(false); + + const effective = resolveEffectiveSandboxCapabilities({ + verifiedMethods, + declared, + narrowing, + }); + expect(effective.persistentProcessSessions).toBe(false); + + // Native sync and reusable lease enforcement stay unchanged on failure. + const syncNarrowing = buildSandboxCapabilityNarrowing({ + leasePolicy: "reuse_by_environment", + leaseMetadata: { backend: "job" }, + config: {}, + configResolutionFailed: true, + }); + expect(syncNarrowing.reusableLeases).toBe(true); + expect(syncNarrowing.nativeSyncIn).toBe(false); + expect(syncNarrowing.nativeSyncOut).toBe(false); + }); + + it("test_builtin_provider_branch_uses_same_normalizer_as_plugin_branch", () => { + const declared = { reusableLeases: true, persistentProcessSessions: true }; + + // A built-in provider maps its own methods to the same verb names. + const builtinMethods = builtinSandboxProviderVerifiedMethods({ + supportsReusableLeases: true, + execute: () => undefined, + }); + const builtinEffective = resolveEffectiveSandboxCapabilities({ + verifiedMethods: builtinMethods, + declared, + }); + + // A plug-in provider that advertises the equivalent verbs. + const pluginEffective = resolveEffectiveSandboxCapabilities({ + verifiedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + "environmentExecute", + ], + declared, + }); + + // The one normalizer drives both branches, so equivalent verb sets resolve + // to the identical effective capabilities. + expect(builtinEffective).toEqual(pluginEffective); + expect(builtinEffective.reusableLeases).toBe(true); + expect(builtinEffective.persistentProcessSessions).toBe(true); + // A built-in provider has no native sync hooks, so it never verifies sync. + expect(builtinEffective.nativeSyncIn).toBe(false); + + // A built-in provider without an execute method verifies no exec capability. + const noExec = resolveEffectiveSandboxCapabilities({ + verifiedMethods: builtinSandboxProviderVerifiedMethods({ supportsReusableLeases: false }), + declared: { persistentProcessSessions: true }, + }); + expect(noExec.persistentProcessSessions).toBe(false); + }); + + it("test_present_declaration_never_grants_beyond_verified_supported_methods", () => { + // One case per capability: the declaration sets the flag `true`, the worker + // lacks a prerequisite verb, and the effective value stays `false`. + for (const key of SANDBOX_CAPABILITY_KEYS) { + const effective = resolveEffectiveSandboxCapabilities({ + verifiedMethods: [], + declared: { [key]: true }, + }); + expect(effective[key]).toBe(false); + } + + // A single missing prerequisite verb is enough: reusable leases needs + // resume, release, and destroy, so resume alone does not grant it. + const resumeOnly = resolveEffectiveSandboxCapabilities({ + verifiedMethods: ["environmentResumeLease"], + declared: { reusableLeases: true }, + }); + expect(resumeOnly.reusableLeases).toBe(false); + }); + + it("test_reusable_provider_without_destroy_support_resolves_false", () => { + // A provider that verifies resume and release but not destroy is not + // eligible for reusable leases. The reuse path destroys a stale lease when a + // resume fails, so a provider without destroy support would strand the lease. + const resumeAndReleaseOnly = resolveEffectiveSandboxCapabilities({ + verifiedMethods: ["environmentResumeLease", "environmentReleaseLease"], + declared: { reusableLeases: true }, + }); + expect(resumeAndReleaseOnly.reusableLeases).toBe(false); + + // Adding the destroy verb makes the same provider eligible. + const allReuseVerbs = resolveEffectiveSandboxCapabilities({ + verifiedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + declared: { reusableLeases: true }, + }); + expect(allReuseVerbs.reusableLeases).toBe(true); + }); + + it("test_unknown_or_unavailable_verification_resolves_false", () => { + const declaredAll = { + reusableLeases: true, + nativeSyncIn: true, + nativeSyncOut: true, + persistentProcessSessions: true, + independentControlCommands: true, + }; + + for (const verifiedMethods of [null, undefined, [] as string[]]) { + const effective = resolveEffectiveSandboxCapabilities({ verifiedMethods, declared: declaredAll }); + for (const key of SANDBOX_CAPABILITY_KEYS) { + expect(effective[key]).toBe(false); + } + } + }); +}); diff --git a/server/src/routes/environments.ts b/server/src/routes/environments.ts index d7f157c1c3..2590215ce9 100644 --- a/server/src/routes/environments.ts +++ b/server/src/routes/environments.ts @@ -8,6 +8,7 @@ import { finishEnvironmentCustomImageSetupSessionSchema, getEnvironmentCapabilities, probeEnvironmentConfigSchema, + resolveDeclaredSandboxCapabilities, redactEnvironmentCustomImageSetupSession, redactEnvironmentCustomImageTemplate, startEnvironmentCustomImageSetupSessionSchema, @@ -715,9 +716,19 @@ export function environmentRoutes( supportsSavedProbe: true, supportsUnsavedProbe: true, supportsRunExecution: true, - // Default absent to false, so the presentation agrees with the - // execution guard (=== true). - supportsReusableLeases: driver.supportsReusableLeases ?? false, + // Publish reusable-lease support only when the declaration allows it + // AND the live worker verified all reuse lifecycle methods, so the + // presentation matches the acquisition guard, which requires them. + // The declaration part uses the same resolver acquisition uses, so + // the nested `sandboxCapabilities` override wins over the legacy + // `supportsReusableLeases` flag: a manifest with legacy `true` and + // nested `false` presents as not reusable. Default an absent value + // to false with `=== true`. A ready worker that omits any reuse + // lifecycle method presents as not reusable, because acquisition + // would always fall back to an ephemeral lease. + supportsReusableLeases: + resolveDeclaredSandboxCapabilities(driver).reusableLeases === true + && driver.reusableLeaseMethodsVerified, supportsInteractiveSetup: driver.supportsInteractiveSetup, interactiveSetupConnectionTypes: driver.interactiveSetupConnectionTypes, supportsTemplateCapture: driver.supportsTemplateCapture, diff --git a/server/src/services/environment-execution-target.ts b/server/src/services/environment-execution-target.ts index 0cfb88806d..d2bb9d83d3 100644 --- a/server/src/services/environment-execution-target.ts +++ b/server/src/services/environment-execution-target.ts @@ -243,12 +243,71 @@ export async function resolveEnvironmentExecutionTarget(input: { // a recording tracer. const tracer = input.tracer ?? getStartupTracer(); + // Resolve the read-only effective capability snapshot for this lease. The + // runtime resolves it as the provider declaration ∩ the verified worker + // methods ∩ narrowing. Freeze it so a consumer reads it but never changes + // it. Track a resolution error apart from a genuinely absent snapshot: a + // rejected resolution must not read as an open grant. + let effectiveCapabilities: Awaited< + ReturnType> + > | null = null; + let capabilityResolutionFailed = false; + if (input.environmentRuntime?.effectiveSandboxCapabilities && input.lease) { + try { + effectiveCapabilities = await input.environmentRuntime.effectiveSandboxCapabilities({ + environment: input.environment as Environment, + lease: input.lease, + }); + } catch { + // The runtime could not resolve the snapshot. Fail closed for the + // persistent-session gates below; never grant persistent-session + // behavior from an unknown capability set. + capabilityResolutionFailed = true; + effectiveCapabilities = null; + } + } + + // Gate the sync, session, and execution decisions below on the effective + // snapshot. A genuinely absent snapshot (no runtime, no lease) keeps the + // prior behavior, so it never removes a working path. A present snapshot + // can only remove a capability, never add one back. + // + // Native file sync needs BOTH sync verbs: the runner exposes syncIn and + // syncOut both-or-neither, so a consumer either uses the native path for + // both directions or keeps the base64 fallback for both. When the snapshot + // removes either verb, keep the byte-identical base64 fallback. When the + // resolution failed, fail closed and keep the base64 fallback too: an + // unverified provider never gets the native sync path. This preserves the + // existing reusable-lease sync enforcement unchanged. + const nativeSyncAllowed = + !capabilityResolutionFailed && + (!effectiveCapabilities || + (effectiveCapabilities.nativeSyncIn && effectiveCapabilities.nativeSyncOut)); + // The persistent-session output-streaming path needs the provider to keep a + // persistent process session AND to run independent one-shot control + // commands beside the long-lived agent command. When the snapshot removes + // either capability, drop back to the host output-file poll path. When the + // resolution failed, fail closed and keep the poll path too. + const sessionOutputStreamingAllowed = + !capabilityResolutionFailed && + (!effectiveCapabilities || + (effectiveCapabilities.persistentProcessSessions && + effectiveCapabilities.independentControlCommands)); + // A command that opts onto the persistent session needs the provider to keep + // persistent process sessions. When the snapshot removes that capability, + // never force the session; the command runs one-shot instead. When the + // resolution failed, fail closed and never force the session. + const persistentSessionsAllowed = + !capabilityResolutionFailed && + (!effectiveCapabilities || effectiveCapabilities.persistentProcessSessions); + return { kind: "remote", transport: "sandbox", providerKey: parsed.config.provider, shellCommand, remoteCwd, + ...(effectiveCapabilities ? { effectiveCapabilities: Object.freeze({ ...effectiveCapabilities }) } : {}), environmentId: input.environment.id ?? null, leaseId: input.leaseId ?? null, timeoutMs, @@ -258,8 +317,11 @@ export async function resolveEnvironmentExecutionTarget(input: { streamRunLogs: parsed.config.streamRunLogs !== false, // Interactive ACP output streaming through the persistent session log // stream. Default OFF: the process session bridge keeps the output-file - // poll unless an operator opts a sandbox environment in. - streamAgentSessionOutput: parsed.config.streamAgentSessionOutput === true, + // poll unless an operator opts a sandbox environment in. The effective + // snapshot gates it too: a provider that cannot keep persistent process + // sessions or run independent control commands keeps the poll path. + streamAgentSessionOutput: + parsed.config.streamAgentSessionOutput === true && sessionOutputStreamingAllowed, runner: input.environmentRuntime && input.lease ? { // Provider-backed sandbox RPCs do not surface bounded mid-stream @@ -328,7 +390,10 @@ export async function resolveEnvironmentExecutionTarget(input: { // The ACP process session bridge sets `useSession` so its // long-lived agent command opens the persistent session and // streams output, even though it runs with no active step. - forceSession: commandInput.useSession, + // The effective snapshot gates it: a provider that cannot + // keep persistent process sessions never forces the session, + // so the command runs one-shot instead. + forceSession: persistentSessionsAllowed ? commandInput.useSession : false, // The bridge control-plane execs set `bypassSession` so they // run one-shot and never queue behind the long-lived agent // command on the persistent session. An explicit bypass wins @@ -423,9 +488,11 @@ export async function resolveEnvironmentExecutionTarget(input: { } }, // Expose the native file-sync capability only when the provider's - // worker advertises BOTH sync verbs; otherwise leave syncIn/syncOut - // undefined so the orchestrator keeps the byte-identical base64 path. - ...(input.environmentRuntime.supportsSync({ + // worker advertises BOTH sync verbs AND the effective snapshot still + // grants native sync; otherwise leave syncIn/syncOut undefined so + // the orchestrator keeps the byte-identical base64 path. + ...(nativeSyncAllowed && + input.environmentRuntime.supportsSync({ environment: input.environment as Environment, lease: input.lease, }) diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index 3035819281..58e721be0a 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -10,7 +10,10 @@ import type { IssueExecutionWorkspaceSettings, PluginEnvironmentConfig, SandboxEnvironmentConfig, + SandboxProviderCapabilities, } from "@paperclipai/shared"; +import { resolveDeclaredSandboxCapabilities } from "@paperclipai/shared"; +import type { EffectiveSandboxCapabilities } from "@paperclipai/adapter-utils/execution-target"; import type { PluginEnvironmentAcquireLeaseParams, PluginEnvironmentExecuteResult, @@ -50,16 +53,195 @@ import { import { pluginRegistryService } from "./plugin-registry.js"; import type { ExecuteLogSink, PluginWorkerManager } from "./plugin-worker-manager.js"; import { + REUSABLE_LEASE_WORKER_METHODS, destroyPluginEnvironmentLease, executePluginEnvironmentCommand, realizePluginEnvironmentWorkspace, resolvePluginSandboxProviderDriverByKey, + resolvePluginSandboxProviderDriverById, resolvePluginExecuteRpcTimeoutMs, resumePluginEnvironmentLease, } from "./plugin-environment-driver.js"; import { collectSecretRefPaths } from "./json-schema-secret-refs.js"; import { buildWorkspaceRealizationRecordFromDriverInput } from "./workspace-realization.js"; +// --------------------------------------------------------------------------- +// Sandbox capability contract — one normalizer for both branches +// --------------------------------------------------------------------------- + +export const SANDBOX_CAPABILITY_KEYS = [ + "reusableLeases", + "nativeSyncIn", + "nativeSyncOut", + "persistentProcessSessions", + "independentControlCommands", +] as const; + +export type SandboxCapabilityKey = (typeof SANDBOX_CAPABILITY_KEYS)[number]; + +/** + * Verified prerequisite mapping: the worker methods each capability requires. + * + * Each capability maps to a list of requirement groups. A group holds one or + * more verbs; the group is met when the runtime verified AT LEAST ONE verb in + * it. A capability verifies only when EVERY group is met. + * + * The verbs are the worker method names from the worker discovery list (the + * plugin worker reports them from `handleInitialize`). A built-in provider maps + * its own methods to the same verb names through + * {@link builtinSandboxProviderVerifiedMethods}, so both branches share this + * mapping and the one normalizer below. + * + * The mapping was audited against the worker discovery list and the runtime + * execution guards: + * - `nativeSyncIn`/`nativeSyncOut` require the matching sync verb; the native + * sync guard checks both verbs before it routes a lease to the native hook. + * - `reusableLeases` requires `environmentResumeLease` (reattach), + * `environmentReleaseLease` (end-of-run release), and `environmentDestroyLease` + * (stale-lease teardown). All three run on the reuse path: the runtime resumes + * or releases the lease, and it destroys the stale lease when a resume fails + * before it acquires a fresh lease. + * - `persistentProcessSessions` and `independentControlCommands` require + * `environmentExecute`; both run commands through it. + */ +const SANDBOX_CAPABILITY_PREREQUISITE_METHODS: Record = { + // Reusable leases require ALL reuse verbs. Each verb is its own required + // group, so every one must be verified. The list function that publishes + // provider-level reusable support checks the same verbs; both read from + // `REUSABLE_LEASE_WORKER_METHODS`, so the runtime guard and the published + // value cannot drift. + reusableLeases: REUSABLE_LEASE_WORKER_METHODS.map((method) => [method]), + nativeSyncIn: [["environmentSyncIn"]], + nativeSyncOut: [["environmentSyncOut"]], + persistentProcessSessions: [["environmentExecute"]], + independentControlCommands: [["environmentExecute"]], +}; + +function capabilityIsVerified( + key: SandboxCapabilityKey, + verifiedMethods: ReadonlySet, +): boolean { + return SANDBOX_CAPABILITY_PREREQUISITE_METHODS[key].every((group) => + group.some((verb) => verifiedMethods.has(verb)), + ); +} + +/** + * Map a built-in sandbox provider's own methods to the worker verb names the + * prerequisite mapping uses, so the built-in branch and the plug-in branch feed + * the SAME normalizer. A built-in provider has no native sync hooks, so it never + * verifies a sync verb. It verifies `environmentExecute` when it implements + * `execute`, and the reuse verbs only when it declares `supportsReusableLeases`. + * A built-in reusable provider destroys its own leases in-process, so it + * verifies `environmentDestroyLease` with the two reuse verbs. + */ +export function builtinSandboxProviderVerifiedMethods( + provider: { supportsReusableLeases?: boolean; execute?: unknown } | null | undefined, +): string[] { + if (!provider) return []; + const methods: string[] = []; + if (typeof provider.execute === "function") { + methods.push("environmentExecute"); + } + if (provider.supportsReusableLeases === true) { + methods.push( + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ); + } + return methods; +} + +/** + * The one normalizer for the sandbox capability contract. It resolves the + * effective capability as verified ∩ declared ∩ narrowing. + * + * - `verifiedMethods` is the runtime's verified worker verb list (the plug-in + * worker's `supportedMethods`, or a built-in provider mapped through + * {@link builtinSandboxProviderVerifiedMethods}). A missing or empty list + * verifies nothing, so every capability resolves `false` (fail closed). + * - `declared` is the provider's declaration. An absent flag defers to the + * verified discovery baseline; it never grants a capability. A present flag + * can only remove a verified capability, never add one. + * - `narrowing` is the per-target restriction from the config or lease. An + * absent key applies no restriction; a `false` value removes the capability. + * + * A declaration never grants a capability the runtime did not verify, so a + * declared capability whose worker lacks a prerequisite verb resolves `false`. + */ +export function resolveEffectiveSandboxCapabilities(input: { + verifiedMethods?: readonly string[] | null; + declared?: Partial | null; + narrowing?: Partial> | null; +}): EffectiveSandboxCapabilities { + const verifiedMethods = new Set(input.verifiedMethods ?? []); + const declared = input.declared ?? {}; + const narrowing = input.narrowing ?? {}; + + const resolve = (key: SandboxCapabilityKey): boolean => { + const verified = capabilityIsVerified(key, verifiedMethods); + // An absent declaration defers to the verified baseline (true = no extra + // restriction). A present declaration can only narrow. + const declaredAllows = declared[key] ?? true; + // An absent narrowing applies no restriction. + const narrowingAllows = narrowing[key] ?? true; + return verified && declaredAllows && narrowingAllows; + }; + + return { + reusableLeases: resolve("reusableLeases"), + nativeSyncIn: resolve("nativeSyncIn"), + nativeSyncOut: resolve("nativeSyncOut"), + persistentProcessSessions: resolve("persistentProcessSessions"), + independentControlCommands: resolve("independentControlCommands"), + }; +} + +/** + * Build the per-target narrowing for a sandbox lease. Narrowing removes a + * capability that the provider verified and declared but that this specific + * lease or config cannot use. Each source is grounded in existing runtime + * behavior: + * - `reusableLeases` follows this lease's resolved policy (an ephemeral lease + * never reuses). + * - a Kubernetes Job lease disables native sync (mirrors the native sync guard, + * which falls back for a `job` backend or a `nativeFileSyncUnsupported` lease). + * - a session-based provider follows its `useSessions` config for persistent + * process sessions (default off); a config without the key adds no narrowing. + * - `configResolutionFailed` marks that the runtime could not resolve the + * provider config. The runtime cannot read `useSessions`, so it fails closed + * and narrows `persistentProcessSessions` to false. An empty config alone does + * not fail closed; only a resolution error does. + */ +export function buildSandboxCapabilityNarrowing(input: { + leasePolicy?: EnvironmentLease["leasePolicy"] | null; + leaseMetadata?: Record | null; + config?: Record | null; + configResolutionFailed?: boolean; +}): Partial> { + const narrowing: Partial> = {}; + const metadata = input.leaseMetadata ?? {}; + const config = input.config ?? {}; + + narrowing.reusableLeases = input.leasePolicy === "reuse_by_environment"; + + if (metadata.backend === "job" || metadata.nativeFileSyncUnsupported === true) { + narrowing.nativeSyncIn = false; + narrowing.nativeSyncOut = false; + } + + if (input.configResolutionFailed === true) { + // The runtime could not read `useSessions`, so it fails closed and denies + // persistent process sessions. + narrowing.persistentProcessSessions = false; + } else if ("useSessions" in config) { + narrowing.persistentProcessSessions = config.useSessions === true; + } + + return narrowing; +} + export function buildEnvironmentLeaseContext(input: { persistedExecutionWorkspace: Pick | null; }) { @@ -243,6 +425,11 @@ export interface EnvironmentRuntimeDriver { syncOut?(input: EnvironmentDriverSyncInput): Promise; /** True when the lease's plugin worker advertises both sync verbs. */ supportsSync?(input: EnvironmentDriverLeaseInput): boolean; + /** + * Resolve the read-only effective sandbox capability snapshot for a lease. + * Only the sandbox driver implements it; other drivers omit it. + */ + effectiveSandboxCapabilities?(input: EnvironmentDriverLeaseInput): Promise; } export interface EnvironmentRuntimeLeaseRecord { @@ -754,7 +941,7 @@ function createSandboxEnvironmentDriver( config: sandboxConfigForLeaseMetadata(metadataConfig), }); if (parsed.driver === "sandbox") { - return parsed.config as unknown as Record; + return dropInternalPluginSandboxConfigKeys(parsed.config as unknown as Record); } } @@ -766,7 +953,7 @@ function createSandboxEnvironmentDriver( input.environment, ); if (parsed.driver === "sandbox" && parsed.config.provider === input.provider) { - return parsed.config as unknown as Record; + return dropInternalPluginSandboxConfigKeys(parsed.config as unknown as Record); } } catch { // Lease metadata below is intentionally kept sufficient for cleanup @@ -776,7 +963,7 @@ function createSandboxEnvironmentDriver( return { provider: input.provider, - ...sanitizePluginSandboxConfigFromLeaseMetadata(input.lease.metadata), + ...dropInternalPluginSandboxConfigKeys(input.lease.metadata), }; } @@ -873,7 +1060,24 @@ function createSandboxEnvironmentDriver( const workerConfig = stripSandboxProviderEnvelope(parsed.config); const storedConfig = storedParsed.config; const providerConfigForLease = sandboxConfigForLeaseMetadata(storedConfig); - const supportsReusableLeases = pluginProvider.resolved.driver.supportsReusableLeases === true; + // Require the reusable-lease capability AND a worker that verifies the + // reuse methods. The provider must first opt in through the declaration: + // the nested `sandboxCapabilities.reusableLeases` wins over the legacy + // `supportsReusableLeases` flag. The worker must also verify + // `environmentResumeLease`, `environmentReleaseLease`, and + // `environmentDestroyLease` (the reuse prerequisite verbs) before the + // runtime resumes, releases, or tears down a reusable lease. A provider + // that declares `reusableLeases` true but whose worker does not verify + // all three methods fails closed and uses an ephemeral lease, so the + // runtime never dispatches a resume, a release, or a destroy the worker + // cannot serve, and it never strands a stale lease it cannot destroy. + const declaredReusableLeases = + resolveDeclaredSandboxCapabilities(pluginProvider.resolved.driver).reusableLeases === true; + const pluginVerifiedMethods = new Set( + pluginWorkerManager.getWorker(pluginProvider.resolved.plugin.id)?.supportedMethods ?? [], + ); + const supportsReusableLeases = + declaredReusableLeases && capabilityIsVerified("reusableLeases", pluginVerifiedMethods); const leaseFingerprint = supportsReusableLeases && parsed.config.reuseLease && @@ -955,33 +1159,48 @@ function createSandboxEnvironmentDriver( let providerLease: PluginEnvironmentLease | null = null; if (reusableLease?.providerLeaseId) { - try { - const resumed = await pluginWorkerManager.call( - pluginProvider.resolved.plugin.id, - "environmentResumeLease", - { - driverKey: parsed.config.provider, - companyId: input.companyId, - environmentId: input.environment.id, - issueId: input.issueId, - config: workerConfig, - providerLeaseId: reusableLease.providerLeaseId, - leaseMetadata: reusableLease.metadata ?? undefined, - }, - resolvePluginSandboxRpcTimeoutMs(workerConfig), - ); - providerLease = - typeof resumed.providerLeaseId === "string" && resumed.providerLeaseId.length > 0 - ? resumed - : null; - } catch { - providerLease = null; + // The `supportsReusableLeases` check above reads a snapshot of the + // worker methods. The runtime then does asynchronous database work + // (list, fingerprint, obsolete-lease cleanup) before this dispatch. A + // worker restart in that window can drop `environmentResumeLease` + // while the snapshot still marks the method verified. Re-check the + // live worker here and fail closed when the method is absent: skip the + // resume, destroy the stale reusable lease, and acquire a fresh lease + // below. The runtime never dispatches a resume the live worker cannot + // serve. + const workerVerifiesResume = pluginWorkerVerifiesLifecycleMethod( + pluginProvider.resolved.plugin.id, + "environmentResumeLease", + ); + if (workerVerifiesResume) { + try { + const resumed = await pluginWorkerManager.call( + pluginProvider.resolved.plugin.id, + "environmentResumeLease", + { + driverKey: parsed.config.provider, + companyId: input.companyId, + environmentId: input.environment.id, + issueId: input.issueId, + config: workerConfig, + providerLeaseId: reusableLease.providerLeaseId, + leaseMetadata: reusableLease.metadata ?? undefined, + }, + resolvePluginSandboxRpcTimeoutMs(workerConfig), + ); + providerLease = + typeof resumed.providerLeaseId === "string" && resumed.providerLeaseId.length > 0 + ? resumed + : null; + } catch { + providerLease = null; + } } if (!providerLease) { await destroyReusableSandboxLease({ environment: input.environment, lease: reusableLease, - failureReason: "resume_failed", + failureReason: workerVerifiesResume ? "resume_failed" : "resume_capability_lost", }); } } @@ -1067,7 +1286,12 @@ function createSandboxEnvironmentDriver( // heartbeat run that shares it. Filter to reusable policies and statuses // so non-reusable, cleanup-pending, or terminal rows can never be matched. const builtinSandboxProvider = getBuiltinSandboxProvider(parsed.config.provider); - const supportsReusableLeases = builtinSandboxProvider?.supportsReusableLeases === true; + // Resolve the DECLARED reusable-lease capability through the same contract + // as the plugin path, so the nested capability wins over the legacy flag. + const supportsReusableLeases = + resolveDeclaredSandboxCapabilities({ + supportsReusableLeases: builtinSandboxProvider?.supportsReusableLeases, + }).reusableLeases === true; const providerConfigForLease = sandboxConfigForLeaseMetadata(parsed.config); const leaseFingerprint = supportsReusableLeases && @@ -1437,6 +1661,86 @@ function createSandboxEnvironmentDriver( return await callPluginEnvironmentSync("environmentSyncOut", input); }, + async effectiveSandboxCapabilities(input) { + const metadata = input.lease.metadata ?? {}; + const providerKey = + readString(metadata.provider) ?? + (input.environment.driver === "sandbox" + ? readString((parseEnvironmentDriverConfig(input.environment).config as SandboxEnvironmentConfig).provider) + : null); + if (!providerKey) { + return resolveEffectiveSandboxCapabilities({ verifiedMethods: [], declared: null, narrowing: null }); + } + + let declared: SandboxProviderCapabilities | null = null; + let verifiedMethods: readonly string[] = []; + let config: Record = {}; + let configResolutionFailed = false; + + if (metadata.sandboxProviderPlugin) { + const pluginId = readString(metadata.pluginId); + // Read the declaration from the exact plugin that acquired the lease, + // not the first installed plugin with this driver key. A driver key is + // only unique inside one manifest, so two plugins can share it. The + // by-key resolver could intersect this lease's verified methods with a + // different plugin's declaration. This resolver fails closed: it returns + // null when the pinned plugin id is absent, or when that exact plugin no + // longer declares this provider key with the `sandbox_provider` kind. + const resolvedDriver = pluginId + ? await resolvePluginSandboxProviderDriverById({ + db, + pluginId, + driverKey: providerKey, + }) + : null; + if (!pluginId || !resolvedDriver) { + // Exact-plugin identity failure. The lease pins a plugin id, but the + // id is missing, that plugin is absent, or it no longer declares this + // provider key. Fail closed: resolve every effective capability to + // false, no matter what methods a stale or running worker still + // advertises. Do not read the worker methods here; passing them would + // let an identity-less lease keep a verified baseline. This differs + // from a valid plugin whose manifest merely omits + // `sandboxCapabilities`: that case keeps `declared` null below and + // defers to verified worker discovery. + return resolveEffectiveSandboxCapabilities({ + verifiedMethods: [], + declared: null, + narrowing: null, + }); + } + verifiedMethods = pluginWorkerManager?.getWorker(pluginId)?.supportedMethods ?? []; + declared = resolveDeclaredSandboxCapabilities(resolvedDriver.driver); + try { + config = (await resolvePluginSandboxRuntimeConfig({ + environment: input.environment, + lease: input.lease, + provider: providerKey, + })) as unknown as Record; + } catch { + // The runtime could not resolve the provider config. It cannot read + // `useSessions`, so it fails closed on persistent process sessions + // instead of leaving the config empty and allowing them. + configResolutionFailed = true; + } + } else { + const builtin = getBuiltinSandboxProvider(providerKey); + verifiedMethods = builtinSandboxProviderVerifiedMethods(builtin); + declared = resolveDeclaredSandboxCapabilities({ + supportsReusableLeases: builtin?.supportsReusableLeases, + }); + } + + const narrowing = buildSandboxCapabilityNarrowing({ + leasePolicy: input.lease.leasePolicy, + leaseMetadata: metadata, + config, + configResolutionFailed, + }); + + return resolveEffectiveSandboxCapabilities({ verifiedMethods, declared, narrowing }); + }, + async destroyRunLease(input) { return await destroyReusableSandboxLease({ environment: input.environment, @@ -1446,6 +1750,21 @@ function createSandboxEnvironmentDriver( }, }; + /** + * Verify that the live plugin worker still advertises a reusable-lease + * lifecycle method before the runtime dispatches that RPC. The worker reports + * `supportedMethods` from its discovery list on every start. A worker restart + * can drop a lifecycle method a reusable lease was created under. The runtime + * must not dispatch a lifecycle RPC the live worker does not advertise. It + * fails closed when the worker is absent or the method is stale, so a lease + * that a worker can no longer clean up goes to the cleanup reaper instead of + * a doomed RPC. + */ + function pluginWorkerVerifiesLifecycleMethod(pluginId: string, method: string): boolean { + const advertised = pluginWorkerManager?.getWorker(pluginId)?.supportedMethods ?? []; + return advertised.includes(method); + } + async function releasePluginBackedSandboxLease( input: EnvironmentDriverReleaseInput, ): Promise { @@ -1454,7 +1773,12 @@ function createSandboxEnvironmentDriver( const providerKey = readString(metadata.provider); let cleanupStatus: "success" | "failed" = "success"; - if (pluginId && providerKey && pluginWorkerManager?.isRunning(pluginId)) { + if ( + pluginId && + providerKey && + pluginWorkerManager?.isRunning(pluginId) && + pluginWorkerVerifiesLifecycleMethod(pluginId, "environmentReleaseLease") + ) { try { const config = await resolvePluginSandboxRuntimeConfig({ environment: input.environment, @@ -1479,12 +1803,26 @@ function createSandboxEnvironmentDriver( cleanupStatus = "failed"; } - const releaseStatus = - input.lease.leasePolicy === "retain_on_failure" && input.status === "failed" - ? ("retained" as const) + // A failed release verification leaves the provider resource active. The + // cleanup reaper retries only `pending_cleanup` leases, so route a failed + // release into that retry flow. A `retain_on_failure` lease keeps the + // resource on purpose for reuse, so it stays `retained` and never enters the + // reaper, which would destroy the resource the retain policy wants to keep. + const retained = + input.lease.leasePolicy === "retain_on_failure" && input.status === "failed"; + const releaseStatus = retained + ? ("retained" as const) + : cleanupStatus === "failed" + ? ("pending_cleanup" as const) : input.status; + const failureReason = + input.status === "failed" + ? "adapter_or_run_failure" + : cleanupStatus === "failed" + ? "release_cleanup_failed" + : undefined; return await environmentsSvc.releaseLease(input.lease.id, releaseStatus, { - failureReason: input.status === "failed" ? "adapter_or_run_failure" : undefined, + failureReason, cleanupStatus, }); } @@ -1501,7 +1839,12 @@ function createSandboxEnvironmentDriver( if (metadata.sandboxProviderPlugin) { const pluginId = readString(metadata.pluginId); const providerKey = readString(metadata.provider); - if (!pluginId || !providerKey || !pluginWorkerManager?.isRunning(pluginId)) { + if ( + !pluginId || + !providerKey || + !pluginWorkerManager?.isRunning(pluginId) || + !pluginWorkerVerifiesLifecycleMethod(pluginId, "environmentDestroyLease") + ) { cleanupStatus = "failed"; } else { const config = await resolvePluginSandboxRuntimeConfig({ @@ -1568,21 +1911,32 @@ function readString(value: unknown): string | null { return typeof value === "string" && value.length > 0 ? value : null; } +// Keys the runtime stores in the lease metadata that are not part of the +// provider driver config. Some are host-internal control fields. `remoteCwd` is +// a per-lease runtime value. The host reads `remoteCwd` from the lease metadata +// directly, so the worker never needs it as config. Drop every key here before +// the runtime sends a config to a lifecycle RPC. const INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS = new Set([ "driver", "executionWorkspaceMode", "pluginId", "pluginKey", "providerMetadata", + "remoteCwd", "shellCommand", "sandboxProviderPlugin", ]); -function sanitizePluginSandboxConfigFromLeaseMetadata( - metadata: Record | null | undefined, +// Drop the host-internal and per-lease runtime keys from a sandbox config +// record. The runtime stores these keys in the lease metadata and in some +// resolved configs, but the plugin worker must receive only the provider driver +// config. Use this on every config the runtime sends to a lifecycle RPC, so no +// host-internal field reaches the worker. +function dropInternalPluginSandboxConfigKeys( + config: Record | null | undefined, ): Record { const sanitized: Record = {}; - for (const [key, value] of Object.entries(metadata ?? {})) { + for (const [key, value] of Object.entries(config ?? {})) { if (INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS.has(key)) continue; sanitized[key] = value; } @@ -2114,6 +2468,13 @@ export function environmentRuntimeService( return driver?.supportsSync?.(input) ?? false; }, + async effectiveSandboxCapabilities( + input: EnvironmentDriverLeaseInput, + ): Promise { + const driver = getDriver(getLeaseDriverKey(input.lease, input.environment)); + return (await driver?.effectiveSandboxCapabilities?.(input)) ?? null; + }, + async syncIn(input: EnvironmentDriverSyncInput): Promise { const driver = requireDriverKey(getLeaseDriverKey(input.lease, input.environment)); if (!driver.syncIn) { diff --git a/server/src/services/plugin-environment-driver.ts b/server/src/services/plugin-environment-driver.ts index 077864bc84..19789984e7 100644 --- a/server/src/services/plugin-environment-driver.ts +++ b/server/src/services/plugin-environment-driver.ts @@ -30,6 +30,25 @@ import { import { pluginRegistryService } from "./plugin-registry.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; +/** + * The worker methods a sandbox provider must advertise before the host reuses + * a provider lease across runs. The host resumes a lease through + * `environmentResumeLease`, ends it through `environmentReleaseLease`, and tears + * down a stale lease through `environmentDestroyLease`. The reuse path destroys + * the stale lease when a resume fails and then acquires a fresh lease, so a + * provider that omits any of the three methods can strand the stale lease and + * can never complete the reuse path. + * + * The runtime capability normalizer maps `reusableLeases` to these same methods, + * so the acquisition guard, the effective-capability snapshot, and the published + * provider-capabilities value all read one source and cannot drift. + */ +export const REUSABLE_LEASE_WORKER_METHODS = [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", +] as const; + export interface ReadyPluginWorkerRecovery { pluginKeys: readonly string[]; startWorker(plugin: { id: string; pluginKey: string }): Promise; @@ -44,6 +63,15 @@ export interface ReadyPluginEnvironmentDriver { description?: string; configSchema: PluginEnvironmentDriverDeclaration["configSchema"]; supportsReusableLeases?: PluginEnvironmentDriverDeclaration["supportsReusableLeases"]; + sandboxCapabilities?: PluginEnvironmentDriverDeclaration["sandboxCapabilities"]; + /** + * The running worker for this exact plugin advertises ALL reusable-lease + * lifecycle methods (`REUSABLE_LEASE_WORKER_METHODS`: resume, release, and + * destroy). The published provider-capabilities value grants reusable leases + * only when the declaration allows them AND this flag is true, so presentation + * matches the acquisition guard, which also verifies the same methods live. + */ + reusableLeaseMethodsVerified: boolean; supportsInteractiveSetup?: PluginEnvironmentDriverDeclaration["supportsInteractiveSetup"]; interactiveSetupConnectionTypes?: PluginEnvironmentDriverDeclaration["interactiveSetupConnectionTypes"]; supportsTemplateCapture?: PluginEnvironmentDriverDeclaration["supportsTemplateCapture"]; @@ -130,6 +158,35 @@ export async function resolvePluginSandboxProviderDriverByKey(input: { return null; } +/** + * Resolve the sandbox-provider driver declaration from one exact plugin id. + * + * A driver key is only unique inside a single manifest. Two installed plugins + * can declare the same driver key. A lease pins the plugin that acquired it + * through `metadata.pluginId`. Use this resolver, not the by-key resolver, when + * the caller must read the declaration from that exact plugin. The by-key + * resolver returns the first installed plugin with the key, which can be a + * different, even disabled, plugin. + * + * This resolver fails closed. It returns `null` when the plugin id is unknown, + * or when that plugin no longer declares a `sandbox_provider` driver with the + * given key. + */ +export async function resolvePluginSandboxProviderDriverById(input: { + db: Db; + pluginId: string; + driverKey: string; +}): Promise<{ plugin: Awaited["getById"]>>; driver: PluginEnvironmentDriverDeclaration } | null> { + const pluginRegistry = pluginRegistryService(input.db); + const plugin = await pluginRegistry.getById(input.pluginId); + if (!plugin) return null; + const driver = plugin.manifestJson.environmentDrivers?.find( + (candidate) => candidate.driverKey === input.driverKey && candidate.kind === "sandbox_provider", + ) as PluginEnvironmentDriverDeclaration | undefined; + if (!driver) return null; + return { plugin, driver }; +} + export async function listReadyPluginEnvironmentDrivers(input: { db: Db; workerManager?: PluginWorkerManager; @@ -173,6 +230,13 @@ export async function listReadyPluginEnvironmentDrivers(input: { if (!input.workerManager.isRunning(plugin.id)) { continue; } + // The plugin is running, so read the live worker's verified methods once per + // plugin. A provider advertises reusable leases only when its worker carries + // all reuse lifecycle methods; the declaration alone never grants them. + const workerMethods = new Set(input.workerManager.getWorker(plugin.id)?.supportedMethods ?? []); + const reusableLeaseMethodsVerified = REUSABLE_LEASE_WORKER_METHODS.every( + (method) => workerMethods.has(method), + ); rows.push( ...(plugin.manifestJson.environmentDrivers ?? []) .filter((driver) => driver.kind === "sandbox_provider") @@ -184,6 +248,8 @@ export async function listReadyPluginEnvironmentDrivers(input: { description: driver.description, configSchema: driver.configSchema, supportsReusableLeases: driver.supportsReusableLeases, + sandboxCapabilities: driver.sandboxCapabilities, + reusableLeaseMethodsVerified, supportsInteractiveSetup: driver.supportsInteractiveSetup, interactiveSetupConnectionTypes: driver.interactiveSetupConnectionTypes, supportsTemplateCapture: driver.supportsTemplateCapture,