diff --git a/doc/architecture/paperclip-runner-daytona-networking.md b/doc/architecture/paperclip-runner-daytona-networking.md new file mode 100644 index 0000000000..a2a3d72945 --- /dev/null +++ b/doc/architecture/paperclip-runner-daytona-networking.md @@ -0,0 +1,41 @@ +# Paperclip Runner in Daytona sandboxes + +## Intended topology + +Paperclip creates the run and a short-lived, one-use runner bootstrap ticket. The Daytona sandbox starts only `paperclip-runnerd`; runnerd then starts the configured provider (Codex initially). Runnerd listens on a fixed sandbox-local port and Paperclip connects through Daytona's authenticated preview WebSocket ingress. The provider never receives a Paperclip API credential or Daytona preview credential. + +```text +Paperclip control plane + | + | outbound WSS + X-Daytona-Preview-Token + v +Daytona preview proxy -> paperclip-runnerd:43127 -> Codex app-server +``` + +The PRP identity binds company, issue, agent, run, environment lease, runner instance, normalized session, artifact version, artifact digest, and catalog digest. After the one-use ticket challenge succeeds, runnerd receives a renewable connection lease. Revocation, expiry, replay cursors, and the durable outbox continue to work across transient network loss and sandbox/provider restarts. + +## Network policy + +- Allow Paperclip to make outbound TCP 443 connections to the configured Daytona preview origin. The Paperclip host does not need public ingress. +- Allow the sandbox provider's private preview proxy to reach runnerd on TCP 43127. Do not expose the port with a public sandbox or signed URL. +- Allow the sandbox outbound access only to destinations explicitly required by the provider runtime. +- Use Daytona's `wss://` preview URL with normal certificate and hostname validation. The preview token is sent only as the `X-Daytona-Preview-Token` header by Paperclip. +- Do not put the bootstrap ticket in argv, files, provider environment, logs, or model context. Inject it into runnerd's initial environment/secret channel; runnerd already removes it from its environment immediately. +- Runnerd is the only process allowed to reach PRP. The provider communicates with runnerd over inherited pipes. + +## Lifecycle + +1. Paperclip allocates the Daytona environment and persists its environment lease. +2. Paperclip creates a runner ticket with a very short expiry and binds it to the run, environment lease, runner digest/version, and allowed catalog digest. +3. The sandbox startup command launches the pinned runnerd artifact in listener mode on `0.0.0.0:43127` and the run-specific PRP path. The ticket is supplied separately as secret environment material. +4. Paperclip acquires an authenticated Daytona preview endpoint and dials runnerd. After PRP authentication, Paperclip sends `run.prepare`; only then may runnerd start Codex and advertise the run-authorized tools. +5. Suspend/drain/revoke commands stop new turns and durably flush terminal events. Paperclip revokes the lease before destroying or recycling the sandbox. +6. A recovered sandbox reuses its durable runner state and an unexpired connection lease; it must not mint a second provider session when a resumable one exists. + +## Deployment-mode boundary + +Daytona always selects provider ingress for `paperclip_runner`; it does not fall +back to direct outbound WSS or a legacy callback bridge. Same-host native runs +keep loopback `ws://`, and other remote providers may use direct outbound +`wss://` only when an operator explicitly configures a reachable Paperclip +runner URL. Legacy adapters retain their existing transport paths. diff --git a/docs/deploy/environment-variables.md b/docs/deploy/environment-variables.md index 20bbdd332d..baed34649b 100644 --- a/docs/deploy/environment-variables.md +++ b/docs/deploy/environment-variables.md @@ -19,9 +19,41 @@ All environment variables that Paperclip uses for server configuration. | `PAPERCLIP_DEPLOYMENT_MODE` | `local_trusted` | Runtime mode override | | `PAPERCLIP_DEPLOYMENT_EXPOSURE` | `private` | Exposure policy when deployment mode is `authenticated` | | `PAPERCLIP_API_URL` | (auto-derived) | Paperclip API base URL. When set externally (e.g., via Kubernetes ConfigMap, load balancer, or reverse proxy), the server preserves the value instead of deriving it from the listen host and port. Useful for deployments where the public-facing URL differs from the local bind address. | +| `PAPERCLIP_RUNNER_PUBLIC_URL` | (unset) | Explicit `wss://` base URL used only when a remote `paperclip_runner` target dials Paperclip directly. Paperclip appends `/api/runner/v1/connect/`; the reverse proxy must forward WebSocket upgrades for that route. This value is never inferred from request headers. Daytona ignores it and uses provider ingress. | +| `PAPERCLIP_RUNNER_CA_BUNDLE_PATH` | (unset) | Optional PEM CA bundle for direct runner WSS. Platform roots remain enabled. There is no insecure TLS bypass. | +| `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` | (host build) | Host-local path to a `paperclip-runnerd` artifact built for the remote target OS and architecture. Required when Paperclip and the remote sandbox do not share a compatible platform; build metadata and the required transport mode are verified before launch. | +| `PAPERCLIP_RUNNER_REMOTE_CODEX_PATH` | (unset) | Optional host-local path to a Codex executable built for the remote target OS and architecture. For remote Codex-backed runners, Paperclip stages and verifies this executable beside `paperclip-runnerd`. | +| `PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC` | (unset) | Optional pinned npm package spec (for example, `@openai/codex@0.148.0`) installed inside each fresh remote lease when its Codex harness is not baked into the sandbox image. Mutually exclusive with `PAPERCLIP_RUNNER_REMOTE_CODEX_PATH`; Paperclip verifies the installed executable before starting `runnerd`. | +| `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` | (unset) | Host-local path to the immutable provider pack built by `pnpm --filter @paperclipai/paperclip-runner build:provider-pack`. The pack includes its target-built Node 24.11 runtime, locked production dependencies, OpenCode proxy/executable, and ACPX sidecar. Remote OpenCode and ACPX fail closed without it. A preinstalled pack is accepted only when its complete digested manifest matches this build-owned pack; otherwise Paperclip stages this pack into the sandbox. | | `PAPERCLIP_HIDDEN_SETTINGS` | (unset) | Comma-separated settings surfaces to hide from the UI and floor at the API, for operators hosting Paperclip for others (managed cloud, internal shared server). See [Hiding settings surfaces](#hiding-settings-surfaces). | | `PAPERCLIP_SETTING_DEFAULTS` | (unset) | JSON object replacing the schema default of selected instance settings, for hosting operators. See [Operator setting defaults](#operator-setting-defaults). | +Daytona connectivity for `paperclip_runner` uses authenticated preview WSS and +is gated by the instance experimental setting `enableRunnerPreviewIngress` +(default `false`). The setting has no effect on legacy adapters or callback +bridges. + +### Preinstalled remote runner images + +Remote sandbox images may preinstall `paperclip-runnerd`, `codex`, and the +provider pack at `/opt/paperclip-runner/provider-pack` instead of +paying the upload and npm-install cost on every fresh lease. Put both executable +names on the sandbox user's `PATH`; `$HOME/.local/bin` is checked explicitly +before `PATH`. Paperclip verifies runner build metadata, the selected PRP +transport capability, Codex startup, the provider-pack digest, exact harness +pins, Node compatibility, and packaged bridge digests before linking artifacts +into the run-specific runtime directory. A missing or incompatible executable falls back +to `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and +`PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC` (or +`PAPERCLIP_RUNNER_REMOTE_CODEX_PATH`) without changing the selected transport. +OpenCode and ACPX instead fall back only to +`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH`; they never start a provider +process on the Paperclip host for a remote target. +The Daytona environment editor's **Configure image** action can create this +image without a separate container registry: install the executables in its +setup sandbox, finish setup, and Paperclip captures and promotes the resulting +Daytona snapshot for future leases. + ### Hiding settings surfaces `PAPERCLIP_HIDDEN_SETTINGS` takes keys from the registry in diff --git a/packages/adapter-utils/src/execution-target-sandbox.test.ts b/packages/adapter-utils/src/execution-target-sandbox.test.ts index 47e7a276e4..0b3c5134af 100644 --- a/packages/adapter-utils/src/execution-target-sandbox.test.ts +++ b/packages/adapter-utils/src/execution-target-sandbox.test.ts @@ -2965,6 +2965,7 @@ describe("sandbox adapter execution targets", () => { incrementalSessionOutput: false, concurrentSyncOperations: false, duplexCommandStream, + runnerWebSocketIngress: false, }; } @@ -6364,6 +6365,7 @@ describe("EffectiveSandboxCapabilities deprecated alias", () => { incrementalSessionOutput: false, concurrentSyncOperations: false, duplexCommandStream: false, + runnerWebSocketIngress: false, }; const aliased: EffectiveSandboxCapabilities = snapshot; expect(aliased).toEqual(snapshot); diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index 845c65aed8..db0fed0d72 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -80,6 +80,7 @@ import { } from "./acpx-engine/startup-timing.js"; import type { RuntimeProgressSink, RuntimeStatusSink } from "./runtime-progress.js"; import type { LocalProcessSandboxOptions } from "./local-process-sandbox.js"; +import type { RunnerIngressEndpoint } from "./runner-connectivity.js"; export type { RuntimeProgressSink } from "./runtime-progress.js"; @@ -145,6 +146,8 @@ export interface EffectiveExecutionCapabilities { readonly incrementalSessionOutput: boolean; readonly concurrentSyncOperations: boolean; readonly duplexCommandStream: boolean; + /** Provider can expose a private authenticated WebSocket endpoint for runnerd. */ + readonly runnerWebSocketIngress: boolean; } /** @@ -153,6 +156,13 @@ export interface EffectiveExecutionCapabilities { */ export interface EffectiveSandboxCapabilities extends EffectiveExecutionCapabilities {} +export interface SandboxLeaseAcquisition { + outcome: "created" | "resumed" | "replacement"; + providerLeaseId: string; + previousProviderLeaseId?: string; + reason?: "not_found" | "expired" | "identity_mismatch" | "resume_failed"; +} + export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWorkspaceMetadata { kind: "remote"; transport: "sandbox"; @@ -171,12 +181,27 @@ export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWor * environment. Absent means no grant. */ readonly enableSandboxDuplexBridge?: boolean; + /** Host-owned lifecycle override for paperclip_runner in this environment. */ + readonly runnerLifecyclePolicy?: + | { mode: "per_turn"; idleTimeoutMs: null } + | { mode: "warm"; idleTimeoutMs: number } + | null; + /** Whether this environment is configured to reuse its provider lease. */ + readonly reusableLeaseConfigured?: boolean; + /** Host-observed provenance for this exact sandbox acquisition. */ + readonly sandboxLeaseAcquisition?: SandboxLeaseAcquisition | null; shellCommand?: "bash" | "sh" | null; environmentId?: string | null; leaseId?: string | null; remoteCwd: string; timeoutMs?: number | null; runner?: CommandManagedRuntimeRunner; + /** Host-only provider operation. It is never serialized into the sandbox. */ + getRunnerIngressEndpoint?: (input: { + leaseId: string; + port: number; + path: string; + }) => Promise; /** * Sandbox-backed adapter runs stream the agent CLI's stdout/stderr * incrementally via a log-tail loop beside the callback bridge instead of @@ -346,6 +371,7 @@ function parseEffectiveExecutionCapabilities(value: unknown): EffectiveExecution incrementalSessionOutput: parsed.incrementalSessionOutput === true, concurrentSyncOperations: parsed.concurrentSyncOperations === true, duplexCommandStream: parsed.duplexCommandStream === true, + runnerWebSocketIngress: parsed.runnerWebSocketIngress === true, }; } diff --git a/packages/adapter-utils/src/index.ts b/packages/adapter-utils/src/index.ts index 5d1979c826..987a4e156d 100644 --- a/packages/adapter-utils/src/index.ts +++ b/packages/adapter-utils/src/index.ts @@ -119,8 +119,22 @@ export { isPaperclipRunnerProvider, resolvePaperclipRunnerPermissionMode, } from "./paperclip-runner-permissions.js"; +export { + PAPERCLIP_RUNNER_INGRESS_PORT, + PAPERCLIP_RUNNER_CONNECT_PATH_PREFIX, + PaperclipRunnerTransportError, + buildDirectRunnerConnectUrl, + resolvePaperclipRunnerTransport, +} from "./runner-connectivity.js"; export type { + SecretHeader, + RunnerIngressEndpoint, + PaperclipRunnerTransport, +} from "./runner-connectivity.js"; +export type { + AcpxPermissionMode, CodexPermissionMode, + OpenCodePermissionMode, PaperclipRunnerPermissionCapability, PaperclipRunnerPermissionMode, PaperclipRunnerPermissionOption, diff --git a/packages/adapter-utils/src/paperclip-runner-permissions.ts b/packages/adapter-utils/src/paperclip-runner-permissions.ts index 40f2bc3638..51f169fc8f 100644 --- a/packages/adapter-utils/src/paperclip-runner-permissions.ts +++ b/packages/adapter-utils/src/paperclip-runner-permissions.ts @@ -1,7 +1,16 @@ -export type PaperclipRunnerProvider = "codex"; +export type PaperclipRunnerProvider = + | "codex" + | "opencode" + | "acpx"; export type CodexPermissionMode = "never" | "on-request" | "untrusted"; -export type PaperclipRunnerPermissionMode = CodexPermissionMode; +export type OpenCodePermissionMode = "allow" | "ask" | "deny"; +export type AcpxPermissionMode = "approve-all" | "approve-reads" | "deny-all"; + +export type PaperclipRunnerPermissionMode = + | CodexPermissionMode + | OpenCodePermissionMode + | AcpxPermissionMode; export interface PaperclipRunnerPermissionOption { value: TMode; @@ -11,7 +20,7 @@ export interface PaperclipRunnerPermissionOption export interface PaperclipRunnerPermissionCapability { configurable: true; - configKey: "codexPermissionMode"; + configKey: "codexPermissionMode" | "opencodePermissionMode" | "acpxPermissionMode"; defaultMode: PaperclipRunnerPermissionMode; options: readonly PaperclipRunnerPermissionOption[]; description: string; @@ -34,6 +43,28 @@ export const PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES = { { value: "untrusted", label: "Ask for untrusted operations", description: "Prompt for operations Codex does not classify as trusted." }, ], }, + opencode: { + configurable: true, + configKey: "opencodePermissionMode", + defaultMode: "allow", + description: "Controls OpenCode tool permissions inside the assigned Paperclip environment.", + options: [ + { value: "allow", label: "Full auto (allow)", description: "Allow OpenCode operations without approval pauses." }, + { value: "ask", label: "Ask for permission", description: "Prompt before protected OpenCode operations." }, + { value: "deny", label: "Deny operations", description: "Reject protected OpenCode operations." }, + ], + }, + acpx: { + configurable: true, + configKey: "acpxPermissionMode", + defaultMode: "approve-all", + description: "Controls ACPX agent operations inside the assigned Paperclip environment.", + options: [ + { value: "approve-all", label: "Full auto (approve all)", description: "Approve ACPX operations without approval pauses." }, + { value: "approve-reads", label: "Ask for mutations", description: "Approve reads and prompt for writes, edits, and execution." }, + { value: "deny-all", label: "Deny all", description: "Reject harness permission requests." }, + ], + }, } as const satisfies Record; export function isPaperclipRunnerProvider(value: unknown): value is PaperclipRunnerProvider { diff --git a/packages/adapter-utils/src/runner-connectivity.test.ts b/packages/adapter-utils/src/runner-connectivity.test.ts new file mode 100644 index 0000000000..5d825559bc --- /dev/null +++ b/packages/adapter-utils/src/runner-connectivity.test.ts @@ -0,0 +1,182 @@ +import { describe, expect, it, vi } from "vitest"; +import type { AdapterExecutionTarget } from "./execution-target.js"; +import { + buildDirectRunnerConnectUrl, + resolvePaperclipRunnerTransport, + type RunnerIngressEndpoint, +} from "./runner-connectivity.js"; + +const capabilities = { + reusableLeases: false, + nativeSyncIn: false, + nativeSyncOut: false, + persistentProcessSessions: true, + independentControlCommands: true, + incrementalSessionOutput: true, + concurrentSyncOperations: false, + duplexCommandStream: false, + runnerWebSocketIngress: true, +} as const; + +function ingress(): RunnerIngressEndpoint { + const endpoint: RunnerIngressEndpoint = { + kind: "authenticated_websocket", + websocketUrl: + "wss://43127-sandbox.proxy.daytona.test/api/runner/v1/connect/00000000-0000-4000-8000-000000000001", + secretHeaders: [{ name: "X-Daytona-Preview-Token", value: "secret" }], + generation: "generation-1", + refresh: async () => endpoint, + close: async () => undefined, + }; + return endpoint; +} + +describe("paperclip runner transport routing", () => { + it("keeps same-host runnerd on plaintext loopback", async () => { + const result = await resolvePaperclipRunnerTransport({ + target: { kind: "local" }, + runId: "00000000-0000-4000-8000-000000000001", + localConnectUrl: + "ws://127.0.0.1:3100/api/runner/v1/connect/00000000-0000-4000-8000-000000000001", + enableRunnerPreviewIngress: false, + }); + expect(result.mode).toBe("local_loopback"); + }); + + it("selects provider ingress for Daytona-style capability even when a public URL exists", async () => { + const getRunnerIngressEndpoint = vi.fn(async () => ingress()); + const target: AdapterExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "daytona", + remoteCwd: "/workspace", + leaseId: "lease-1", + effectiveCapabilities: capabilities, + getRunnerIngressEndpoint, + }; + const result = await resolvePaperclipRunnerTransport({ + target, + runId: "00000000-0000-4000-8000-000000000001", + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: "wss://paperclip.example.test", + enableRunnerPreviewIngress: true, + }); + expect(result.mode).toBe("provider_ingress"); + expect(getRunnerIngressEndpoint).toHaveBeenCalledOnce(); + }); + + it("does not request preview ingress while the new-runner rollout flag is off", async () => { + const getRunnerIngressEndpoint = vi.fn(async () => ingress()); + const target: AdapterExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "daytona", + remoteCwd: "/workspace", + leaseId: "lease-1", + effectiveCapabilities: capabilities, + getRunnerIngressEndpoint, + }; + await expect( + resolvePaperclipRunnerTransport({ + target, + runId: "00000000-0000-4000-8000-000000000001", + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: "wss://paperclip.example.test", + enableRunnerPreviewIngress: false, + }), + ).rejects.toMatchObject({ code: "runner_ingress_unavailable" }); + expect(getRunnerIngressEndpoint).not.toHaveBeenCalled(); + }); + + it("selects direct WSS only for a remote target with an explicit URL", async () => { + const target: AdapterExecutionTarget = { + kind: "remote", + transport: "ssh", + remoteCwd: "/workspace", + spec: { + host: "runner.internal", + port: 22, + username: "runner", + remoteWorkspacePath: "/workspace", + remoteCwd: "/workspace", + privateKey: null, + knownHosts: null, + strictHostKeyChecking: true, + }, + }; + const result = await resolvePaperclipRunnerTransport({ + target, + runId: "00000000-0000-4000-8000-000000000001", + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: "wss://paperclip.example.test/runner-base/", + runnerCaBundlePath: "/etc/paperclip/runner-ca.pem", + enableRunnerPreviewIngress: false, + }); + expect(result).toEqual({ + mode: "direct_outbound", + connectUrl: + "wss://paperclip.example.test/runner-base/api/runner/v1/connect/00000000-0000-4000-8000-000000000001", + caBundlePath: "/etc/paperclip/runner-ca.pem", + }); + }); + + it("fails the selected ingress mode without falling through to direct WSS", async () => { + const target: AdapterExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "daytona", + remoteCwd: "/workspace", + leaseId: "lease-1", + effectiveCapabilities: capabilities, + getRunnerIngressEndpoint: async () => { + throw new Error("preview unavailable"); + }, + }; + await expect( + resolvePaperclipRunnerTransport({ + target, + runId: "00000000-0000-4000-8000-000000000001", + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: "wss://paperclip.example.test", + enableRunnerPreviewIngress: true, + }), + ).rejects.toThrow("preview unavailable"); + }); + + it("never routes Daytona through direct outbound when ingress capability is unavailable", async () => { + const target: AdapterExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "daytona", + remoteCwd: "/workspace", + leaseId: "lease-1", + effectiveCapabilities: { + ...capabilities, + runnerWebSocketIngress: false, + }, + }; + await expect( + resolvePaperclipRunnerTransport({ + target, + runId: "00000000-0000-4000-8000-000000000001", + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: "wss://paperclip.example.test", + enableRunnerPreviewIngress: true, + }), + ).rejects.toMatchObject({ code: "runner_ingress_unavailable" }); + }); + + it.each([ + "ws://paperclip.example.test", + "wss://user@paperclip.example.test", + "wss://paperclip.example.test?token=secret", + "wss://paperclip.example.test#fragment", + ])("rejects unsafe direct runner URL %s", (runnerPublicUrl) => { + expect(() => + buildDirectRunnerConnectUrl({ + runnerPublicUrl, + runId: "00000000-0000-4000-8000-000000000001", + }), + ).toThrow(); + }); +}); diff --git a/packages/adapter-utils/src/runner-connectivity.ts b/packages/adapter-utils/src/runner-connectivity.ts new file mode 100644 index 0000000000..c50985acd6 --- /dev/null +++ b/packages/adapter-utils/src/runner-connectivity.ts @@ -0,0 +1,173 @@ +import type { AdapterExecutionTarget } from "./execution-target.js"; + +export const PAPERCLIP_RUNNER_INGRESS_PORT = 43_127; +export const PAPERCLIP_RUNNER_CONNECT_PATH_PREFIX = "/api/runner/v1/connect"; + +export interface SecretHeader { + readonly name: string; + readonly value: string; +} + +export interface RunnerIngressEndpoint { + readonly kind: "authenticated_websocket"; + readonly websocketUrl: string; + readonly secretHeaders: readonly SecretHeader[]; + readonly generation: string; + refresh(): Promise; + close(): Promise; +} + +export type PaperclipRunnerTransport = + | { + readonly mode: "local_loopback"; + readonly connectUrl: string; + } + | { + readonly mode: "direct_outbound"; + readonly connectUrl: string; + readonly caBundlePath?: string; + } + | { + readonly mode: "provider_ingress"; + readonly listenAddress: "0.0.0.0"; + readonly listenPort: number; + readonly listenPath: string; + readonly ingress: RunnerIngressEndpoint; + }; + +export class PaperclipRunnerTransportError extends Error { + readonly code: + | "runner_transport_ineligible" + | "runner_direct_wss_failed" + | "runner_ingress_unavailable"; + + constructor( + code: PaperclipRunnerTransportError["code"], + message: string, + options?: ErrorOptions, + ) { + super(`${code}: ${message}`, options); + this.name = "PaperclipRunnerTransportError"; + this.code = code; + } +} + +function connectPath(runId: string): string { + if (!runId || runId.includes("/") || runId.includes("?") || runId.includes("#")) { + throw new PaperclipRunnerTransportError( + "runner_transport_ineligible", + "Runner run id is not safe for a WebSocket route.", + ); + } + return `${PAPERCLIP_RUNNER_CONNECT_PATH_PREFIX}/${encodeURIComponent(runId)}`; +} + +export function buildDirectRunnerConnectUrl(input: { + runnerPublicUrl: string; + runId: string; +}): string { + let url: URL; + try { + url = new URL(input.runnerPublicUrl); + } catch (error) { + throw new PaperclipRunnerTransportError( + "runner_direct_wss_failed", + "The configured runner public URL is invalid.", + { cause: error }, + ); + } + if (url.protocol !== "wss:") { + throw new PaperclipRunnerTransportError( + "runner_direct_wss_failed", + "Remote runner connectivity requires a wss: runner public URL.", + ); + } + if (url.username || url.password || url.search || url.hash) { + throw new PaperclipRunnerTransportError( + "runner_direct_wss_failed", + "Runner public URLs cannot contain userinfo, a query string, or a fragment.", + ); + } + url.pathname = `${url.pathname.replace(/\/$/, "")}${connectPath(input.runId)}`; + return url.toString(); +} + +export async function resolvePaperclipRunnerTransport(input: { + target: AdapterExecutionTarget; + runId: string; + localConnectUrl: string; + runnerPublicUrl?: string | null; + runnerCaBundlePath?: string | null; + enableRunnerPreviewIngress: boolean; + getRunnerIngressEndpoint?: (input: { + leaseId: string; + port: number; + path: string; + }) => Promise; +}): Promise { + if (input.target.kind === "local") { + return { mode: "local_loopback", connectUrl: input.localConnectUrl }; + } + + if ( + input.target.transport === "sandbox" && + input.target.providerKey === "daytona" && + input.target.effectiveCapabilities?.runnerWebSocketIngress !== true + ) { + throw new PaperclipRunnerTransportError( + "runner_ingress_unavailable", + "Daytona runner execution requires provider WebSocket ingress capability.", + ); + } + + if ( + input.target.transport === "sandbox" && + input.target.effectiveCapabilities?.runnerWebSocketIngress === true + ) { + if (!input.enableRunnerPreviewIngress) { + throw new PaperclipRunnerTransportError( + "runner_ingress_unavailable", + "Runner preview ingress is disabled for this Paperclip instance.", + ); + } + const getRunnerIngressEndpoint = + input.getRunnerIngressEndpoint ?? input.target.getRunnerIngressEndpoint; + if (!input.target.leaseId || !getRunnerIngressEndpoint) { + throw new PaperclipRunnerTransportError( + "runner_ingress_unavailable", + "The sandbox runner ingress provider is unavailable for this lease.", + ); + } + const path = connectPath(input.runId); + const ingress = await getRunnerIngressEndpoint({ + leaseId: input.target.leaseId, + port: PAPERCLIP_RUNNER_INGRESS_PORT, + path, + }); + return { + mode: "provider_ingress", + listenAddress: "0.0.0.0", + listenPort: PAPERCLIP_RUNNER_INGRESS_PORT, + listenPath: path, + ingress, + }; + } + + if (input.runnerPublicUrl) { + return { + mode: "direct_outbound", + connectUrl: buildDirectRunnerConnectUrl({ + runnerPublicUrl: input.runnerPublicUrl, + runId: input.runId, + }), + ...(input.runnerCaBundlePath + ? { caBundlePath: input.runnerCaBundlePath } + : {}), + }; + } + + throw new PaperclipRunnerTransportError( + "runner_transport_ineligible", + "The remote execution target has neither runner WebSocket ingress nor an explicit runner public URL.", + ); +} diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 8b01e49023..b0f4e16ee9 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -741,34 +741,35 @@ export interface DurablePrpControlPlaneStore { readonly state: StoredCoreState; } -class PrpWebSocketConnection { +class RawWebSocketWireConnection implements PrpWireConnection { readonly socket: Duplex; - pendingChallenge: PendingChallenge | null = null; - secureChannel: SecureChannel | null = null; - lease: ConnectionLeaseRecord | null = null; - connectionId: string | null = null; #buffer = Buffer.alloc(0); #closed = false; - #onText: (text: string) => void | Promise; - #onClose: () => void; - #processing = Promise.resolve(); + #onJson: (value: unknown) => void = () => undefined; + #onClose: (reason: TransportCloseReason) => void = () => undefined; - constructor( - socket: Duplex, - onText: (text: string) => void | Promise, - onClose: () => void, - ) { + constructor(socket: Duplex) { this.socket = socket; - this.#onText = onText; - this.#onClose = onClose; socket.on("data", (chunk: Buffer) => this.#consume(chunk)); socket.on("close", () => { if (!this.#closed) { this.#closed = true; - this.#onClose(); + this.#onClose({ message: "socket_closed" }); } }); - socket.on("error", () => this.close()); + socket.on("error", (error) => { + if (this.#closed) return; + this.#closed = true; + this.#onClose({ message: "socket_error", error }); + }); + } + + onJson(listener: (value: unknown) => void): void { + this.#onJson = listener; + } + + onClose(listener: (reason: TransportCloseReason) => void): void { + this.#onClose = listener; } acceptInitialData(data: Buffer): void { @@ -776,11 +777,7 @@ class PrpWebSocketConnection { } sendJson(value: unknown): void { - const wire = - this.secureChannel === null - ? value - : encryptSecureJson(this.secureChannel, value); - this.sendText(JSON.stringify(wire)); + this.sendText(JSON.stringify(value)); } sendText(text: string): void { @@ -803,13 +800,13 @@ class PrpWebSocketConnection { this.socket.write(Buffer.concat([Buffer.from(header), payload])); } - close(): void { + close(_code?: number): void { if (this.#closed) { return; } this.#closed = true; this.socket.destroy(); - this.#onClose(); + this.#onClose({ message: "local_close" }); } #consume(chunk: Buffer): void { @@ -850,10 +847,14 @@ class PrpWebSocketConnection { payload[index] = payload[index]! ^ mask[index % 4]!; } if (opcode === 0x1) { - const text = payload.toString("utf8"); - this.#processing = this.#processing - .then(() => this.#onText(text)) - .catch(() => this.close()); + try { + this.#onJson(JSON.parse(payload.toString("utf8")) as unknown); + } catch (error) { + this.#closed = true; + this.socket.destroy(); + this.#onClose({ message: "invalid_json", error }); + return; + } } else if (opcode === 0x8) { this.close(); return; @@ -874,6 +875,49 @@ class PrpWebSocketConnection { } } +class AuthorityConnection { + pendingChallenge: PendingChallenge | null = null; + secureChannel: SecureChannel | null = null; + lease: ConnectionLeaseRecord | null = null; + connectionId: string | null = null; + readonly wire: PrpWireConnection; + #closed = false; + #onClose: () => void; + + constructor(input: { + wire: PrpWireConnection; + onJson: (value: unknown) => void; + onClose: () => void; + }) { + this.wire = input.wire; + this.#onClose = input.onClose; + this.wire.onJson(input.onJson); + this.wire.onClose(() => this.#markClosed()); + } + + sendJson(value: unknown): void { + if (this.#closed) return; + this.wire.sendJson( + this.secureChannel === null + ? value + : encryptSecureJson(this.secureChannel, value), + ); + } + + close(code?: number): void { + if (this.#closed) return; + this.#closed = true; + this.wire.close(code); + this.#onClose(); + } + + #markClosed(): void { + if (this.#closed) return; + this.#closed = true; + this.#onClose(); + } +} + /** Authenticated, replay-safe PRP transport authority. Business operations are caller supplied. */ export class DurablePrpControlPlane { readonly #identity: DurableRecoveryIdentity; @@ -881,7 +925,7 @@ export class DurablePrpControlPlane { #expectedRunnerVersion: string; #expectedRunnerDigest: string; #server: Server | null = null; - #connections = new Set(); + #connections = new Set(); #pendingSemanticCalls = new Set(); #port: number | null = null; #onSemanticToolInput?: DurablePrpControlPlaneOptions["onSemanticToolInput"]; @@ -1110,23 +1154,36 @@ export class DurablePrpControlPlane { "\r\n", ].join("\r\n"), ); - let connection!: PrpWebSocketConnection; - connection = new PrpWebSocketConnection( - socket, - (text): Promise => this.#handleText(connection, text), - () => this.#connections.delete(connection), - ); - this.#connections.add(connection); - connection.acceptInitialData(head); + const wire = new RawWebSocketWireConnection(socket); + this.attachWireConnection(wire); + wire.acceptInitialData(head); } - async #handleText( - connection: PrpWebSocketConnection, - text: string, + /** Attach either an accepted inbound WebSocket or a Paperclip-opened peer. */ + attachWireConnection(wire: PrpWireConnection): PrpWireAttachment { + let connection!: AuthorityConnection; + let processing = Promise.resolve(); + connection = new AuthorityConnection({ + wire, + onJson: (value) => { + processing = processing + .then(() => this.#handleJson(connection, value)) + .catch(() => connection.close()); + }, + onClose: () => this.#connections.delete(connection), + }); + this.#connections.add(connection); + return { + isAuthenticated: () => connection.secureChannel !== null, + }; + } + + async #handleJson( + connection: AuthorityConnection, + wire: unknown, ): Promise { let envelope: Record; try { - const wire = JSON.parse(text) as unknown; envelope = connection.secureChannel === null ? (wire as Record) @@ -1306,7 +1363,7 @@ export class DurablePrpControlPlane { } #authHello( - connection: PrpWebSocketConnection, + connection: AuthorityConnection, envelope: Record, ): void { if (connection.pendingChallenge !== null) { @@ -1371,7 +1428,7 @@ export class DurablePrpControlPlane { } #authResponse( - connection: PrpWebSocketConnection, + connection: AuthorityConnection, envelope: Record, ): void { const pending = connection.pendingChallenge; @@ -1451,7 +1508,7 @@ export class DurablePrpControlPlane { } #welcome( - connection: PrpWebSocketConnection, + connection: AuthorityConnection, leaseToken: string | null, ): void { const lease = connection.lease; @@ -1522,7 +1579,7 @@ export class DurablePrpControlPlane { } #controlEnvelope( - connection: PrpWebSocketConnection, + connection: AuthorityConnection, envelopeId: string, kind: string, payload: Record, @@ -1550,7 +1607,7 @@ export class DurablePrpControlPlane { }; } - #sendNextCommand(connection: PrpWebSocketConnection): void { + #sendNextCommand(connection: AuthorityConnection): void { const [command] = this.#nextPendingCommand(); if (command === undefined) return; this.#store.state.commandDeliveryCounts[command.commandId] = @@ -1567,7 +1624,7 @@ export class DurablePrpControlPlane { } #commandResult( - connection: PrpWebSocketConnection, + connection: AuthorityConnection, envelope: Record, ): void { const result = envelope.payload as Record | undefined; @@ -1609,7 +1666,7 @@ export class DurablePrpControlPlane { } async #event( - connection: PrpWebSocketConnection, + connection: AuthorityConnection, envelope: Record, ): Promise { const validated = validatePrpEvent(envelope.payload); diff --git a/packages/plugins/sandbox-providers/daytona/src/manifest.ts b/packages/plugins/sandbox-providers/daytona/src/manifest.ts index 89ae0af1b6..3eced73a22 100644 --- a/packages/plugins/sandbox-providers/daytona/src/manifest.ts +++ b/packages/plugins/sandbox-providers/daytona/src/manifest.ts @@ -11,7 +11,9 @@ const PLUGIN_ID = "paperclip.daytona-sandbox-provider"; // neutral `supportsLoginPty`. // 0.1.4 adds the `concurrentSyncOperations` sandbox capability to the driver. // 0.1.5 adds the `duplexCommandStream` sandbox capability to the driver. -const PLUGIN_VERSION = "0.1.5"; +// 0.1.6 adds private authenticated WebSocket ingress for paperclip_runner. +// 0.1.7 exposes host-owned warm/cold runner lifecycle controls. +const PLUGIN_VERSION = "0.1.7"; const manifest: PaperclipPluginManifestV1 = { id: PLUGIN_ID, @@ -58,6 +60,7 @@ const manifest: PaperclipPluginManifestV1 = { incrementalSessionOutput: true, concurrentSyncOperations: true, duplexCommandStream: true, + runnerWebSocketIngress: true, }, supportsInteractiveSetup: true, interactiveSetupConnectionTypes: ["ssh"], @@ -169,6 +172,21 @@ const manifest: PaperclipPluginManifestV1 = { "Whether to stop and later resume the sandbox across runs instead of deleting it on release.", default: false, }, + runnerLifecycleMode: { + type: "string", + enum: ["inherit", "per_turn", "warm"], + description: + "paperclip_runner lifecycle for this environment. Inherit uses the agent setting; warm keeps runnerd and the sandbox available between turns.", + default: "inherit", + }, + runnerIdleTimeoutMs: { + type: "integer", + description: + "How long an idle warm paperclip_runner stays alive before it checkpoints and suspends.", + minimum: 1000, + maximum: 86400000, + default: 300000, + }, }, }, }, diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 28bef8db09..c2efc24b04 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -46,6 +46,7 @@ function createMockSandbox(overrides: { recoverable?: boolean; workDir?: string; autoDestroyAt?: string | null; + updatedAt?: string; } = {}) { return { id: overrides.id ?? "sandbox-123", @@ -57,6 +58,7 @@ function createMockSandbox(overrides: { // A configured provider TTL populates `autoDestroyAt` after `setTtl` + // `refreshData`. The default mock leaves it unset (no TTL configured). autoDestroyAt: overrides.autoDestroyAt ?? undefined, + updatedAt: overrides.updatedAt, getWorkDir: vi.fn().mockResolvedValue(overrides.workDir ?? "/home/daytona"), getUserHomeDir: vi.fn().mockResolvedValue("/home/daytona"), start: vi.fn().mockResolvedValue(undefined), @@ -75,6 +77,10 @@ function createMockSandbox(overrides: { token: "ssh-token-secret", command: "ssh ssh-token-secret@ssh.app.daytona.io", }), + getPreviewLink: vi.fn().mockResolvedValue({ + url: "https://43127-sandbox-123.proxy.daytona.test", + token: "preview-token-secret", + }), _experimental_createSnapshot: vi.fn().mockResolvedValue(undefined), fs: { createFolder: vi.fn().mockResolvedValue(undefined), @@ -161,11 +167,90 @@ describe("Daytona sandbox provider plugin", () => { expect(plugin.definition.onDuplexChannelClose).toBeTypeOf("function"); }); + it("declares and returns private authenticated runner WebSocket ingress", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockCreate.mockResolvedValue(sandbox); + const base = { + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + config: { image: "node:20", timeoutMs: 300_000, reuseLease: false }, + }; + const lease = await plugin.definition.onEnvironmentAcquireLease?.({ + ...base, + runId: "00000000-0000-4000-8000-000000000001", + }); + const endpoint = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.({ + ...base, + lease: lease!, + port: 43_127, + path: "/api/runner/v1/connect/00000000-0000-4000-8000-000000000001", + }); + + expect(manifest.environmentDrivers?.[0]?.sandboxCapabilities).toMatchObject({ + runnerWebSocketIngress: true, + }); + expect(endpoint).toMatchObject({ + kind: "authenticated_websocket", + websocketUrl: + "wss://43127-sandbox-123.proxy.daytona.test/api/runner/v1/connect/00000000-0000-4000-8000-000000000001", + secretHeaders: [ + { name: "X-Daytona-Preview-Token", value: "preview-token-secret" }, + ], + }); + expect(endpoint?.websocketUrl).not.toContain("preview-token-secret"); + expect(endpoint?.websocketUrl).not.toContain("host-key"); + }); + + it("keeps ingress generation independent of token rotation and changes it after a sandbox lifecycle revision", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox({ updatedAt: "2026-08-25T10:00:00.000Z" }); + sandbox.getPreviewLink + .mockResolvedValueOnce({ + url: "https://43127-sandbox-123.proxy.daytona.test", + token: "preview-token-1", + }) + .mockResolvedValueOnce({ + url: "https://43127-sandbox-123.proxy.daytona.test", + token: "preview-token-2", + }) + .mockResolvedValueOnce({ + url: "https://43127-sandbox-123.proxy.daytona.test", + token: "preview-token-3", + }); + mockCreate.mockResolvedValue(sandbox); + const base = { + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + config: { image: "node:20", timeoutMs: 300_000, reuseLease: false }, + }; + const lease = await plugin.definition.onEnvironmentAcquireLease?.({ + ...base, + runId: "00000000-0000-4000-8000-000000000001", + }); + const request = { + ...base, + lease: lease!, + port: 43_127, + path: "/api/runner/v1/connect/00000000-0000-4000-8000-000000000001", + }; + const first = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.(request); + const second = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.(request); + expect(second?.generation).toBe(first?.generation); + expect(second?.secretHeaders).not.toEqual(first?.secretHeaders); + + sandbox.updatedAt = "2026-08-25T10:05:00.000Z"; + const restarted = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.(request); + expect(restarted?.generation).not.toBe(first?.generation); + }); + it("bumps the plugin version so the server reconciles the stored manifest", () => { // The bundled-plugin boot reconcile refreshes the stored manifest for an // existing install only when the version changes. The duplex capability needs // the bump to reach an existing install. - expect(manifest.version).toBe("0.1.5"); + expect(manifest.version).toBe("0.1.7"); }); it("opens a duplex channel, forwards a host write, and closes it on lease release", async () => { @@ -1074,7 +1159,7 @@ describe("Daytona sandbox provider plugin", () => { }); }); - it("deletes the sandbox if resume setup throws after the sandbox starts", async () => { + it("preserves the sandbox if resume setup throws after the sandbox starts", async () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox({ id: "sandbox-resume", state: "stopped" }); sandbox.getWorkDir.mockRejectedValue(new Error("workdir lookup failed")); @@ -1094,7 +1179,7 @@ describe("Daytona sandbox provider plugin", () => { ).rejects.toThrow("workdir lookup failed"); expect(sandbox.start).toHaveBeenCalled(); - expect(sandbox.delete).toHaveBeenCalledTimes(1); + expect(sandbox.delete).not.toHaveBeenCalled(); }); it("marks missing reusable leases as expired on resume", async () => { diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index ff1c4327da..9e38e0b381 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -23,6 +23,8 @@ import type { PluginEnvironmentDestroyLeaseParams, PluginEnvironmentExecuteParams, PluginEnvironmentExecuteResult, + PluginEnvironmentRunnerIngressEndpointParams, + PluginEnvironmentRunnerIngressEndpoint, PluginEnvironmentGetInteractiveSetupParams, PluginEnvironmentInteractiveSetupSession, PluginEnvironmentLease, @@ -1304,6 +1306,34 @@ const sandboxHandleCache = (() => { return { get, seed, clear, reset, markFresh, findByProviderLeaseId }; })(); +// Preview credentials can rotate without a sandbox restart, so they must not +// define endpoint generation. Daytona's lifecycle revision does: refreshData +// updates `updatedAt` after stop/start. When Daytona does not expose a +// lifecycle revision, the in-memory generation remains stable for the worker. +const runnerIngressGenerationStore = (() => { + const entries = new Map< + string, + { revision: string | null; generation: string } + >(); + + function get(sandbox: Sandbox): string { + const revision = sandbox.updatedAt ?? sandbox.createdAt ?? null; + const current = entries.get(sandbox.id); + if (current && current.revision === revision) return current.generation; + const generation = createHash("sha256") + .update(`${sandbox.id}\0${revision ?? randomUUID()}`) + .digest("hex"); + entries.set(sandbox.id, { revision, generation }); + return generation; + } + + function reset(): void { + entries.clear(); + } + + return { get, reset }; +})(); + // Advisory writable-set store. It holds, per lease scope, the sandbox // directories that a sync operation declared read-write (`access: "rw"`). The // store is advisory and best-effort in-memory state: it adds no security (the @@ -1422,6 +1452,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void { sandboxHandleLeaseAdmissionStates.reset(); sandboxHandleWritableDirs.reset(); sandboxHandleSessionStore.reset(); + runnerIngressGenerationStore.reset(); } /** @@ -2204,7 +2235,10 @@ const plugin = definePlugin({ }; } catch (error) { evictSandboxHandle(scope); - await sandbox.delete(toTimeoutSeconds(config.timeoutMs)).catch(() => undefined); + // A timeout, rate limit, or provider 5xx does not prove this sandbox is + // lost. Preserve the exact resource and let the host retry its recorded + // lease; replacement is permitted only after an explicit not-found or + // an immutable workspace identity mismatch. throw error; } }, { allowClosed: true }); @@ -2632,7 +2666,6 @@ const plugin = definePlugin({ }, }); const getDurationMs = timingNow() - getStart; - await ensureSandboxStarted(sandbox, toTimeoutSeconds(resolveTimeoutMs(params.timeoutMs, config))); const scope: SandboxScope = { driverKey: params.driverKey, companyId: params.companyId, @@ -2640,6 +2673,13 @@ const plugin = definePlugin({ providerLeaseId, config, }; + if (sandbox.state !== "started") { + // A provider restart destroys Daytona process sessions. Drop the stale + // session id before starting the sandbox so runnerd recovery opens a + // new session instead of retrying a dead one for its whole grace. + sandboxHandleSessionStore.clear(scope); + } + await ensureSandboxStarted(sandbox, toTimeoutSeconds(resolveTimeoutMs(params.timeoutMs, config))); // Dispatch the command. A normal command runs in the persistent session: // the provider opens the one session on a cache miss and runs every command // in it. The provider never falls back to a one-shot command to open a @@ -2668,6 +2708,70 @@ const plugin = definePlugin({ }); }, + async onEnvironmentRunnerIngressEndpoint( + params: PluginEnvironmentRunnerIngressEndpointParams, + ): Promise { + if (params.port !== 43_127) { + throw new Error("Daytona runner ingress must use fixed port 43127."); + } + if (!/^\/api\/runner\/v1\/connect\/[^/?#]+$/.test(params.path)) { + throw new Error("Daytona runner ingress path is invalid."); + } + const providerLeaseId = params.lease.providerLeaseId; + if (!providerLeaseId) { + throw new Error("Daytona runner ingress requires a provider lease id."); + } + const config = parseDriverConfig(params.config); + return await withSandboxActivityGate( + { + driverKey: params.driverKey, + companyId: params.companyId, + environmentId: params.environmentId, + providerLeaseId, + config, + }, + async () => { + const sandbox = await getSandbox({ + driverKey: params.driverKey, + companyId: params.companyId, + environmentId: params.environmentId, + providerLeaseId, + config, + }); + await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs)); + await withLivenessTimeout( + "sandbox.refreshData", + config.livenessTimeoutMs, + () => sandbox.refreshData(), + ); + const preview = await sandbox.getPreviewLink(params.port); + if (typeof preview.url !== "string" || typeof preview.token !== "string") { + throw new Error("Daytona returned an incomplete private preview endpoint."); + } + const url = new URL(preview.url); + if ( + url.protocol !== "https:" || + url.username || + url.password || + url.search || + url.hash + ) { + throw new Error("Daytona returned an invalid private preview URL."); + } + url.protocol = "wss:"; + url.pathname = `${url.pathname.replace(/\/$/, "")}${params.path}`; + return { + kind: "authenticated_websocket", + websocketUrl: url.toString(), + secretHeaders: [ + { name: "X-Daytona-Preview-Token", value: preview.token }, + ], + generation: runnerIngressGenerationStore.get(sandbox), + }; + }, + ); + }, + // Opt-in native inbound transfer. Defining this hook (with onEnvironmentSyncOut) // makes the worker advertise `environmentSyncIn`/`environmentSyncOut`, so the // host runner routes Daytona workspace/asset transfers through the SDK's batch @@ -2915,6 +3019,7 @@ const plugin = definePlugin({ await entry.session.close().catch(() => undefined); } sandboxHandleCache.reset(); + runnerIngressGenerationStore.reset(); }, }); diff --git a/packages/plugins/sdk/src/define-plugin.ts b/packages/plugins/sdk/src/define-plugin.ts index 0ff9ef74f7..495c00783f 100644 --- a/packages/plugins/sdk/src/define-plugin.ts +++ b/packages/plugins/sdk/src/define-plugin.ts @@ -55,6 +55,8 @@ import type { PluginEnvironmentDestroyLeaseParams, PluginEnvironmentExecuteParams, PluginEnvironmentExecuteResult, + PluginEnvironmentRunnerIngressEndpointParams, + PluginEnvironmentRunnerIngressEndpoint, PluginEnvironmentSyncInParams, PluginEnvironmentSyncOutParams, PluginEnvironmentSyncResult, @@ -399,6 +401,11 @@ export interface PluginDefinition { params: PluginEnvironmentExecuteParams, ): Promise; + /** Return an authenticated private WebSocket ingress for runnerd. */ + onEnvironmentRunnerIngressEndpoint?( + params: PluginEnvironmentRunnerIngressEndpointParams, + ): Promise; + /** * Optional, opt-in: called before execution to place host files/directories at * target sandbox paths using a provider-native transport instead of the default diff --git a/packages/plugins/sdk/src/index.ts b/packages/plugins/sdk/src/index.ts index d477e718e3..52e0886c82 100644 --- a/packages/plugins/sdk/src/index.ts +++ b/packages/plugins/sdk/src/index.ts @@ -188,6 +188,8 @@ export type { PluginEnvironmentRealizeWorkspaceResult, PluginEnvironmentExecuteParams, PluginEnvironmentExecuteResult, + PluginEnvironmentRunnerIngressEndpointParams, + PluginEnvironmentRunnerIngressEndpoint, PluginSyncFileMapping, PluginPostUploadCommand, PluginSyncOperation, diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts index af1a36b868..81ba4e6161 100644 --- a/packages/plugins/sdk/src/protocol.ts +++ b/packages/plugins/sdk/src/protocol.ts @@ -614,6 +614,21 @@ export interface PluginEnvironmentLease { expiresAt?: string | null; } +/** Serializable provider result. The host adds refresh/close lifecycle methods. */ +export interface PluginEnvironmentRunnerIngressEndpoint { + kind: "authenticated_websocket"; + websocketUrl: string; + secretHeaders: Array<{ name: string; value: string }>; + generation: string; +} + +export interface PluginEnvironmentRunnerIngressEndpointParams + extends PluginEnvironmentDriverBaseParams { + lease: PluginEnvironmentLease; + port: number; + path: string; +} + export interface PluginEnvironmentAcquireLeaseParams extends PluginEnvironmentDriverBaseParams { runId: string; workspaceMode?: string; @@ -1350,6 +1365,10 @@ export interface HostToWorkerMethods { params: PluginEnvironmentExecuteParams, result: PluginEnvironmentExecuteResult, ]; + environmentRunnerIngressEndpoint: [ + params: PluginEnvironmentRunnerIngressEndpointParams, + result: PluginEnvironmentRunnerIngressEndpoint, + ]; environmentSyncIn: [ params: PluginEnvironmentSyncInParams, result: PluginEnvironmentSyncResult, @@ -1440,6 +1459,7 @@ export const HOST_TO_WORKER_OPTIONAL_METHODS: readonly HostToWorkerMethodName[] "environmentDestroyLease", "environmentRealizeWorkspace", "environmentExecute", + "environmentRunnerIngressEndpoint", "environmentSyncIn", "environmentSyncOut", "environmentStartInteractiveSetup", diff --git a/packages/plugins/sdk/src/worker-rpc-host.ts b/packages/plugins/sdk/src/worker-rpc-host.ts index 5630e2a8e5..73fd52c44a 100644 --- a/packages/plugins/sdk/src/worker-rpc-host.ts +++ b/packages/plugins/sdk/src/worker-rpc-host.ts @@ -88,6 +88,7 @@ import type { PluginEnvironmentAcquireLeaseParams, PluginEnvironmentDestroyLeaseParams, PluginEnvironmentExecuteParams, + PluginEnvironmentRunnerIngressEndpointParams, PluginEnvironmentSyncInParams, PluginEnvironmentSyncOutParams, PluginEnvironmentRealizeWorkspaceParams, @@ -1640,6 +1641,11 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost case "environmentExecute": return handleEnvironmentExecute(params as PluginEnvironmentExecuteParams); + case "environmentRunnerIngressEndpoint": + return handleEnvironmentRunnerIngressEndpoint( + params as PluginEnvironmentRunnerIngressEndpointParams, + ); + case "environmentSyncIn": return handleEnvironmentSyncIn(params as PluginEnvironmentSyncInParams); @@ -1729,6 +1735,9 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost if (plugin.definition.onEnvironmentDestroyLease) supportedMethods.push("environmentDestroyLease"); if (plugin.definition.onEnvironmentRealizeWorkspace) supportedMethods.push("environmentRealizeWorkspace"); if (plugin.definition.onEnvironmentExecute) supportedMethods.push("environmentExecute"); + if (plugin.definition.onEnvironmentRunnerIngressEndpoint) { + supportedMethods.push("environmentRunnerIngressEndpoint"); + } if (plugin.definition.onEnvironmentSyncIn) supportedMethods.push("environmentSyncIn"); if (plugin.definition.onEnvironmentSyncOut) supportedMethods.push("environmentSyncOut"); if (plugin.definition.onEnvironmentStartInteractiveSetup) supportedMethods.push("environmentStartInteractiveSetup"); @@ -2043,6 +2052,15 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost return plugin.definition.onEnvironmentExecute(params); } + async function handleEnvironmentRunnerIngressEndpoint( + params: PluginEnvironmentRunnerIngressEndpointParams, + ) { + if (!plugin.definition.onEnvironmentRunnerIngressEndpoint) { + throw methodNotImplemented("environmentRunnerIngressEndpoint"); + } + return plugin.definition.onEnvironmentRunnerIngressEndpoint(params); + } + async function handleEnvironmentSyncIn(params: PluginEnvironmentSyncInParams) { if (!plugin.definition.onEnvironmentSyncIn) { throw methodNotImplemented("environmentSyncIn"); diff --git a/packages/shared/src/feature-catalog.ts b/packages/shared/src/feature-catalog.ts index c83f247759..81366d14f0 100644 --- a/packages/shared/src/feature-catalog.ts +++ b/packages/shared/src/feature-catalog.ts @@ -269,6 +269,14 @@ export const INSTANCE_FEATURE_CATALOG: Record; diff --git a/server/src/__tests__/agent-adapter-validation-routes.test.ts b/server/src/__tests__/agent-adapter-validation-routes.test.ts index 7b7b0732ca..4c0f8f64c7 100644 --- a/server/src/__tests__/agent-adapter-validation-routes.test.ts +++ b/server/src/__tests__/agent-adapter-validation-routes.test.ts @@ -598,6 +598,65 @@ describe("agent routes adapter validation", () => { expect(mockAgentService.create).toHaveBeenCalledOnce(); }); + it("rejects non-Codex providers on fresh paperclip_runner agents and hires", async () => { + mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableNativeRunner: true }); + const app = await createApp(); + const createResponse = await requestApp(app, (baseUrl) => + request(baseUrl) + .post("/api/companies/company-1/agents") + .send({ + name: "Native OpenCode", + adapterType: "paperclip_runner", + adapterConfig: { provider: "opencode" }, + }), + ); + const hireResponse = await requestApp(app, (baseUrl) => + request(baseUrl) + .post("/api/companies/company-1/agent-hires") + .send({ + name: "Native ACPX", + adapterType: "paperclip_runner", + adapterConfig: { provider: "acpx" }, + }), + ); + + expect(createResponse.status, JSON.stringify(createResponse.body)).toBe(422); + expect(createResponse.body.details).toMatchObject({ + code: "paperclip_runner_provider_unavailable", + }); + expect(hireResponse.status, JSON.stringify(hireResponse.body)).toBe(422); + expect(hireResponse.body.details).toMatchObject({ + code: "paperclip_runner_provider_unavailable", + }); + expect(mockAgentService.create).not.toHaveBeenCalled(); + }); + + it("rejects provider changes but preserves edits to historical runner agents", async () => { + const existing = await mockAgentService.getById(); + mockAgentService.getById.mockResolvedValue({ + ...existing, + adapterType: "paperclip_runner", + adapterConfig: { provider: "opencode", model: "historical" }, + }); + const app = await createApp(); + const ordinaryEdit = await requestApp(app, (baseUrl) => + request(baseUrl) + .patch("/api/agents/11111111-1111-4111-8111-111111111111") + .send({ name: "Historical Runner" }), + ); + const providerChange = await requestApp(app, (baseUrl) => + request(baseUrl) + .patch("/api/agents/11111111-1111-4111-8111-111111111111") + .send({ adapterConfig: { provider: "acpx" } }), + ); + + expect(ordinaryEdit.status, JSON.stringify(ordinaryEdit.body)).toBe(200); + expect(providerChange.status, JSON.stringify(providerChange.body)).toBe(422); + expect(providerChange.body.details).toMatchObject({ + code: "paperclip_runner_provider_unavailable", + }); + }); + it("keeps an existing paperclip_runner agent editable after the flag is disabled", async () => { const existing = await mockAgentService.getById(); mockAgentService.getById.mockResolvedValue({ diff --git a/server/src/__tests__/agent-permissions-routes.test.ts b/server/src/__tests__/agent-permissions-routes.test.ts index 6ce5150aa9..982cca00a5 100644 --- a/server/src/__tests__/agent-permissions-routes.test.ts +++ b/server/src/__tests__/agent-permissions-routes.test.ts @@ -42,11 +42,13 @@ const baseAgent = { const mockAgentService = vi.hoisted(() => ({ getById: vi.fn(), + getConfigRevision: vi.fn(), list: vi.fn(), create: vi.fn(), activatePendingApproval: vi.fn(), terminate: vi.fn(), update: vi.fn(), + rollbackConfigRevision: vi.fn(), updatePermissions: vi.fn(), getChainOfCommand: vi.fn(), resolveByReference: vi.fn(), @@ -288,11 +290,13 @@ describe.sequential("agent permission routes", () => { beforeEach(() => { vi.resetAllMocks(); mockAgentService.getById.mockReset(); + mockAgentService.getConfigRevision.mockReset(); mockAgentService.list.mockReset(); mockAgentService.create.mockReset(); mockAgentService.activatePendingApproval.mockReset(); mockAgentService.terminate.mockReset(); mockAgentService.update.mockReset(); + mockAgentService.rollbackConfigRevision.mockReset(); mockAgentService.updatePermissions.mockReset(); mockAgentService.getChainOfCommand.mockReset(); mockAgentService.resolveByReference.mockReset(); @@ -332,6 +336,7 @@ describe.sequential("agent permission routes", () => { mockSyncInstructionsBundleConfigFromFilePath.mockImplementation((_agent, config) => config); mockGetTelemetryClient.mockReturnValue({ track: vi.fn() }); mockAgentService.getById.mockResolvedValue(baseAgent); + mockAgentService.getConfigRevision.mockResolvedValue(null); mockAgentService.list.mockResolvedValue([baseAgent]); mockAgentService.getChainOfCommand.mockResolvedValue([]); mockAgentService.resolveByReference.mockResolvedValue({ ambiguous: false, agent: baseAgent }); @@ -570,6 +575,63 @@ describe.sequential("agent permission routes", () => { expect(mockAgentService.create).not.toHaveBeenCalled(); }); + it("allows instance administrators to create and hire with raw provider traces", async () => { + const app = await createApp({ + type: "board", + userId: "instance-admin-user", + source: "session", + isInstanceAdmin: true, + companyIds: [companyId], + }); + const body = { + name: "Trace capture agent", + adapterType: "process", + runtimeConfig: { debug: { providerTrace: "raw" } }, + }; + + const createResponse = await requestApp(app, (baseUrl) => + request(baseUrl) + .post(`/api/companies/${companyId}/agents`) + .send(body), + ); + const hireResponse = await requestApp(app, (baseUrl) => + request(baseUrl) + .post(`/api/companies/${companyId}/agent-hires`) + .send(body), + ); + + expect(createResponse.status, JSON.stringify(createResponse.body)).toBe(201); + expect(hireResponse.status, JSON.stringify(hireResponse.body)).toBe(201); + expect(mockAgentService.create).toHaveBeenCalledTimes(2); + }); + + it("rejects non-admin rollback into raw provider trace capture", async () => { + mockAgentService.getConfigRevision.mockResolvedValue({ + id: "33333333-3333-4333-8333-333333333333", + afterConfig: { + adapterType: "process", + adapterConfig: {}, + runtimeConfig: { debug: { providerTrace: "raw" } }, + }, + }); + const app = await createApp({ + type: "board", + userId: "agent-admin-user", + source: "session", + isInstanceAdmin: false, + companyIds: [companyId], + }); + + const response = await requestApp(app, (baseUrl) => + request(baseUrl).post( + `/api/agents/${agentId}/config-revisions/33333333-3333-4333-8333-333333333333/rollback`, + ), + ); + + expect(response.status).toBe(403); + expect(mockAgentService.rollbackConfigRevision).not.toHaveBeenCalled(); + }); + it("blocks api key creation for authenticated company members without agent admin permission", async () => { mockAccessService.canUser.mockResolvedValue(false); diff --git a/server/src/__tests__/environment-capability-contract.test.ts b/server/src/__tests__/environment-capability-contract.test.ts index 278a241084..b331ef869e 100644 --- a/server/src/__tests__/environment-capability-contract.test.ts +++ b/server/src/__tests__/environment-capability-contract.test.ts @@ -401,8 +401,13 @@ describe("general runtime capability resolver — four-driver matrix", () => { incrementalSessionOutput: true, concurrentSyncOperations: true, duplexCommandStream: true, + runnerWebSocketIngress: true, }; - const VERIFY_ALL = [...ALL_PLUGIN_METHODS, "duplexChannelOpen"]; + const VERIFY_ALL = [ + ...ALL_PLUGIN_METHODS, + "duplexChannelOpen", + "environmentRunnerIngressEndpoint", + ]; it("test_local_and_ssh_drivers_support_no_capability_regardless_of_declaration_or_worker", () => { // The `local` and `ssh` static support definitions name none of the diff --git a/server/src/__tests__/environment-execution-target-capabilities.test.ts b/server/src/__tests__/environment-execution-target-capabilities.test.ts index b9cb1245cb..5a35dec846 100644 --- a/server/src/__tests__/environment-execution-target-capabilities.test.ts +++ b/server/src/__tests__/environment-execution-target-capabilities.test.ts @@ -23,6 +23,7 @@ const SNAPSHOT: EffectiveExecutionCapabilities = { // inbound sync, so the opt-in stays off. concurrentSyncOperations: false, duplexCommandStream: false, + runnerWebSocketIngress: false, }; // A snapshot that grants every capability. A test overrides one flag to prove @@ -36,6 +37,7 @@ const FULL_GRANT: EffectiveExecutionCapabilities = { incrementalSessionOutput: true, concurrentSyncOperations: true, duplexCommandStream: true, + runnerWebSocketIngress: true, }; // Build a sandbox execution target with a fixed snapshot and a fixed @@ -197,6 +199,62 @@ describe("resolveEnvironmentExecutionTarget effective capability snapshot", () = expect(target?.kind).toBe("remote"); expect(resolveCapabilities).not.toHaveBeenCalled(); }); + + it("carries the environment-owned warm runner lifecycle and reuse requirement", async () => { + const { target } = await buildSandboxTarget({ + snapshot: FULL_GRANT, + supportsSync: false, + config: { + reuseLease: true, + runnerLifecycleMode: "warm", + runnerIdleTimeoutMs: 45_000, + }, + }); + + expect(target.runnerLifecyclePolicy).toEqual({ + mode: "warm", + idleTimeoutMs: 45_000, + }); + expect(target.reusableLeaseConfigured).toBe(true); + }); + + it("carries host-owned sandbox acquisition provenance without persisting provider ids in metadata", async () => { + mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({ + driver: "sandbox", + config: { provider: "daytona", reuseLease: true, timeoutMs: 30_000 }, + }); + const target = await resolveEnvironmentExecutionTarget({ + db: {} as never, + companyId: "company-1", + // This substrate PR does not advertise remote paperclip_runner support + // until the Rust WSS transport lands. A supported direct adapter exercises + // the same host-owned acquisition contract without widening rollout here. + adapterType: "codex_local", + environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } }, + leaseId: "lease-row-1", + leaseMetadata: { + remoteCwd: "/work", + sandboxLeaseAcquisition: { outcome: "resumed" }, + }, + lease: { + id: "lease-row-1", + providerLeaseId: "daytona-sandbox-1", + leasePolicy: "reuse_by_environment", + metadata: { sandboxLeaseAcquisition: { outcome: "resumed" } }, + } as never, + environmentRuntime: { + supportsSync: () => false, + resolveCapabilities: vi.fn(async () => ({ ...FULL_GRANT })), + } as unknown as EnvironmentRuntimeService, + }); + if (target?.kind !== "remote" || target.transport !== "sandbox") { + throw new Error("expected a sandbox target"); + } + expect(target.sandboxLeaseAcquisition).toEqual({ + outcome: "resumed", + providerLeaseId: "daytona-sandbox-1", + }); + }); }); describe("effective snapshot gates the sync decision", () => { diff --git a/server/src/__tests__/environment-execution-target-duplex.test.ts b/server/src/__tests__/environment-execution-target-duplex.test.ts index 39310c29be..faa8bafb81 100644 --- a/server/src/__tests__/environment-execution-target-duplex.test.ts +++ b/server/src/__tests__/environment-execution-target-duplex.test.ts @@ -48,6 +48,7 @@ const DUPLEX_GRANT: EffectiveExecutionCapabilities = { incrementalSessionOutput: true, concurrentSyncOperations: true, duplexCommandStream: true, + runnerWebSocketIngress: true, }; const DUPLEX_ABSENT: EffectiveExecutionCapabilities = { diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index ed5d31ab26..087eea9577 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -1,5 +1,5 @@ import { createHash, randomUUID } from "node:crypto"; -import { mkdtemp, readdir, rm } from "node:fs/promises"; +import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; @@ -49,6 +49,8 @@ import { } from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; import { traceparentFromContextToken } from "../instrumentation.ts"; import { ROOT_CONTEXT, trace } from "@opentelemetry/api"; +import { buildNativeHarnessBackupManifest } from "../services/native-runtime/native-session-executor.ts"; +import { createNativeHarnessBackupStamp } from "../services/native-runtime/native-harness-backup-stamp.ts"; const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -314,7 +316,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }; } - async function seedReusablePluginSandboxLease() { + async function seedReusablePluginSandboxLease(adapterType: string | null = null) { const pluginId = randomUUID(); const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); const providerConfig = { @@ -418,11 +420,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { environmentId: environment.id, executionWorkspaceId, agentId, - adapterType: null, + adapterType, provider: "fake-plugin", runtimeFingerprint: reusableRuntimeFingerprint({ provider: "fake-plugin", - adapterType: null, + adapterType, config: providerConfig, }), }, @@ -432,6 +434,244 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { return { pluginId, companyId, agentId, environment, runId, executionWorkspaceId, reusableLease }; } + it("retains a successful reusable sandbox lease without stopping the provider resource", async () => { + const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease(); + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + throw new Error(`Unexpected plugin method while retaining warm lease: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const released = await runtimeWithPlugin.releaseRunLeases( + runId, + "released", + undefined, + "keep_running", + ); + + expect(released).toHaveLength(1); + expect(released[0]?.lease).toMatchObject({ + id: reusableLease.id, + status: "retained", + cleanupStatus: "success", + }); + await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({ + status: "retained", + cleanupStatus: "success", + }); + expect(workerManager.call).not.toHaveBeenCalled(); + }); + + it("stops a reusable sandbox and keeps its lease resumable", async () => { + const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease(); + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentReleaseLease") return undefined; + throw new Error(`Unexpected plugin method while stopping lease: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const released = await runtimeWithPlugin.releaseRunLeases( + runId, + "failed", + undefined, + "stop_and_retain", + ); + + expect(released[0]?.lease).toMatchObject({ + id: reusableLease.id, + status: "released", + }); + expect(workerManager.call).toHaveBeenCalledWith( + pluginId, + "environmentReleaseLease", + expect.objectContaining({ providerLeaseId: reusableLease.providerLeaseId }), + expect.any(Number), + ); + }); + + it("resumes the same provider lease on the next per-turn run", async () => { + const seeded = await seedReusablePluginSandboxLease(); + const workerManager = { + isRunning: vi.fn((id: string) => id === seeded.pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "sandbox-exact-resume", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + remoteCwd: "/workspace", + }, + }; + } + if (method === "environmentReleaseLease") return undefined; + if (method === "environmentResumeLease") { + return { + providerLeaseId: "sandbox-exact-resume", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + remoteCwd: "/workspace", + }, + }; + } + throw new Error(`Unexpected plugin method during exact resume: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + await environmentService(db).releaseLease(seeded.reusableLease.id, "expired"); + const first = await runtimeWithPlugin.acquireRunLease({ + companyId: seeded.companyId, + environment: seeded.environment, + issueId: null, + agentId: seeded.agentId, + heartbeatRunId: seeded.runId, + persistedExecutionWorkspace: { + id: seeded.executionWorkspaceId, + mode: "shared_workspace", + }, + }); + await runtimeWithPlugin.releaseRunLeases( + seeded.runId, + "released", + undefined, + "stop_and_retain", + ); + + const nextRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ + id: nextRunId, + companyId: seeded.companyId, + agentId: seeded.agentId, + invocationSource: "manual", + status: "running", + createdAt: new Date(), + updatedAt: new Date(), + }); + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId: seeded.companyId, + environment: seeded.environment, + issueId: null, + agentId: seeded.agentId, + heartbeatRunId: nextRunId, + persistedExecutionWorkspace: { + id: seeded.executionWorkspaceId, + mode: "shared_workspace", + }, + }); + + expect(first.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "created" }); + expect(acquired.lease.providerLeaseId).toBe("sandbox-exact-resume"); + expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" }); + expect(workerManager.call.mock.calls.filter((call) => call[1] === "environmentAcquireLease")) + .toHaveLength(1); + }); + + it("destroys a disposable paperclip_runner sandbox after the turn", async () => { + const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease(); + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentDestroyLease") return undefined; + throw new Error(`Unexpected plugin method while destroying lease: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const released = await runtimeWithPlugin.releaseRunLeases( + runId, + "released", + undefined, + "destroy", + ); + + expect(released[0]?.lease).toMatchObject({ + id: reusableLease.id, + status: "expired", + }); + expect(workerManager.call).toHaveBeenCalledWith( + pluginId, + "environmentDestroyLease", + expect.objectContaining({ providerLeaseId: reusableLease.providerLeaseId }), + expect.any(Number), + ); + }); + + it("does not destroy native sandbox state when its verified backup is missing", async () => { + const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease(); + await environmentService(db).updateLeaseMetadata(reusableLease.id, { + ...(reusableLease.metadata ?? {}), + sandboxLeaseAcquisition: { outcome: "created" }, + }); + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + throw new Error(`Provider teardown must not run without a backup: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + const errors: unknown[] = []; + + const released = await runtimeWithPlugin.releaseRunLeases( + runId, + "released", + (_leaseId, error) => errors.push(error), + "destroy", + ); + + expect(released).toEqual([]); + expect(errors).toHaveLength(1); + expect(String(errors[0])).toContain("runner_harness_backup_unavailable"); + expect(workerManager.call).not.toHaveBeenCalled(); + await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({ + status: "active", + }); + }); + it("acquires and releases a local run lease through the runtime seam", async () => { const { companyId, environment, runId } = await seedEnvironment(); @@ -4166,7 +4406,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { ); }); - it("falls back to acquire when plugin-backed sandbox lease resume throws", async () => { + it("preserves the exact lease when plugin-backed sandbox resume throws", async () => { const pluginId = randomUUID(); const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); const providerConfig = { @@ -4308,7 +4548,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); - const acquired = await runtimeWithPlugin.acquireRunLease({ + await expect(runtimeWithPlugin.acquireRunLease({ companyId, environment, issueId: null, @@ -4318,34 +4558,196 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { id: executionWorkspaceId, mode: "shared_workspace", }, - }); + })).rejects.toThrow("the lease was preserved and no replacement was created"); - expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease"); expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentResumeLease", expect.objectContaining({ driverKey: "fake-plugin", providerLeaseId: "stale-plugin-lease", }), 31234); - expect(workerManager.call).toHaveBeenNthCalledWith(2, pluginId, "environmentDestroyLease", expect.objectContaining({ - driverKey: "fake-plugin", - providerLeaseId: "stale-plugin-lease", - }), 31234); - expect(workerManager.call).toHaveBeenNthCalledWith(3, pluginId, "environmentAcquireLease", expect.objectContaining({ - driverKey: "fake-plugin", - config: { - image: "fake:test", - timeoutMs: 1234, - reuseLease: true, - }, - agentId, - executionWorkspaceId, - runId, - }), 31234); + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentDestroyLease", + expect.anything(), + expect.anything(), + ); + expect(workerManager.call).not.toHaveBeenCalledWith( + pluginId, + "environmentAcquireLease", + expect.anything(), + expect.anything(), + ); await expect(environmentService(db).getLeaseById(staleLease.id)).resolves.toMatchObject({ - status: "expired", - cleanupStatus: "success", + status: "active", }); }); + it("does not allocate a native-runner replacement without a verified backup", async () => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + const workerManager = { + isRunning: vi.fn((id: string) => id === seeded.pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentResumeLease") { + return { providerLeaseId: null, metadata: { expired: true } }; + } + if (method === "environmentDestroyLease") return undefined; + if (method === "environmentAcquireLease") { + throw new Error("replacement must not be allocated"); + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + await expect(runtimeWithPlugin.acquireRunLease({ + companyId: seeded.companyId, + environment: seeded.environment, + issueId: null, + agentId: seeded.agentId, + adapterType: "paperclip_runner", + heartbeatRunId: seeded.runId, + persistedExecutionWorkspace: { + id: seeded.executionWorkspaceId, + mode: "shared_workspace", + }, + })).rejects.toThrow("runner_harness_backup_unavailable"); + + expect(workerManager.call).not.toHaveBeenCalledWith( + seeded.pluginId, + "environmentAcquireLease", + expect.anything(), + expect.anything(), + ); + expect(workerManager.call).not.toHaveBeenCalledWith( + seeded.pluginId, + "environmentDestroyLease", + expect.anything(), + expect.anything(), + ); + }); + + it("permits native-runner replacement only after verifying the stamped backup", async () => { + const seeded = await seedReusablePluginSandboxLease("paperclip_runner"); + const backupBase = await mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-replacement-")); + const previousStateDirectory = process.env.PAPERCLIP_RUNNER_STATE_DIR; + process.env.PAPERCLIP_RUNNER_STATE_DIR = backupBase; + try { + const normalizedSessionId = "native-replacement-session"; + const runnerInstanceId = "native-replacement-runner"; + const sessionRoot = path.join( + backupBase, + createHash("sha256").update(normalizedSessionId).digest("hex"), + ); + const current = path.join(sessionRoot, "failover-backups", "current"); + await mkdir(path.join(current, "runner"), { recursive: true }); + await mkdir(path.join(current, "codex-home", "sessions"), { recursive: true }); + await writeFile(path.join(current, "runner", "runner-state.json"), "runner-state"); + await writeFile(path.join(current, "codex-home", "sessions", "thread.jsonl"), "thread-state"); + const execution = { + provider: { kind: "codex", model: null, approvalPolicy: "never" }, + binding: { + companyId: seeded.companyId, + runId: seeded.runId, + issueId: "issue", + agentId: seeded.agentId, + executionWorkspaceId: seeded.executionWorkspaceId, + }, + workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + session: { + normalizedSessionId, + driverKind: "codex_app_server", + protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + } as never; + const manifest = buildNativeHarnessBackupManifest({ + backupRoot: current, + execution, + runnerInstanceId, + providerSessionIdentity: { + providerSessionId: "thread-1", + providerBackendSessionId: "session-1", + providerSessionIdentity: null, + }, + sourceProviderLeaseId: seeded.reusableLease.providerLeaseId!, + }); + const manifestPath = path.join(current, "manifest.json"); + await writeFile(manifestPath, JSON.stringify(manifest)); + const stamp = createNativeHarnessBackupStamp({ + manifestPath, + normalizedSessionId, + runnerInstanceId, + completedAt: manifest.completedAt, + }); + await environmentService(db).updateLeaseMetadata(seeded.reusableLease.id, { + ...(seeded.reusableLease.metadata ?? {}), + nativeHarnessBackup: stamp, + }); + + const workerManager = { + isRunning: vi.fn((id: string) => id === seeded.pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentResumeLease") { + return { providerLeaseId: null, metadata: { expired: true } }; + } + if (method === "environmentDestroyLease") return undefined; + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "replacement-plugin-lease", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: true, + remoteCwd: "/workspace", + }, + }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + getWorker: vi.fn(() => ({ + supportedMethods: [ + "environmentResumeLease", + "environmentReleaseLease", + "environmentDestroyLease", + ], + })), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId: seeded.companyId, + environment: seeded.environment, + issueId: null, + agentId: seeded.agentId, + adapterType: "paperclip_runner", + heartbeatRunId: seeded.runId, + persistedExecutionWorkspace: { + id: seeded.executionWorkspaceId, + mode: "shared_workspace", + }, + }); + + expect(acquired.lease.providerLeaseId).toBe("replacement-plugin-lease"); + expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ + outcome: "replacement", + reason: "not_found", + }); + } finally { + if (previousStateDirectory === undefined) { + delete process.env.PAPERCLIP_RUNNER_STATE_DIR; + } else { + process.env.PAPERCLIP_RUNNER_STATE_DIR = previousStateDirectory; + } + await rm(backupBase, { recursive: true, force: true }); + } + }); + it("fails closed and does not resume when a worker restart drops the resume method after the capability snapshot", async () => { const pluginId = randomUUID(); const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); @@ -4505,7 +4907,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { } as unknown as PluginWorkerManager; const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); - const acquired = await runtimeWithPlugin.acquireRunLease({ + await expect(runtimeWithPlugin.acquireRunLease({ companyId, environment, issueId: null, @@ -4515,7 +4917,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { id: executionWorkspaceId, mode: "shared_workspace", }, - }); + })).rejects.toThrow("the lease was preserved and no replacement was created"); // The runtime re-checks the live worker before the resume dispatch. The // restarted worker no longer advertises `environmentResumeLease`, so the @@ -4526,23 +4928,21 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { expect.anything(), expect.anything(), ); - // It destroys the stale reusable lease and acquires a fresh one. - expect(workerManager.call).toHaveBeenCalledWith( + // It preserves the stale reusable lease and does not allocate a replacement. + expect(workerManager.call).not.toHaveBeenCalledWith( pluginId, "environmentDestroyLease", - expect.objectContaining({ driverKey: "fake-plugin", providerLeaseId: "stale-plugin-lease" }), - 31234, + expect.anything(), + expect.anything(), ); - expect(workerManager.call).toHaveBeenCalledWith( + expect(workerManager.call).not.toHaveBeenCalledWith( pluginId, "environmentAcquireLease", - expect.objectContaining({ driverKey: "fake-plugin", agentId, executionWorkspaceId, runId }), - 31234, + expect.anything(), + expect.anything(), ); - expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease"); await expect(environmentService(db).getLeaseById(staleLease.id)).resolves.toMatchObject({ - status: "expired", - cleanupStatus: "success", + status: "active", }); }); diff --git a/server/src/__tests__/general-capability-classifier.test.ts b/server/src/__tests__/general-capability-classifier.test.ts index 95ec0c3c07..40d00f6456 100644 --- a/server/src/__tests__/general-capability-classifier.test.ts +++ b/server/src/__tests__/general-capability-classifier.test.ts @@ -16,6 +16,7 @@ const ALL_PROVIDER_METHODS = [ "environmentSyncIn", "environmentSyncOut", "duplexChannelOpen", + "environmentRunnerIngressEndpoint", ]; describe("general capability classifier", () => { @@ -34,6 +35,7 @@ describe("general capability classifier", () => { incrementalSessionOutput: true, concurrentSyncOperations: true, duplexCommandStream: true, + runnerWebSocketIngress: true, }, supportedCapabilities: ENVIRONMENT_DRIVER_CAPABILITY_SUPPORT.sandbox.supportedCapabilities, }); @@ -96,6 +98,7 @@ describe("general capability classifier", () => { incrementalSessionOutput: true, concurrentSyncOperations: true, duplexCommandStream: true, + runnerWebSocketIngress: true, }, supportedCapabilities: ENVIRONMENT_DRIVER_CAPABILITY_SUPPORT[driver].supportedCapabilities, }); diff --git a/server/src/__tests__/heartbeat-native-runner-cancellation.test.ts b/server/src/__tests__/heartbeat-native-runner-cancellation.test.ts index 3b01bdf9a9..31749f638f 100644 --- a/server/src/__tests__/heartbeat-native-runner-cancellation.test.ts +++ b/server/src/__tests__/heartbeat-native-runner-cancellation.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { cancelHeartbeatNativeRun } from "../services/heartbeat.js"; +import { resolveNativeCancellationStatus } from "../services/native-runtime/native-session-executor.js"; describe("native heartbeat cancellation authority", () => { it("does not enter native cancellation for a direct-adapter run", async () => { @@ -54,4 +55,31 @@ describe("native heartbeat cancellation authority", () => { cancel, })).rejects.toThrow("native_cancellation_outcome_not_audited"); }); + + it("keeps generalized executor cancellation scoped to the requested authority", () => { + expect(resolveNativeCancellationStatus({ + scope: "run", + priorIssueStatus: "in_progress", + agentId: "remote-runner", + })).toMatchObject({ + statusAction: "preserve", + toStatus: "in_progress", + reasonCode: "cancellation_run_only", + effects: [{ kind: "release_run_resources" }], + }); + + expect(resolveNativeCancellationStatus({ + scope: "issue", + priorIssueStatus: "in_progress", + agentId: "remote-runner", + })).toMatchObject({ + statusAction: "cancelled", + toStatus: "cancelled", + reasonCode: "cancellation_issue_authorized", + effects: [ + { kind: "release_checkout" }, + { kind: "cancel_continuations" }, + ], + }); + }); }); diff --git a/server/src/__tests__/instance-settings-service.test.ts b/server/src/__tests__/instance-settings-service.test.ts index a11ad6069d..27f80f5411 100644 --- a/server/src/__tests__/instance-settings-service.test.ts +++ b/server/src/__tests__/instance-settings-service.test.ts @@ -55,6 +55,7 @@ describe("instance settings service", () => { enableWorkspaceDirtyQuarantineRepair: false, enableOwnerInstanceAdmin: false, enableSandboxDuplexBridge: false, + enableRunnerPreviewIngress: false, enableWorktreeRunExecution: false, worktreeRunExecutionActivatedAt: null, worktreeRunExecutionActivationInstanceId: null, diff --git a/server/src/__tests__/native-sandbox-lifecycle.test.ts b/server/src/__tests__/native-sandbox-lifecycle.test.ts new file mode 100644 index 0000000000..4cc88c1a20 --- /dev/null +++ b/server/src/__tests__/native-sandbox-lifecycle.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it } from "vitest"; +import { resolveNativeSandboxLifecycle } from "../services/heartbeat.js"; + +const reusableSandbox = { + kind: "remote" as const, + transport: "sandbox", + reusableLeaseConfigured: true, + effectiveCapabilities: { reusableLeases: true }, +}; + +describe("paperclip_runner sandbox lifecycle", () => { + it("keeps a warm reusable sandbox running", () => { + expect(resolveNativeSandboxLifecycle({ + adapterType: "paperclip_runner", + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 }, + target: reusableSandbox, + })).toEqual({ + runnerProcess: "warm", + sandboxResource: "keep_running", + failoverBackup: "verified", + }); + }); + + it("stops and reuses a per-turn reusable sandbox", () => { + expect(resolveNativeSandboxLifecycle({ + adapterType: "paperclip_runner", + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + target: reusableSandbox, + })).toEqual({ + runnerProcess: "per_turn", + sandboxResource: "stop_and_reuse", + failoverBackup: "verified", + }); + }); + + it("destroys a per-turn disposable sandbox", () => { + expect(resolveNativeSandboxLifecycle({ + adapterType: "paperclip_runner", + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + target: { + ...reusableSandbox, + reusableLeaseConfigured: false, + }, + })).toEqual({ + runnerProcess: "per_turn", + sandboxResource: "destroy_after_turn", + failoverBackup: "verified", + }); + }); + + it("rejects warm mode without an effective reusable-lease capability", () => { + expect(() => resolveNativeSandboxLifecycle({ + adapterType: "paperclip_runner", + lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 }, + target: { + ...reusableSandbox, + effectiveCapabilities: { reusableLeases: false }, + }, + })).toThrow("runner_warm_lifecycle_requires_reusable_provider_lease"); + }); + + it("leaves legacy and non-sandbox targets untouched", () => { + expect(resolveNativeSandboxLifecycle({ + adapterType: "codex_local", + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + target: reusableSandbox, + })).toBeNull(); + expect(resolveNativeSandboxLifecycle({ + adapterType: "paperclip_runner", + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + target: { kind: "local" }, + })).toBeNull(); + }); +}); diff --git a/server/src/__tests__/native-session-resumption.test.ts b/server/src/__tests__/native-session-resumption.test.ts index 42d5491ef3..096e56359c 100644 --- a/server/src/__tests__/native-session-resumption.test.ts +++ b/server/src/__tests__/native-session-resumption.test.ts @@ -232,7 +232,7 @@ describe("P6-25 pre-result native session recovery", () => { nextAttemptAt: new Date(0), failureCode: "native_session_interrupted", failureDetail: { - message: "provider_initialize_timeout: provider=codex stage=health", + message: "provider_initialize_timeout: provider=opencode stage=health", originalFailureCode: "provider_initialize_timeout", recoveryMode: "bootstrap_retry", providerSessionEstablished: false, @@ -288,7 +288,7 @@ describe("P6-25 pre-result native session recovery", () => { await expect(db.select().from(nativeRunFinalizations)).resolves.toHaveLength(8); }); - it("uses checkpoint-free bootstrap retry only when durable evidence proves no provider session existed", async () => { + it("uses provider-neutral checkpoint-free bootstrap retry only when durable evidence proves no session existed", async () => { await expect(claimNativeSessionResumptions({ db, runnerInstanceId: "reaper", diff --git a/server/src/realtime/runner-prp-outbound.test.ts b/server/src/realtime/runner-prp-outbound.test.ts new file mode 100644 index 0000000000..4c814c59c5 --- /dev/null +++ b/server/src/realtime/runner-prp-outbound.test.ts @@ -0,0 +1,171 @@ +import { EventEmitter } from "node:events"; +import { describe, expect, it, vi } from "vitest"; +import WebSocket from "ws"; +import type { DurablePrpControlPlane } from "../vendor/paperclip-runner/index.js"; +import { + __runnerPrpOutboundTesting, + connectRunnerPrpIngress, + WsJsonWireConnection, +} from "./runner-prp-outbound.js"; + +class FakeSocket extends EventEmitter { + readyState: number = WebSocket.OPEN; + send = vi.fn(); + close = vi.fn((code?: number) => { + this.readyState = WebSocket.CLOSED; + this.emit("close", code ?? 1000, Buffer.alloc(0)); + }); +} + +describe("runner provider-ingress WebSocket wire", () => { + it("bounds credential refresh by the fixed deadline and cancellation", async () => { + const signal = new AbortController().signal; + await expect( + __runnerPrpOutboundTesting.awaitWithinDeadline({ + operation: async () => await new Promise(() => undefined), + deadline: Date.now() - 1, + signal, + }), + ).rejects.toThrow("deadline elapsed"); + + const abort = new AbortController(); + const pending = __runnerPrpOutboundTesting.awaitWithinDeadline({ + operation: async () => await new Promise(() => undefined), + deadline: Date.now() + 60_000, + signal: abort.signal, + }); + abort.abort(); + await expect(pending).rejects.toThrow("cancelled"); + }); + + it("delivers one terminal close to both PRP authority and reconnect ownership", () => { + const socket = new FakeSocket(); + const wire = new WsJsonWireConnection( + socket as unknown as WebSocket, + ); + const authorityClose = vi.fn(); + const reconnectClose = vi.fn(); + wire.onClose(authorityClose); + wire.onClose(reconnectClose); + + const error = new Error("preview disconnected"); + socket.emit("error", error); + socket.emit("close", 1006, Buffer.from("duplicate close")); + + expect(authorityClose).toHaveBeenCalledOnce(); + expect(reconnectClose).toHaveBeenCalledOnce(); + expect(authorityClose).toHaveBeenCalledWith({ + message: "websocket_error", + error, + }); + }); + + it("replays a close to a listener registered after the socket ended", () => { + const socket = new FakeSocket(); + const wire = new WsJsonWireConnection( + socket as unknown as WebSocket, + ); + socket.emit("close", 1001, Buffer.from("sandbox restart")); + const listener = vi.fn(); + wire.onClose(listener); + expect(listener).toHaveBeenCalledWith({ + code: 1001, + message: "sandbox restart", + }); + }); + + it("reports a terminal ingress failure to startup and active-run ownership", async () => { + const endpoint = { + kind: "authenticated_websocket" as const, + websocketUrl: "ws://preview.invalid/api/runner/v1/connect/run-1", + secretHeaders: [], + generation: "generation-1", + refresh: async () => endpoint, + close: async () => undefined, + }; + const handle = connectRunnerPrpIngress({ + authority: { + attachWireConnection: vi.fn(), + activeRunnerConnectionCount: () => 0, + } as unknown as DurablePrpControlPlane, + endpoint, + startupDeadlineMs: 0, + recoveryGraceMs: 0, + }); + + const [ready, failure] = await Promise.allSettled([ + handle.ready, + handle.failure, + ]); + expect(ready).toMatchObject({ + status: "rejected", + reason: expect.objectContaining({ code: "runner_ingress_unavailable" }), + }); + expect(failure).toMatchObject({ + status: "rejected", + reason: expect.objectContaining({ code: "runner_ingress_unavailable" }), + }); + await handle.close(); + }); + + it("keeps an unobserved active-run failure from becoming process-global", async () => { + const endpoint = { + kind: "authenticated_websocket" as const, + websocketUrl: "ws://preview.invalid/api/runner/v1/connect/run-unobserved", + secretHeaders: [], + generation: "generation-1", + refresh: async () => endpoint, + close: async () => undefined, + }; + const unhandled = vi.fn(); + process.on("unhandledRejection", unhandled); + try { + const handle = connectRunnerPrpIngress({ + authority: { + attachWireConnection: vi.fn(), + activeRunnerConnectionCount: () => 0, + } as unknown as DurablePrpControlPlane, + endpoint, + startupDeadlineMs: 0, + recoveryGraceMs: 0, + }); + await expect(handle.ready).rejects.toMatchObject({ + code: "runner_ingress_unavailable", + }); + await new Promise((resolve) => setImmediate(resolve)); + expect(unhandled).not.toHaveBeenCalled(); + await handle.close(); + } finally { + process.off("unhandledRejection", unhandled); + } + }); + + it("does not report intentional close as an active-run ingress failure", async () => { + const endpoint = { + kind: "authenticated_websocket" as const, + websocketUrl: "wss://127.0.0.1:9/api/runner/v1/connect/run-1", + secretHeaders: [], + generation: "generation-1", + refresh: async () => endpoint, + close: vi.fn(async () => undefined), + }; + const handle = connectRunnerPrpIngress({ + authority: { + attachWireConnection: vi.fn(), + activeRunnerConnectionCount: () => 0, + } as unknown as DurablePrpControlPlane, + endpoint, + startupDeadlineMs: 60_000, + recoveryGraceMs: 60_000, + }); + const failureObserver = vi.fn(); + void handle.failure.catch(failureObserver); + void handle.ready.catch(() => undefined); + + await handle.close(); + await new Promise((resolve) => setImmediate(resolve)); + + expect(failureObserver).not.toHaveBeenCalled(); + expect(endpoint.close).toHaveBeenCalledOnce(); + }); +}); diff --git a/server/src/realtime/runner-prp-outbound.ts b/server/src/realtime/runner-prp-outbound.ts new file mode 100644 index 0000000000..6146b8b7fc --- /dev/null +++ b/server/src/realtime/runner-prp-outbound.ts @@ -0,0 +1,400 @@ +import WebSocket from "ws"; +import type { RunnerIngressEndpoint } from "@paperclipai/adapter-utils/runner-connectivity"; +import type { + DurablePrpControlPlane, + PrpWireConnection, + TransportCloseReason, +} from "../vendor/paperclip-runner/index.js"; + +const DEFAULT_STARTUP_DEADLINE_MS = 60_000; +const DEFAULT_RECOVERY_GRACE_MS = 120_000; +const MAX_FRAME_BYTES = 4 * 1024 * 1024; +const RECONNECT_DELAYS_MS = [250, 500, 1_000, 2_000, 5_000] as const; + +export type RunnerIngressFailureCode = + | "runner_ingress_unavailable" + | "runner_ingress_auth_failed"; + +export class RunnerIngressConnectionError extends Error { + constructor( + readonly code: RunnerIngressFailureCode, + message: string, + options?: ErrorOptions, + ) { + super(`${code}: ${message}`, options); + this.name = "RunnerIngressConnectionError"; + } +} + +export class WsJsonWireConnection implements PrpWireConnection { + #onJson: ((value: unknown) => void) | null = null; + #onClose = new Set<(reason: TransportCloseReason) => void>(); + #pendingJson: unknown[] = []; + #closeReason: TransportCloseReason | null = null; + + constructor(readonly socket: WebSocket) { + socket.on("message", (data, isBinary) => { + if (isBinary) { + this.close(1003); + return; + } + try { + const value = JSON.parse(data.toString()) as unknown; + if (this.#onJson === null) { + this.#pendingJson.push(value); + } else { + this.#onJson(value); + } + } catch { + this.close(1007); + } + }); + socket.on("close", (code, reason) => { + this.#notifyClose({ code, message: reason.toString() }); + }); + socket.on("error", (error) => { + this.#notifyClose({ message: "websocket_error", error }); + }); + } + + sendJson(value: unknown): void { + if (this.socket.readyState === WebSocket.OPEN) { + this.socket.send(JSON.stringify(value)); + } + } + + close(code = 1000): void { + if ( + this.socket.readyState === WebSocket.OPEN || + this.socket.readyState === WebSocket.CONNECTING + ) { + this.socket.close(code); + } + } + + onJson(listener: (value: unknown) => void): void { + this.#onJson = listener; + for (const value of this.#pendingJson.splice(0)) listener(value); + } + + onClose(listener: (reason: TransportCloseReason) => void): void { + if (this.#closeReason) listener(this.#closeReason); + else this.#onClose.add(listener); + } + + #notifyClose(reason: TransportCloseReason): void { + if (this.#closeReason) return; + this.#closeReason = reason; + for (const listener of this.#onClose) listener(reason); + this.#onClose.clear(); + } +} + +function validateEndpoint(endpoint: RunnerIngressEndpoint): void { + const url = new URL(endpoint.websocketUrl); + if ( + endpoint.kind !== "authenticated_websocket" || + url.protocol !== "wss:" || + url.username || + url.password || + url.search || + url.hash + ) { + throw new RunnerIngressConnectionError( + "runner_ingress_unavailable", + "The provider returned an invalid runner ingress endpoint.", + ); + } +} + +function endpointHeaders(endpoint: RunnerIngressEndpoint): Record { + const headers: Record = {}; + for (const header of endpoint.secretHeaders) { + if (!/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/.test(header.name) || !header.value) { + throw new RunnerIngressConnectionError( + "runner_ingress_unavailable", + "The provider returned an invalid runner ingress credential.", + ); + } + headers[header.name] = header.value; + } + return headers; +} + +async function delay(ms: number, signal: AbortSignal): Promise { + if (signal.aborted) return; + await new Promise((resolve) => { + const timer = setTimeout(resolve, ms); + signal.addEventListener( + "abort", + () => { + clearTimeout(timer); + resolve(); + }, + { once: true }, + ); + }); +} + +async function awaitWithinDeadline(input: { + operation: () => Promise; + deadline: number; + signal: AbortSignal; +}): Promise { + if (input.signal.aborted) { + throw new Error("runner ingress operation was cancelled"); + } + const remaining = input.deadline - Date.now(); + if (remaining <= 0) { + throw new Error("runner ingress operation deadline elapsed"); + } + let timer: ReturnType | undefined; + let onAbort: (() => void) | undefined; + const gate = new Promise((_resolve, reject) => { + timer = setTimeout( + () => reject(new Error("runner ingress operation deadline elapsed")), + remaining, + ); + onAbort = () => reject(new Error("runner ingress operation was cancelled")); + input.signal.addEventListener("abort", onAbort, { once: true }); + }); + const operation = Promise.resolve().then(input.operation); + try { + return await Promise.race([operation, gate]); + } finally { + if (timer !== undefined) clearTimeout(timer); + if (onAbort) input.signal.removeEventListener("abort", onAbort); + } +} + +async function openEndpoint(input: { + endpoint: RunnerIngressEndpoint; + signal: AbortSignal; +}): Promise<{ wire: WsJsonWireConnection; statusCode: number | null }> { + validateEndpoint(input.endpoint); + return await new Promise((resolve, reject) => { + let statusCode: number | null = null; + const socket = new WebSocket(input.endpoint.websocketUrl, { + headers: endpointHeaders(input.endpoint), + followRedirects: false, + perMessageDeflate: false, + maxPayload: MAX_FRAME_BYTES, + handshakeTimeout: 15_000, + }); + const wire = new WsJsonWireConnection(socket); + const onAbort = () => { + wire.close(1001); + reject( + new RunnerIngressConnectionError( + "runner_ingress_unavailable", + "Runner ingress connection was cancelled.", + ), + ); + }; + input.signal.addEventListener("abort", onAbort, { once: true }); + socket.once("unexpected-response", (_request, response) => { + statusCode = response.statusCode ?? null; + response.resume(); + input.signal.removeEventListener("abort", onAbort); + socket.terminate(); + reject( + Object.assign(new Error("runner_ingress_upgrade_rejected"), { + statusCode, + }), + ); + }); + socket.once("open", () => { + input.signal.removeEventListener("abort", onAbort); + resolve({ wire, statusCode }); + }); + socket.once("error", (error) => { + input.signal.removeEventListener("abort", onAbort); + reject(Object.assign(error, { statusCode })); + }); + }); +} + +function socketClosed(wire: WsJsonWireConnection): Promise { + return new Promise((resolve) => wire.onClose(() => resolve())); +} + +async function waitForPrpReady(input: { + isAuthenticated: () => boolean; + wire: WsJsonWireConnection; + deadline: number; + signal: AbortSignal; +}): Promise { + while ( + !input.signal.aborted && + input.wire.socket.readyState === WebSocket.OPEN && + Date.now() < input.deadline + ) { + if (input.isAuthenticated()) return true; + await delay(25, input.signal); + } + return false; +} + +export interface RunnerPrpOutboundHandle { + readonly ready: Promise; + /** Rejects only when active-run recovery has exhausted its fixed budget. */ + readonly failure: Promise; + close(): Promise; +} + +/** + * Keep a Paperclip-originated provider-ingress WebSocket attached to one PRP + * authority. Authentication is the readiness signal; HTTP success alone is not. + */ +export function connectRunnerPrpIngress(input: { + authority: DurablePrpControlPlane; + endpoint: RunnerIngressEndpoint; + startupDeadlineMs?: number; + recoveryGraceMs?: number; + random?: () => number; + onStateChange?: ( + state: "connecting" | "authenticated" | "reconnecting" | "failed" | "closed", + failureCode?: RunnerIngressFailureCode, + ) => void; +}): RunnerPrpOutboundHandle { + const abort = new AbortController(); + let activeWire: WsJsonWireConnection | null = null; + let endpoint = input.endpoint; + const startupDeadline = + Date.now() + (input.startupDeadlineMs ?? DEFAULT_STARTUP_DEADLINE_MS); + const recoveryGraceMs = input.recoveryGraceMs ?? DEFAULT_RECOVERY_GRACE_MS; + const random = input.random ?? Math.random; + let readyResolve!: () => void; + let readyReject!: (error: unknown) => void; + let readySettled = false; + const ready = new Promise((resolve, reject) => { + readyResolve = resolve; + readyReject = reject; + }); + let failureReject!: (error: unknown) => void; + const failure = new Promise((_resolve, reject) => { + failureReject = reject; + }); + // The owning native session normally races this promise against the provider + // turn. A turn can finish first, though, and a later preview outage must not + // become an unhandled process-level rejection. Keep the original promise + // rejected for explicit observers while registering an internal handler. + void failure.catch(() => undefined); + void ready.catch(() => undefined); + + const loop = (async () => { + let attempt = 0; + let recoveryDeadline = startupDeadline; + while (!abort.signal.aborted) { + try { + input.onStateChange?.(attempt === 0 ? "connecting" : "reconnecting"); + const opened = await openEndpoint({ endpoint, signal: abort.signal }); + activeWire = opened.wire; + const attachment = input.authority.attachWireConnection(opened.wire); + const authenticated = await waitForPrpReady({ + isAuthenticated: attachment.isAuthenticated, + wire: opened.wire, + deadline: recoveryDeadline, + signal: abort.signal, + }); + if (!authenticated) { + opened.wire.close(1008); + throw new RunnerIngressConnectionError( + "runner_ingress_auth_failed", + "Runner ingress did not complete PRP authentication before the deadline.", + ); + } + if (!readySettled) { + readySettled = true; + readyResolve(); + } + input.onStateChange?.("authenticated"); + attempt = 0; + await socketClosed(opened.wire); + activeWire = null; + recoveryDeadline = Date.now() + recoveryGraceMs; + } catch (error) { + activeWire = null; + // close() owns intentional shutdown. openEndpoint rejects its pending + // handshake when the abort signal fires, but that cancellation is not + // an active-run transport failure and must not reject the long-lived + // failure promise (an unobserved rejection here can terminate Node). + if (abort.signal.aborted) return; + let failure: unknown = error; + const statusCode = + typeof error === "object" && error !== null && "statusCode" in error + ? Number((error as { statusCode?: unknown }).statusCode) + : null; + if ( + (statusCode === 401 || statusCode === 403) && + Date.now() < recoveryDeadline + ) { + try { + const refreshedEndpoint = await awaitWithinDeadline({ + operation: () => endpoint.refresh(), + deadline: recoveryDeadline, + signal: abort.signal, + }); + // Preview credentials can rotate without changing the sandbox + // generation. Refresh every rejected credential, but never let a + // successful refresh reset or step past the fixed recovery budget. + if (Date.now() < recoveryDeadline) { + endpoint = refreshedEndpoint; + continue; + } + } catch (refreshError) { + if (abort.signal.aborted) return; + failure = refreshError; + } + } + if (Date.now() >= recoveryDeadline) { + const terminal = + failure instanceof RunnerIngressConnectionError + ? failure + : new RunnerIngressConnectionError( + statusCode === 401 || statusCode === 403 + ? "runner_ingress_auth_failed" + : "runner_ingress_unavailable", + "Runner ingress connection failed.", + { cause: failure }, + ); + if (!readySettled) { + readySettled = true; + readyReject(terminal); + } + input.onStateChange?.("failed", terminal.code); + failureReject(terminal); + return; + } + const base = RECONNECT_DELAYS_MS[ + Math.min(attempt, RECONNECT_DELAYS_MS.length - 1) + ]!; + attempt += 1; + await delay(Math.max(1, Math.round(base * (0.75 + random() * 0.5))), abort.signal); + } + } + })(); + + return { + ready, + failure, + async close() { + abort.abort(); + activeWire?.close(1001); + await loop; + await endpoint.close(); + input.onStateChange?.("closed"); + if (!readySettled) { + readySettled = true; + readyReject( + new RunnerIngressConnectionError( + "runner_ingress_unavailable", + "Runner ingress was closed before it became ready.", + ), + ); + } + }, + }; +} + +export const __runnerPrpOutboundTesting = { awaitWithinDeadline }; diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 37988c52cd..2155902915 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -1690,6 +1690,18 @@ export function agentRoutes( ); } + function assertProviderTraceSettingTransition( + req: Request, + nextRuntimeConfig: unknown, + previousRuntimeConfig?: unknown, + ): void { + const previousRaw = + asRecord(asRecord(previousRuntimeConfig)?.debug)?.providerTrace === "raw"; + const nextRaw = + asRecord(asRecord(nextRuntimeConfig)?.debug)?.providerTrace === "raw"; + if (previousRaw !== nextRaw) assertInstanceAdmin(req); + } + async function assertAgentDefaultEnvironmentSelection( companyId: string, environmentId: string | null | undefined, @@ -1785,19 +1797,6 @@ export function agentRoutes( return value as Record; } - function assertCanPersistRawProviderTrace( - req: Request, - runtimeConfig: unknown, - ): void { - const debug = asRecord(asRecord(runtimeConfig)?.debug); - if (debug?.providerTrace === "raw") { - // Raw provider payloads can contain prompts, tool inputs, and provider - // metadata. Apply the same instance-admin boundary on every persistence - // path so create/hire cannot bypass the PATCH guard. - assertInstanceAdmin(req); - } - } - function asNonEmptyString(value: unknown): string | null { if (typeof value !== "string") return null; const trimmed = value.trim(); @@ -3503,6 +3502,11 @@ export function agentRoutes( if (!rollbackConfig) { throw unprocessable("Invalid revision snapshot"); } + assertProviderTraceSettingTransition( + req, + rollbackConfig.runtimeConfig, + existing.runtimeConfig, + ); const rollbackAdapterType = assertKnownAdapterType( typeof rollbackConfig.adapterType === "string" ? rollbackConfig.adapterType @@ -3623,6 +3627,7 @@ export function agentRoutes( } = req.body; hireInput.adapterType = await assertSelectableAdapterType(hireInput.adapterType); const rawHireAdapterConfig = (hireInput.adapterConfig ?? {}) as Record; + assertProviderTraceSettingTransition(req, hireInput.runtimeConfig); assertFreshPaperclipRunnerProvider( hireInput.adapterType, rawHireAdapterConfig, @@ -3633,7 +3638,6 @@ export function agentRoutes( ); assertNoAgentAdapterConfigMutation(req, rawHireAdapterConfig); assertNoAgentRuntimeConfigAdapterConfigMutation(req, hireInput.runtimeConfig); - assertCanPersistRawProviderTrace(req, hireInput.runtimeConfig); const hiredAgentId = randomUUID(); const requestedAdapterConfig = applyCodexLocalKeyIsolation( companyId, @@ -3847,6 +3851,7 @@ export function agentRoutes( } = req.body; createInput.adapterType = await assertSelectableAdapterType(createInput.adapterType); const rawCreateAdapterConfig = (createInput.adapterConfig ?? {}) as Record; + assertProviderTraceSettingTransition(req, createInput.runtimeConfig); assertFreshPaperclipRunnerProvider( createInput.adapterType, rawCreateAdapterConfig, @@ -3857,7 +3862,6 @@ export function agentRoutes( ); assertNoAgentAdapterConfigMutation(req, rawCreateAdapterConfig); assertNoAgentRuntimeConfigAdapterConfigMutation(req, createInput.runtimeConfig); - assertCanPersistRawProviderTrace(req, createInput.runtimeConfig); const agentId = randomUUID(); const requestedAdapterConfig = applyCodexLocalKeyIsolation( companyId, @@ -4301,7 +4305,11 @@ export function agentRoutes( return; } assertNoAgentRuntimeConfigAdapterConfigMutation(req, runtimeConfig); - assertCanPersistRawProviderTrace(req, runtimeConfig); + assertProviderTraceSettingTransition( + req, + runtimeConfig, + existing.runtimeConfig, + ); requestedRuntimeConfig = runtimeConfig; } const touchesAdapterConfiguration = diff --git a/server/src/services/environment-config.ts b/server/src/services/environment-config.ts index 4bb9829f58..f0a14ec5c6 100644 --- a/server/src/services/environment-config.ts +++ b/server/src/services/environment-config.ts @@ -77,6 +77,13 @@ const fakeSandboxEnvironmentConfigSchema = z.object({ .default("ubuntu:24.04"), reuseLease: z.boolean().optional().default(false), streamRunLogs: z.boolean().optional(), + runnerLifecycleMode: z.enum(["inherit", "per_turn", "warm"]).optional(), + runnerIdleTimeoutMs: z.coerce + .number() + .int() + .min(1_000) + .max(86_400_000) + .optional(), archiveOnRelease: z.boolean().optional(), }).strict(); @@ -93,6 +100,13 @@ const pluginSandboxEnvironmentConfigSchema = z.object({ timeoutMs: z.coerce.number().int().min(1).max(86_400_000).optional(), reuseLease: z.boolean().optional().default(false), streamRunLogs: z.boolean().optional(), + runnerLifecycleMode: z.enum(["inherit", "per_turn", "warm"]).optional(), + runnerIdleTimeoutMs: z.coerce + .number() + .int() + .min(1_000) + .max(86_400_000) + .optional(), archiveOnRelease: z.boolean().optional(), }).catchall(z.unknown()); diff --git a/server/src/services/environment-driver-traits.ts b/server/src/services/environment-driver-traits.ts index 989cc4f6bf..aa465efba0 100644 --- a/server/src/services/environment-driver-traits.ts +++ b/server/src/services/environment-driver-traits.ts @@ -37,7 +37,7 @@ export interface EnvironmentDriverCapabilitySupport { const NO_CAPABILITY_SUPPORT: ReadonlySet = new Set(); -// The eight capability keys, written out here as literal strings. This module +// The nine capability keys, written out here as literal strings. This module // does not import `environment-runtime.ts` as a value (see the module comment // above), so it cannot read `SANDBOX_CAPABILITY_KEYS` from there. Keep this // list equal to that list. @@ -50,6 +50,7 @@ const ALL_CAPABILITY_SUPPORT: ReadonlySet = new Set; + if ( + candidate.outcome !== "created" && + candidate.outcome !== "resumed" && + candidate.outcome !== "replacement" + ) return null; + const resolvedProviderLeaseId = + typeof candidate.providerLeaseId === "string" && candidate.providerLeaseId + ? candidate.providerLeaseId + : providerLeaseId; + if (!resolvedProviderLeaseId) return null; + const reason = candidate.reason; + if ( + reason !== undefined && + reason !== "not_found" && + reason !== "expired" && + reason !== "identity_mismatch" && + reason !== "resume_failed" + ) return null; + return { + outcome: candidate.outcome, + providerLeaseId: resolvedProviderLeaseId, + ...(typeof candidate.previousProviderLeaseId === "string" + ? { previousProviderLeaseId: candidate.previousProviderLeaseId } + : {}), + ...(reason ? { reason } : {}), + }; +} + /** * Compute the tail of `final` that the provider did NOT already stream. * @@ -331,11 +366,39 @@ export async function resolveEnvironmentExecutionTarget(input: { shellCommand, remoteCwd, enableSandboxDuplexBridge, + runnerLifecyclePolicy: + parsed.config.runnerLifecycleMode === "warm" + ? { + mode: "warm", + idleTimeoutMs: + typeof parsed.config.runnerIdleTimeoutMs === "number" + ? parsed.config.runnerIdleTimeoutMs + : 300_000, + } + : parsed.config.runnerLifecycleMode === "per_turn" + ? { mode: "per_turn", idleTimeoutMs: null } + : null, + reusableLeaseConfigured: parsed.config.reuseLease === true, + sandboxLeaseAcquisition: sandboxLeaseAcquisitionFromMetadata( + input.lease?.metadata?.sandboxLeaseAcquisition, + input.lease?.providerLeaseId, + ), // Attach the host duplex observability recorder next to the runner. The bridge // binds it to the fixed observability surface. Absent keeps the no-op // default, so the surface stays inert on a run with no injected recorder. duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null, ...(effectiveCapabilities ? { effectiveCapabilities: Object.freeze({ ...effectiveCapabilities }) } : {}), + ...(input.environmentRuntime?.getRunnerIngressEndpoint && input.lease + ? { + getRunnerIngressEndpoint: ({ port, path }) => + input.environmentRuntime!.getRunnerIngressEndpoint({ + environment: input.environment as Environment, + lease: input.lease!, + port, + path, + }), + } + : {}), environmentId: input.environment.id ?? null, leaseId: input.leaseId ?? null, timeoutMs, diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index ccabc7e51d..0c08deb714 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -31,6 +31,7 @@ import { buildEnvironmentLeaseContext, type EnvironmentRuntimeLeaseRecord, type EnvironmentRuntimeService, + type ProviderResourceDisposition, } from "./environment-runtime.js"; import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js"; import { @@ -583,6 +584,17 @@ export function environmentRunOrchestrator( agentId: string; status?: Extract; failureReason?: string; + /** Explicit paperclip_runner resource lifecycle. Omitted for legacy adapters. */ + providerResourceDisposition?: ProviderResourceDisposition; + nativeLifecycleTelemetry?: { + provider: string; + harness: string; + lifecycleMode: "per_turn" | "warm"; + sandboxResource: + | "keep_running" + | "stop_and_reuse" + | "destroy_after_turn"; + }; }): Promise { const status = input.status ?? "released"; const result: EnvironmentReleaseResult = { released: [], errors: [] }; @@ -593,6 +605,7 @@ export function environmentRunOrchestrator( input.heartbeatRunId, status, (leaseId, error) => result.errors.push({ leaseId, error }), + input.providerResourceDisposition, ); } catch (err) { result.errors.push({ leaseId: "*", error: err }); @@ -621,6 +634,8 @@ export function environmentRunOrchestrator( status: released.lease.status, cleanupStatus: released.lease.cleanupStatus, failureReason: input.failureReason ?? released.lease.failureReason, + providerResourceDisposition: + input.providerResourceDisposition ?? "legacy_default", }, }); } catch { diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index 4d4b152066..5451d53c0c 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -15,6 +15,7 @@ import type { } from "@paperclipai/shared"; import { resolveDeclaredSandboxCapabilities } from "@paperclipai/shared"; import type { EffectiveExecutionCapabilities } from "@paperclipai/adapter-utils/execution-target"; +import type { RunnerIngressEndpoint } from "@paperclipai/adapter-utils/runner-connectivity"; import type { CommandManagedDuplexChannel, } from "@paperclipai/adapter-utils/command-managed-runtime"; @@ -34,6 +35,7 @@ import { } from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; import { environmentService } from "./environments.js"; import { instanceSettingsService } from "./instance-settings.js"; +import { verifyNativeHarnessBackupStamp } from "./native-runtime/native-harness-backup-stamp.js"; import { collectEnvironmentSecretRefs, parseEnvironmentDriverConfig, @@ -107,6 +109,7 @@ export const SANDBOX_CAPABILITY_KEYS = [ "incrementalSessionOutput", "concurrentSyncOperations", "duplexCommandStream", + "runnerWebSocketIngress", ] as const; export type SandboxCapabilityKey = (typeof SANDBOX_CAPABILITY_KEYS)[number]; @@ -124,6 +127,7 @@ const SANDBOX_CAPABILITY_OPT_IN_KEYS: ReadonlySet = new Se "incrementalSessionOutput", "concurrentSyncOperations", "duplexCommandStream", + "runnerWebSocketIngress", ]); /** @@ -180,6 +184,7 @@ const SANDBOX_CAPABILITY_PREREQUISITE_METHODS: Record; @@ -583,6 +595,9 @@ export interface EnvironmentRuntimeDriver { openDuplexChannel?( input: EnvironmentDriverOpenDuplexChannelInput, ): Promise; + getRunnerIngressEndpoint?( + input: EnvironmentDriverRunnerIngressInput, + ): Promise; /** True when the lease's plugin worker advertises both sync verbs. */ supportsSync?(input: EnvironmentDriverLeaseInput): boolean; /** @@ -678,12 +693,57 @@ export class SandboxOrphanCleanupWriteError extends Error { } } +/** A reusable sandbox could not be resumed, but has not been proven lost. */ +export class ReusableSandboxResumeError extends Error { + readonly provider: string; + readonly providerLeaseId: string; + + constructor(input: { + provider: string; + providerLeaseId: string; + cause?: unknown; + }) { + super( + `Reusable sandbox lease "${input.providerLeaseId}" could not be resumed; ` + + "the lease was preserved and no replacement was created.", + input.cause === undefined ? undefined : { cause: input.cause }, + ); + this.name = "ReusableSandboxResumeError"; + this.provider = input.provider; + this.providerLeaseId = input.providerLeaseId; + } +} + +export class RunnerHarnessBackupUnavailableError extends Error { + readonly providerLeaseId: string; + + constructor(providerLeaseId: string) { + super( + `runner_harness_backup_unavailable: reusable sandbox "${providerLeaseId}" ` + + "was confirmed lost, but no complete verified failover backup is available", + ); + this.name = "RunnerHarnessBackupUnavailableError"; + this.providerLeaseId = providerLeaseId; + } +} + export interface EnvironmentRuntimeLeaseRecord { environment: Environment; lease: EnvironmentLease; leaseContext: ReturnType; } +/** + * Host-side decision for the provider resource after a run. This is kept + * separate from heartbeat status: a failed turn can still leave a reusable + * sandbox resumable, while a disposable successful turn must destroy it. + * An omitted disposition preserves the legacy adapter behavior. + */ +export type ProviderResourceDisposition = + | "keep_running" + | "stop_and_retain" + | "destroy"; + const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS = 5_000; const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS = 100; @@ -704,6 +764,24 @@ function delay(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); } +function transientSandboxResumeFailure(error: unknown): boolean { + const candidate = error && typeof error === "object" + ? error as Record + : {}; + const status = typeof candidate.status === "number" + ? candidate.status + : typeof candidate.statusCode === "number" + ? candidate.statusCode + : null; + if (status === 429 || (status !== null && status >= 500)) return true; + const code = typeof candidate.code === "string" ? candidate.code.toUpperCase() : ""; + if (["ETIMEDOUT", "ECONNRESET", "ECONNREFUSED", "EAI_AGAIN", "ENETUNREACH"].includes(code)) { + return true; + } + const message = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase(); + return /\b(timeout|timed out|rate limit|temporar|network|connection reset|service unavailable)\b/.test(message); +} + function getLeaseDriverKey( lease: Pick, environment: Pick | null, @@ -1794,23 +1872,40 @@ function createSandboxEnvironmentDriver( : null; let providerLease: PluginEnvironmentLease | null = null; + let replacementReason: + | "not_found" + | "expired" + | "identity_mismatch" + | "resume_failed" + | undefined; if (reusableLease?.providerLeaseId) { // The `supportsReusableLeases` check above reads a snapshot of the // worker methods. The runtime then does asynchronous database work // (list, fingerprint, obsolete-lease cleanup) before this dispatch. A // worker restart in that window can drop `environmentResumeLease` // while the snapshot still marks the method verified. Re-check the - // live worker here and fail closed when the method is absent: skip the - // resume, destroy the stale reusable lease, and acquire a fresh lease - // below. The runtime never dispatches a resume the live worker cannot - // serve. + // live worker here and fail closed when the method is absent. The + // runtime preserves the recorded lease and never dispatches a resume + // the live worker cannot serve. const workerVerifiesResume = pluginWorkerVerifiesLifecycleMethod( pluginProvider.resolved.plugin.id, "environmentResumeLease", ); - if (workerVerifiesResume) { - try { - const resumed = await pluginWorkerManager.call( + if (!workerVerifiesResume) { + throw new ReusableSandboxResumeError({ + provider: parsed.config.provider, + providerLeaseId: reusableLease.providerLeaseId, + }); + } + try { + const resumeDeadline = Date.now() + 60_000; + const configuredResumeTimeoutMs = + resolvePluginSandboxRpcTimeoutMs(workerConfig) ?? 60_000; + let retryDelayMs = 250; + let resumed: PluginEnvironmentLease; + while (true) { + try { + resumed = await pluginWorkerManager.call( pluginProvider.resolved.plugin.id, "environmentResumeLease", { @@ -1822,21 +1917,65 @@ function createSandboxEnvironmentDriver( providerLeaseId: reusableLease.providerLeaseId, leaseMetadata: reusableLease.metadata ?? undefined, }, - resolvePluginSandboxRpcTimeoutMs(workerConfig), + Math.min( + configuredResumeTimeoutMs, + Math.max(1, resumeDeadline - Date.now()), + ), ); - providerLease = - typeof resumed.providerLeaseId === "string" && resumed.providerLeaseId.length > 0 - ? resumed - : null; - } catch { - providerLease = null; + break; + } catch (error) { + if ( + !transientSandboxResumeFailure(error) || + Date.now() + retryDelayMs * 1.25 >= resumeDeadline + ) throw error; + const jitteredDelayMs = Math.max( + 1, + Math.round(retryDelayMs * (0.75 + Math.random() * 0.5)), + ); + await delay(jitteredDelayMs); + retryDelayMs = Math.min(retryDelayMs * 2, 5_000); + } } + providerLease = + typeof resumed.providerLeaseId === "string" && resumed.providerLeaseId.length > 0 + ? resumed + : null; + if (!providerLease) { + const sentinel = isRecord(resumed.metadata?.workspaceSentinel) + ? resumed.metadata.workspaceSentinel + : null; + replacementReason = sentinel && sentinel.result !== "matched" + ? "identity_mismatch" + : resumed.metadata?.expired === true + ? "not_found" + : "expired"; + } + } catch (error) { + throw new ReusableSandboxResumeError({ + provider: parsed.config.provider, + providerLeaseId: reusableLease.providerLeaseId, + cause: error, + }); } if (!providerLease) { + if ( + input.adapterType === "paperclip_runner" && + !verifyNativeHarnessBackupStamp( + reusableLease.metadata?.nativeHarnessBackup, + reusableLease.providerLeaseId, + ) + ) { + throw new RunnerHarnessBackupUnavailableError( + reusableLease.providerLeaseId, + ); + } + // The verified, lease-bound backup authorizes destructive + // replacement. Keep the existing sandbox intact when validation + // fails so the only recoverable provider state is not lost. await destroyReusableSandboxLease({ environment: input.environment, lease: reusableLease, - failureReason: workerVerifiesResume ? "resume_failed" : "resume_capability_lost", + failureReason: replacementReason ?? "resume_failed", }); } } @@ -1906,6 +2045,21 @@ function createSandboxEnvironmentDriver( sandboxProviderPlugin: true, ...sandboxConfigForLeaseMetadata(storedConfig), ...sanitizedProviderMetadata, + sandboxLeaseAcquisition: providerLease + ? { + outcome: "resumed", + } + : reusableLease?.providerLeaseId + ? { + outcome: "replacement", + reason: replacementReason ?? "resume_failed", + } + : { + outcome: "created", + }, + ...(reusableLease?.metadata?.nativeHarnessBackup + ? { nativeHarnessBackup: reusableLease.metadata.nativeHarnessBackup } + : {}), ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}), }; try { @@ -2072,10 +2226,10 @@ function createSandboxEnvironmentDriver( }); } catch (error) { if (reusableLease) { - await destroyReusableSandboxLease({ - environment: input.environment, - lease: reusableLease, - failureReason: "resume_failed", + throw new ReusableSandboxResumeError({ + provider: parsed.config.provider, + providerLeaseId: reusableLease.providerLeaseId!, + cause: error, }); } throw error; @@ -2112,6 +2266,22 @@ function createSandboxEnvironmentDriver( driver: input.environment.driver, executionWorkspaceMode: input.executionWorkspaceMode, ...providerLease.metadata, + sandboxLeaseAcquisition: + reusableLease && providerLease.providerLeaseId === reusableLease.providerLeaseId + ? { + outcome: "resumed", + } + : reusableLease?.providerLeaseId + ? { + outcome: "replacement", + reason: "resume_failed", + } + : { + outcome: "created", + }, + ...(reusableLease?.metadata?.nativeHarnessBackup + ? { nativeHarnessBackup: reusableLease.metadata.nativeHarnessBackup } + : {}), ...(reusableScope ? { reusableSandboxLease: reusableScope } : {}), }; try { @@ -2573,6 +2743,91 @@ function createSandboxEnvironmentDriver( return await resolveSandboxCapabilitiesForLease(input); }, + async getRunnerIngressEndpoint(input) { + if (!input.lease.metadata?.sandboxProviderPlugin || !pluginWorkerManager) { + throw new Error("Sandbox driver does not support runner ingress for this lease."); + } + const pluginId = readString(input.lease.metadata.pluginId); + const providerKey = readString(input.lease.metadata.provider); + if (!pluginId || !providerKey || !input.lease.providerLeaseId) { + throw new Error("Sandbox runner ingress is missing its provider identity."); + } + const config = await resolvePluginSandboxRuntimeConfig({ + environment: input.environment, + lease: input.lease, + provider: providerKey, + }); + const sanitizedConfig = stripSandboxProviderEnvelope( + config as SandboxEnvironmentConfig, + ); + const acquire = async (): Promise => { + const result = await pluginWorkerManager.call( + pluginId, + "environmentRunnerIngressEndpoint", + { + driverKey: providerKey, + companyId: input.lease.companyId, + environmentId: input.environment.id, + issueId: input.lease.issueId, + config: sanitizedConfig, + lease: { + providerLeaseId: input.lease.providerLeaseId, + metadata: input.lease.metadata ?? undefined, + expiresAt: input.lease.expiresAt?.toISOString() ?? null, + }, + port: input.port, + path: input.path, + }, + resolvePluginSandboxRpcTimeoutMs(sanitizedConfig), + ); + const endpointUrl = new URL(result.websocketUrl); + if ( + result.kind !== "authenticated_websocket" || + endpointUrl.protocol !== "wss:" || + endpointUrl.username || + endpointUrl.password || + endpointUrl.search || + endpointUrl.hash || + endpointUrl.pathname !== input.path || + !result.generation + ) { + throw new Error("Sandbox provider returned an invalid runner ingress endpoint."); + } + const secretHeaders = result.secretHeaders.map((header) => { + if (!/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/.test(header.name) || !header.value) { + throw new Error("Sandbox provider returned an invalid runner ingress secret header."); + } + const secretHeader = { name: header.name } as { + name: string; + readonly value: string; + toJSON(): { name: string; value: "[REDACTED]" }; + }; + Object.defineProperty(secretHeader, "value", { + enumerable: false, + configurable: false, + writable: false, + value: header.value, + }); + Object.defineProperty(secretHeader, "toJSON", { + enumerable: false, + configurable: false, + writable: false, + value: () => ({ name: header.name, value: "[REDACTED]" as const }), + }); + return Object.freeze(secretHeader); + }); + return { + kind: "authenticated_websocket", + websocketUrl: endpointUrl.toString(), + secretHeaders: Object.freeze(secretHeaders), + generation: result.generation, + refresh: acquire, + close: async () => undefined, + }; + }; + return await acquire(); + }, + async destroyRunLease(input) { return await destroyReusableSandboxLease({ environment: input.environment, @@ -2897,6 +3152,8 @@ const INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS = new Set([ "remoteCwd", "shellCommand", "sandboxProviderPlugin", + "sandboxLeaseAcquisition", + "nativeHarnessBackup", ]); // Drop the host-internal and per-lease runtime keys from a sandbox config @@ -3386,6 +3643,7 @@ export function environmentRuntimeService( heartbeatRunId: string, status: Extract = "released", onLeaseReleaseError?: (leaseId: string, error: unknown) => void, + providerResourceDisposition?: ProviderResourceDisposition, ): Promise { const leaseRows = await db .select() @@ -3413,14 +3671,72 @@ export function environmentRuntimeService( if (!environment) continue; const leaseSnapshot = toEnvironmentLeaseSnapshot(leaseRow); + if ( + providerResourceDisposition === "keep_running" && + leaseSnapshot.leasePolicy === "reuse_by_environment" + ) { + const lease = await environmentsSvc.releaseLease( + leaseRow.id, + "retained", + { cleanupStatus: "success" }, + ); + if (lease) { + released.push({ + environment, + lease, + leaseContext: { + executionWorkspaceId: lease.executionWorkspaceId, + executionWorkspaceMode: + (lease.metadata?.executionWorkspaceMode as ExecutionWorkspace["mode"] | null | undefined) ?? null, + }, + }); + } + continue; + } const driver = getDriver(getLeaseDriverKey(leaseSnapshot, environment)); - const lease = driver - ? await driver.releaseRunLease({ + if ( + providerResourceDisposition === "keep_running" && + leaseSnapshot.leasePolicy !== "reuse_by_environment" + ) { + throw new Error( + `Cannot keep non-reusable environment lease "${leaseSnapshot.id}" running.`, + ); + } + if ( + providerResourceDisposition === "destroy" && + leaseSnapshot.metadata?.sandboxLeaseAcquisition && + (!leaseSnapshot.providerLeaseId || + !verifyNativeHarnessBackupStamp( + leaseSnapshot.metadata.nativeHarnessBackup, + leaseSnapshot.providerLeaseId, + )) + ) { + throw new RunnerHarnessBackupUnavailableError( + leaseSnapshot.providerLeaseId ?? leaseSnapshot.id, + ); + } + const lease = providerResourceDisposition === "destroy" && driver?.destroyRunLease + ? await driver.destroyRunLease({ environment, lease: leaseSnapshot, - status, + failureReason: "paperclip_runner_destroy_after_turn", }) - : await environmentsSvc.releaseLease(leaseRow.id, status); + : driver + ? await driver.releaseRunLease({ + environment, + lease: leaseSnapshot, + // A stopped reusable provider resource must remain eligible + // for exact-lease resume independently of turn outcome. + status: + providerResourceDisposition === "stop_and_retain" && + leaseSnapshot.leasePolicy === "reuse_by_environment" + ? "released" + : status, + }) + : await environmentsSvc.releaseLease( + leaseRow.id, + providerResourceDisposition === "destroy" ? "expired" : status, + ); if (!lease) continue; released.push({ @@ -3763,6 +4079,26 @@ export function environmentRuntimeService( } return await driver.openDuplexChannel(input); }, + + async getRunnerIngressEndpoint( + input: EnvironmentDriverRunnerIngressInput, + ): Promise { + const driver = requireDriverKey( + getLeaseDriverKey(input.lease, input.environment), + ); + const effective = await driver.resolveCapabilities(input); + if (effective.runnerWebSocketIngress !== true) { + throw new Error( + "Sandbox lease does not grant runner WebSocket ingress.", + ); + } + if (!driver.getRunnerIngressEndpoint) { + throw new Error( + `Environment driver "${driver.driver}" does not support runner ingress.`, + ); + } + return await driver.getRunnerIngressEndpoint(input); + }, }; } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index ee51cb9372..0ae8506387 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -392,7 +392,10 @@ import { extractSkillMentionIds, isUuidLike } from "@paperclipai/shared"; import { evaluateCodexCredentialReadiness } from "@paperclipai/adapter-codex-local/server"; import { environmentService } from "./environments.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; -import { environmentRuntimeService } from "./environment-runtime.js"; +import { + environmentRuntimeService, + type ProviderResourceDisposition, +} from "./environment-runtime.js"; import { skillVersionSelectionMap } from "./runtime-skill-selections.js"; import { environmentRunOrchestrator } from "./environment-run-orchestrator.js"; import { isUnsafeSessionWorkspaceCwd } from "./session-workspace-cwd.js"; @@ -1674,6 +1677,48 @@ export function leaseReleaseStatusForRunStatus( return status === "failed" || status === "timed_out" ? "failed" : "released"; } +export interface NativeSandboxLifecycle { + runnerProcess: "per_turn" | "warm"; + sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; + failoverBackup: "verified"; +} + +export function resolveNativeSandboxLifecycle(input: { + adapterType: string; + lifecyclePolicy: + | { mode: "per_turn"; idleTimeoutMs: null } + | { mode: "warm"; idleTimeoutMs: number }; + target: { + kind: "local" | "remote"; + transport?: string; + reusableLeaseConfigured?: boolean; + effectiveCapabilities?: { reusableLeases: boolean } | null; + } | null; +}): NativeSandboxLifecycle | null { + if ( + input.adapterType !== "paperclip_runner" || + input.target?.kind !== "remote" || + input.target.transport !== "sandbox" + ) + return null; + const reusableLease = + input.target.reusableLeaseConfigured === true && + input.target.effectiveCapabilities?.reusableLeases === true; + if (input.lifecyclePolicy.mode === "warm" && !reusableLease) { + throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease"); + } + return { + runnerProcess: input.lifecyclePolicy.mode, + sandboxResource: + input.lifecyclePolicy.mode === "warm" + ? "keep_running" + : reusableLease + ? "stop_and_reuse" + : "destroy_after_turn", + failoverBackup: "verified", + }; +} + export function applyPersistedExecutionWorkspaceConfig(input: { config: Record; workspaceConfig: ExecutionWorkspaceConfig | null; @@ -8611,6 +8656,13 @@ export function heartbeatService( agentId: string; status: string | null | undefined; failureReason?: string | null; + providerResourceDisposition?: ProviderResourceDisposition; + nativeLifecycleTelemetry?: { + provider: string; + harness: string; + lifecycleMode: "per_turn" | "warm"; + sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; + }; }) { const releaseResult = await envOrchestrator .releaseForRun({ @@ -8619,6 +8671,8 @@ export function heartbeatService( agentId: input.agentId, status: leaseReleaseStatusForRunStatus(input.status), failureReason: input.failureReason ?? undefined, + providerResourceDisposition: input.providerResourceDisposition, + nativeLifecycleTelemetry: input.nativeLifecycleTelemetry, }) .catch((err) => { logger.warn( @@ -17408,6 +17462,17 @@ export function heartbeatService( activeRunExecutions.add(run.id); let runScratch: HeartbeatRunScratch | null = null; let nativeSessionResumeScheduled = false; + let providerResourceDispositionForRun: + ProviderResourceDisposition | undefined; + let nativeLifecycleTelemetryForRun: + | { + provider: string; + harness: string; + lifecycleMode: "per_turn" | "warm"; + sandboxResource: + "keep_running" | "stop_and_reuse" | "destroy_after_turn"; + } + | undefined; let providerTraceCapture: Awaited< ReturnType > | null = null; @@ -19879,7 +19944,26 @@ export function heartbeatService( : 300_000, } : { mode: "per_turn" as const, idleTimeoutMs: null }; - const effectiveLifecyclePolicy = agentLifecyclePolicy; + const environmentLifecyclePolicy = + executionTarget?.kind === "remote" && + executionTarget.transport === "sandbox" + ? executionTarget.runnerLifecyclePolicy ?? null + : null; + const effectiveLifecyclePolicy = + environmentLifecyclePolicy ?? agentLifecyclePolicy; + if ( + effectiveLifecyclePolicy.mode === "warm" && + executionTarget?.kind === "remote" && + executionTarget.transport === "sandbox" && + ( + executionTarget.reusableLeaseConfigured !== true || + executionTarget.effectiveCapabilities?.reusableLeases !== true + ) + ) { + throw new Error( + "runner_warm_environment_requires_reusable_lease", + ); + } const persistedProfile = persistedRunnerProfile; if (persistedNativeExecutionInput) { nativeExecution = persistedNativeExecutionInput; @@ -19977,10 +20061,30 @@ export function heartbeatService( : {}, } : null, + provider: + nativeRuntimeResolution.profile.backend === "opencode_server" + ? "opencode" + : nativeRuntimeResolution.profile.backend === "acpx_runtime" + ? "acpx" + : "codex", + ...(nativeRuntimeResolution.profile.backend === "acpx_runtime" + ? { + acpxAgent: parseObject(runtimeConfig).acpxAgent as + "pi" | "claude" | "codex", + } + : {}), codexApprovalPolicy: resolvePaperclipRunnerPermissionMode( "codex", parseObject(agent.adapterConfig).codexPermissionMode, ) as "never" | "on-request" | "untrusted", + opencodePermissionMode: resolvePaperclipRunnerPermissionMode( + "opencode", + parseObject(runtimeConfig).opencodePermissionMode, + ) as "allow" | "ask" | "deny", + acpxPermissionMode: resolvePaperclipRunnerPermissionMode( + "acpx", + parseObject(runtimeConfig).acpxPermissionMode, + ) as "approve-all" | "approve-reads" | "deny-all", model: typeof parseObject(agent.adapterConfig).model === "string" ? String(parseObject(agent.adapterConfig).model) @@ -20015,6 +20119,44 @@ export function heartbeatService( } } } + const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({ + adapterType: agent.adapterType, + lifecyclePolicy: nativeExecution.session.lifecyclePolicy, + target: executionTarget, + }); + if (nativeSandboxLifecycle) { + nativeLifecycleTelemetryForRun = { + provider: nativeExecution.provider.kind, + harness: nativeExecution.session.driverKind, + lifecycleMode: nativeExecution.session.lifecyclePolicy.mode, + sandboxResource: nativeSandboxLifecycle.sandboxResource, + }; + const selectedLifecycleSpan = getStartupTracer( + "paperclip.environment-lifecycle", + ).startSpan("sandbox.lifecycle.selected", { + attributes: { + "paperclip.native.span.provider": nativeExecution.provider.kind, + "paperclip.native.span.harness": + nativeExecution.session.driverKind, + "paperclip.native.span.lifecycle_mode": + nativeExecution.session.lifecyclePolicy.mode, + "paperclip.native.span.sandbox_resource": + nativeSandboxLifecycle.sandboxResource, + "paperclip.native.span.outcome": "selected", + "paperclip.native.span.bytes_transferred": 0, + }, + }); + selectedLifecycleSpan.end(); + } + providerResourceDispositionForRun = + nativeSandboxLifecycle?.sandboxResource === "keep_running" + ? "keep_running" + : nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse" + ? "stop_and_retain" + : nativeSandboxLifecycle?.sandboxResource === + "destroy_after_turn" + ? "destroy" + : undefined; await db.transaction(async (tx) => { const lockedRun = await tx .select() @@ -20473,6 +20615,27 @@ export function heartbeatService( } : {}), }, + runnerExecutionTarget: executionTarget, + enableRunnerPreviewIngress: + resolvedInstanceSettings.experimental + .enableRunnerPreviewIngress === true, + runnerPublicUrl: + runtimeEnv.PAPERCLIP_RUNNER_PUBLIC_URL?.trim() || null, + runnerCaBundlePath: + runtimeEnv.PAPERCLIP_RUNNER_CA_BUNDLE_PATH?.trim() || + null, + runnerRemoteBinaryPath: + runtimeEnv.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH?.trim() || + null, + runnerRemoteCodexPath: + runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH?.trim() || + null, + runnerRemoteCodexNpmSpec: + runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC?.trim() || + null, + runnerRemoteProviderPackPath: + runtimeEnv.PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH?.trim() || + null, enqueueWakeup, onSpawn: async (meta) => { markDispatchStarted(); @@ -21786,6 +21949,8 @@ export function heartbeatService( agentId: run.agentId, status: latestRun?.status, failureReason: latestRun?.error ?? undefined, + providerResourceDisposition: providerResourceDispositionForRun, + nativeLifecycleTelemetry: nativeLifecycleTelemetryForRun, }); await releaseRuntimeServicesForRun(run.id).catch(() => undefined); } diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index 1776fb2bdd..78c94b8dd9 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -249,6 +249,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta enableWorkspaceDirtyQuarantineRepair: parsed.data.enableWorkspaceDirtyQuarantineRepair ?? true, enableOwnerInstanceAdmin: parsed.data.enableOwnerInstanceAdmin ?? false, enableSandboxDuplexBridge: parsed.data.enableSandboxDuplexBridge ?? false, + enableRunnerPreviewIngress: parsed.data.enableRunnerPreviewIngress ?? false, enableWorktreeRunExecution: parsed.data.enableWorktreeRunExecution ?? false, worktreeRunExecutionActivatedAt: parsed.data.worktreeRunExecutionActivatedAt ?? null, worktreeRunExecutionActivationInstanceId: @@ -288,6 +289,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta enableWorkspaceDirtyQuarantineRepair: true, enableOwnerInstanceAdmin: false, enableSandboxDuplexBridge: false, + enableRunnerPreviewIngress: false, enableWorktreeRunExecution: false, worktreeRunExecutionActivatedAt: null, worktreeRunExecutionActivationInstanceId: null, diff --git a/server/src/services/native-runtime/native-execution-input.ts b/server/src/services/native-runtime/native-execution-input.ts index ff54df3f6f..b43e0fe839 100644 --- a/server/src/services/native-runtime/native-execution-input.ts +++ b/server/src/services/native-runtime/native-execution-input.ts @@ -1,12 +1,18 @@ import type { + NativeAcpxAgent, + NativeAcpxPermissionMode, NativeCodexApprovalPolicy, NativeExecutionInputV4, NativeInteractionResponseEnvelope, + NativeOpenCodePermissionMode, NativePlanningContext, NativeRuntimeContextSnapshot, StrictCompletionContractInput, } from "../../vendor/paperclip-runner/index.js"; -import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { + parseNativeExecutionInput, + resolveQualifiedAcpxProfile, +} from "../../vendor/paperclip-runner/index.js"; import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils"; /** Closed constructor: callers cannot spread legacy context or environment data. */ @@ -38,7 +44,11 @@ export function buildNativeExecutionInput(input: { branchName: string | null; }; normalizedSessionId: string | null; + provider?: "codex" | "opencode" | "acpx"; + acpxAgent?: NativeAcpxAgent; codexApprovalPolicy?: NativeCodexApprovalPolicy; + opencodePermissionMode?: NativeOpenCodePermissionMode; + acpxPermissionMode?: NativeAcpxPermissionMode; model?: string | null; lifecyclePolicy?: NativeExecutionInputV4["session"]["lifecyclePolicy"]; executionMode?: "default" | "plan"; @@ -57,6 +67,12 @@ export function buildNativeExecutionInput(input: { } const executionMode = input.executionMode ?? (input.issue.workMode === "planning" ? "plan" : "default"); + const acpxProfile = input.provider === "acpx" + ? resolveQualifiedAcpxProfile( + input.acpxAgent ?? "pi", + input.model ?? "", + ) + : null; const wakePrompt = renderPaperclipWakePrompt(input.wakePayload, { resumedSession: input.resumedSession === true, suppressIssueDescription: input.taskPrompt.trim().length > 0, @@ -90,15 +106,44 @@ export function buildNativeExecutionInput(input: { }, session: { normalizedSessionId: input.normalizedSessionId, - driverKind: "codex_app_server", + driverKind: input.provider === "opencode" + ? "opencode_server" + : input.provider === "acpx" + ? "acpx_runtime" + : "codex_app_server", protocolVersion: 1, lifecyclePolicy: input.lifecyclePolicy ?? { mode: "per_turn", idleTimeoutMs: null }, }, - provider: { - kind: "codex", - model: input.model ?? null, - approvalPolicy: input.codexApprovalPolicy ?? "never", - }, + provider: input.provider === "acpx" + ? { + kind: "acpx", + agent: acpxProfile!.agent, + model: input.model, + permissionMode: input.acpxPermissionMode ?? "approve-all", + profile: { + driverKind: acpxProfile!.driverKind, + protocolVersion: acpxProfile!.protocolVersion, + acpxVersion: acpxProfile!.acpxVersion, + agent: acpxProfile!.agent, + agentProfileVersion: acpxProfile!.agentProfileVersion, + agentServerPackage: acpxProfile!.agentServerPackage, + agentServerVersion: acpxProfile!.agentServerVersion, + agentRuntimePackage: acpxProfile!.agentRuntimePackage, + agentRuntimeVersion: acpxProfile!.agentRuntimeVersion, + commandDigest: acpxProfile!.commandDigest, + }, + } + : input.provider === "opencode" + ? { + kind: "opencode", + model: input.model, + permissionMode: input.opencodePermissionMode ?? "allow", + } + : { + kind: "codex", + model: input.model ?? null, + approvalPolicy: input.codexApprovalPolicy ?? "never", + }, completionContract: input.completionContract, interactionResponses: input.interactionResponses ?? [], credentialBindings: [], diff --git a/server/src/services/native-runtime/native-harness-backup-stamp.ts b/server/src/services/native-runtime/native-harness-backup-stamp.ts new file mode 100644 index 0000000000..e281bfce71 --- /dev/null +++ b/server/src/services/native-runtime/native-harness-backup-stamp.ts @@ -0,0 +1,137 @@ +import { createHash } from "node:crypto"; +import { + existsSync, + lstatSync, + readFileSync, + readdirSync, + readlinkSync, +} from "node:fs"; +import { resolve } from "node:path"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; + +export interface NativeHarnessBackupStamp { + schema: "paperclip.native-harness-backup-stamp.v1"; + normalizedSessionId: string; + runnerInstanceId: string; + manifestSha256: string; + completedAt: string; +} + +function stateRoot(normalizedSessionId: string): string { + return resolve( + process.env.PAPERCLIP_RUNNER_STATE_DIR + ?? resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", "durable-sessions"), + createHash("sha256").update(normalizedSessionId).digest("hex"), + ); +} + +function digestDirectory(directory: string): { sha256: string; bytes: number } { + const hash = createHash("sha256"); + let bytes = 0; + const visit = (current: string, relative: string) => { + const entries = readdirSync(current, { withFileTypes: true }) + .sort((left, right) => left.name.localeCompare(right.name)); + if (entries.length === 0) hash.update(`directory:${relative}\0`); + for (const entry of entries) { + const path = resolve(current, entry.name); + const relativePath = relative ? `${relative}/${entry.name}` : entry.name; + const stats = lstatSync(path); + if (entry.isDirectory()) { + hash.update(`directory:${relativePath}:${stats.mode & 0o777}\0`); + visit(path, relativePath); + } else if (entry.isSymbolicLink()) { + hash.update(`symlink:${relativePath}:${readlinkSync(path)}\0`); + } else if (entry.isFile()) { + const contents = readFileSync(path); + bytes += contents.byteLength; + hash.update(`file:${relativePath}:${stats.mode & 0o777}:${contents.byteLength}\0`); + hash.update(contents); + } else { + throw new Error(`runner_harness_backup_unsupported_entry:${relativePath}`); + } + } + }; + visit(directory, ""); + return { sha256: `sha256:${hash.digest("hex")}`, bytes }; +} + +export function createNativeHarnessBackupStamp(input: { + manifestPath: string; + normalizedSessionId: string; + runnerInstanceId: string; + completedAt: string; +}): NativeHarnessBackupStamp { + const manifestBytes = readFileSync(input.manifestPath); + return { + schema: "paperclip.native-harness-backup-stamp.v1", + normalizedSessionId: input.normalizedSessionId, + runnerInstanceId: input.runnerInstanceId, + manifestSha256: `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`, + completedAt: input.completedAt, + }; +} + +export function verifyNativeHarnessBackupStamp( + value: unknown, + expectedProviderLeaseId: string, +): boolean { + if (!expectedProviderLeaseId) return false; + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const stamp = value as Record; + if ( + stamp.schema !== "paperclip.native-harness-backup-stamp.v1" || + typeof stamp.normalizedSessionId !== "string" || !stamp.normalizedSessionId || + typeof stamp.runnerInstanceId !== "string" || !stamp.runnerInstanceId || + typeof stamp.manifestSha256 !== "string" || !stamp.manifestSha256.startsWith("sha256:") + ) return false; + const backupRoot = resolve(stateRoot(stamp.normalizedSessionId), "failover-backups"); + for (const candidate of [resolve(backupRoot, "current"), resolve(backupRoot, "previous")]) { + const manifestPath = resolve(candidate, "manifest.json"); + if (!existsSync(manifestPath)) continue; + try { + const bytes = readFileSync(manifestPath); + const manifestSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`; + if (manifestSha256 !== stamp.manifestSha256) continue; + const manifest = JSON.parse(bytes.toString("utf8")) as Record; + if ( + manifest.schema !== "paperclip.native-harness-backup.v1" || + manifest.normalizedSessionId !== stamp.normalizedSessionId || + manifest.runnerInstanceId !== stamp.runnerInstanceId || + manifest.sourceProviderLeaseId !== expectedProviderLeaseId || + !Array.isArray(manifest.directories) || + manifest.directories.length === 0 + ) continue; + let valid = true; + for (const entry of manifest.directories) { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) { + valid = false; + break; + } + const declared = entry as Record; + if ( + typeof declared.name !== "string" || + !/^[A-Za-z0-9._-]+$/.test(declared.name) || + typeof declared.sha256 !== "string" || + typeof declared.bytes !== "number" + ) { + valid = false; + break; + } + const directory = resolve(candidate, declared.name); + if (!existsSync(directory)) { + valid = false; + break; + } + const actual = digestDirectory(directory); + if (actual.sha256 !== declared.sha256 || actual.bytes !== declared.bytes) { + valid = false; + break; + } + } + if (valid) return true; + } catch { + // Try the previous atomically-published backup. + } + } + return false; +} diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index c480b9dc89..c78de591aa 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1,5 +1,6 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; -import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { access, mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises"; +import { execFileSync } from "node:child_process"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -13,9 +14,14 @@ import type { PrpEvent, } from "@paperclipai/paperclip-runner"; import { createHash } from "node:crypto"; +import { + createNativeHarnessBackupStamp, + verifyNativeHarnessBackupStamp, +} from "./native-harness-backup-stamp.js"; type BackendFactoryOptions = { runnerInstanceId?: string; + acpxRuntimeDirectory?: string; codexTransportFactory?: () => unknown; dynamicToolHandler?: (call: unknown) => Promise; onSpawn?: (meta: { @@ -102,25 +108,830 @@ vi.mock("./native-codex-runner.js", () => ({ import { continuingPendingInteractionIds, buildNativeProviderEnvironment, + buildNativeHarnessBackupManifest, cancelNativeSession, createGovernedWaitEventObservation, createRunnerdBackend, executePaperclipNativeSession, getNativeSessionSteeringState, NativeSessionSteeringError, + assertRemoteRunnerBuildMetadata, nativeSessionFailureDisposition, nativeSessionFailureSourceCode, nativeSessionRecoveryProjection, nativeGovernedWaitResult, + parseRemoteExecutableCandidate, + mayUsePreinstalledRunnerArtifact, + nativeUsageCostUsd, + normalizeNativeUsage, + readRemoteProviderPackManifest, + providerSessionIdentityTransitionIsAllowed, providerPlanMarkdown, renewNativeSessionExecutionLease, runtimeInputLifecycleMetric, runtimeQuestionFallbackFromEvent, resolveNativeRuntimeRequest, + resolveNativeHarnessPersistenceProfile, semanticProviderPlanMarkdown, + sha256DirectoryTree, + stageRemoteRunnerDirectory, steerNativeSession, + syncRemoteRunnerDirectoryOut, + verifyNativeHarnessBackup, + shouldRestoreNativeHarnessBackupIntoSandbox, } from "./native-session-executor.js"; +describe("native provider usage normalization", () => { + it("reads remote runner run-delta tokens and provider cost", () => { + const usage = { + total: { + inputTokens: 20_000, + outputTokens: 500, + cacheReadTokens: 8_000, + providerCostUsd: 0.12, + }, + runDelta: { + inputTokens: 4_200, + outputTokens: 180, + cacheReadTokens: 1_500, + providerCostUsd: 0.031, + }, + }; + expect(normalizeNativeUsage(usage)).toEqual({ + inputTokens: 4_200, + outputTokens: 180, + cachedInputTokens: 1_500, + }); + expect(nativeUsageCostUsd(usage)).toBe(0.031); + }); + + it("reads ACPX cumulative usage and a USD cost object", () => { + const usage = { + cumulative: { + inputTokens: 3_000, + outputTokens: 240, + cachedReadTokens: 900, + }, + cost: { amount: 0.044, currency: "USD" }, + }; + expect(normalizeNativeUsage(usage)).toEqual({ + inputTokens: 3_000, + outputTokens: 240, + cachedInputTokens: 900, + }); + expect(nativeUsageCostUsd(usage)).toBe(0.044); + }); + + it("does not treat a non-USD ACPX amount as dollars", () => { + expect( + nativeUsageCostUsd({ cost: { amount: 1.25, currency: "EUR" } }), + ).toBeUndefined(); + }); +}); + +describe("remote provider pack manifest", () => { + const canonical = (value: unknown): string => { + if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`; + if (value && typeof value === "object") { + const object = value as Record; + return `{${Object.keys(object) + .sort() + .map((key) => `${JSON.stringify(key)}:${canonical(object[key])}`) + .join(",")}}`; + } + return JSON.stringify(value); + }; + + it("accepts a fully digested pack and rejects artifact tampering", async () => { + const root = await mkdtemp(join(tmpdir(), "paperclip-provider-pack-")); + await mkdir(join(root, "dist", "cli"), { recursive: true }); + await mkdir(join(root, "node_modules", "node", "bin"), { recursive: true }); + await mkdir(join(root, "node_modules", ".bin"), { recursive: true }); + await mkdir(join(root, "node_modules", "opencode-ai", "bin"), { + recursive: true, + }); + const proxy = "export const proxy = true;\n"; + const sidecar = "export const sidecar = true;\n"; + const node = "provider-node\n"; + const lockfile = "lockfileVersion: '9.0'\n"; + const opencodeCommand = "#!/bin/sh\n"; + const opencodeExecutable = "opencode-binary\n"; + await writeFile( + join(root, "dist", "cli", "opencode-app-server-proxy.js"), + proxy, + ); + await writeFile( + join(root, "dist", "cli", "acpx-runtime-sidecar.js"), + sidecar, + ); + await writeFile(join(root, "node_modules", "node", "bin", "node"), node); + await writeFile(join(root, "pnpm-lock.yaml"), lockfile); + await writeFile( + join(root, "node_modules", ".bin", "opencode"), + opencodeCommand, + ); + await writeFile( + join(root, "node_modules", "opencode-ai", "bin", "opencode.exe"), + opencodeExecutable, + ); + const digest = (value: string) => + `sha256:${createHash("sha256").update(value).digest("hex")}`; + const proxySha = `sha256:${createHash("sha256").update(proxy).digest("hex")}`; + const sidecarSha = `sha256:${createHash("sha256").update(sidecar).digest("hex")}`; + const payload = { + pins: { + nodeMinimum: "24.11.0", + codex: "0.148.0", + opencode: "1.18.17", + acpx: "0.13.1", + claudeAcp: "0.70.0", + codexAcp: "1.6.2", + }, + target: { platform: "linux", architecture: "x64" }, + runnerSourceRevision: "1".repeat(40), + distDigest: sha256DirectoryTree(join(root, "dist")), + bridgeDigest: "", + acpxProfileDigests: { + claude: + "sha256:9d73d1f0f121fb96cc8badb28c22d5bff02d8582eb2e40360a81c189e1b9422a", + codex: + "sha256:94049b3e3c3aee87de62703786e4fa81d031d7bd979f99bdf516d84f28791a79", + }, + artifacts: { + nodeCommand: { + path: "node_modules/node/bin/node", + sha256: digest(node), + }, + productionLock: { path: "pnpm-lock.yaml", sha256: digest(lockfile) }, + opencodeCommand: { + path: "node_modules/.bin/opencode", + sha256: digest(opencodeCommand), + }, + opencodeExecutable: { + path: "node_modules/opencode-ai/bin/opencode.exe", + sha256: digest(opencodeExecutable), + }, + opencodeProxy: { + path: "dist/cli/opencode-app-server-proxy.js", + sha256: proxySha, + }, + acpxSidecar: { + path: "dist/cli/acpx-runtime-sidecar.js", + sha256: sidecarSha, + }, + }, + }; + payload.bridgeDigest = `sha256:${createHash("sha256") + .update(proxySha) + .update("\n") + .update(sidecarSha) + .update("\n") + .update(payload.distDigest) + .digest("hex")}`; + await writeFile( + join(root, "provider-pack.json"), + JSON.stringify({ + schema: "paperclip-runner/remote-provider-pack/v1", + digest: `sha256:${createHash("sha256").update(canonical(payload)).digest("hex")}`, + payload, + }), + ); + expect(readRemoteProviderPackManifest(root).payload.pins.opencode).toBe( + "1.18.17", + ); + await writeFile( + join(root, "dist", "cli", "opencode-app-server-proxy.js"), + "tampered\n", + ); + expect(() => readRemoteProviderPackManifest(root)).toThrow( + "OpenCode proxy digest mismatch", + ); + await writeFile( + join(root, "dist", "cli", "opencode-app-server-proxy.js"), + proxy, + ); + await writeFile( + join(root, "dist", "cli", "transitive-runtime.js"), + "changed transitive module\n", + ); + expect(() => readRemoteProviderPackManifest(root)).toThrow( + "provider dist tree digest mismatch", + ); + await rm(root, { recursive: true, force: true }); + }); +}); + +describe("native harness persistence profiles", () => { + const profile = (provider: Record, driverKind: string) => + resolveNativeHarnessPersistenceProfile({ + provider, + session: { + driverKind, + normalizedSessionId: "session", + protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + } as unknown as NativeExecutionInputV1); + + it.each([ + ["codex", { kind: "codex" }, "codex_app_server", ["runner", "codex-home"]], + [ + "opencode", + { kind: "opencode" }, + "opencode_server", + ["runner", "opencode"], + ], + [ + "acpx pi", + { kind: "acpx", agent: "pi" }, + "acpx_runtime", + ["runner", "acpx"], + ], + [ + "acpx claude", + { kind: "acpx", agent: "claude" }, + "acpx_runtime", + ["runner", "acpx"], + ], + [ + "acpx codex", + { kind: "acpx", agent: "codex" }, + "acpx_runtime", + ["runner", "acpx"], + ], + ])( + "declares the complete %s recovery state", + (_name, provider, driver, directories) => { + expect( + profile( + provider as Record, + driver as string, + ).directories.map((directory) => directory.name), + ).toEqual(directories); + }, + ); + + it("excludes disposable Codex scratch trees and launch-time credentials", () => { + const codex = profile({ kind: "codex" }, "codex_app_server"); + expect( + codex.directories.find((directory) => directory.name === "codex-home"), + ).toMatchObject({ + excludeTopLevelEntries: ["tmp", ".tmp", "auth.json", "config.toml"], + }); + }); +}); + +describe("verified native harness backups", () => { + const backupExecution = { + provider: { kind: "codex", model: "gpt-5.6-sol", approvalPolicy: "never" }, + binding: { + companyId: "company", + runId: "run", + issueId: "issue", + agentId: "agent", + executionWorkspaceId: "workspace", + }, + workspace: { + cwd: "/workspace", + repoUrl: "https://example.test/repo.git", + repoRef: "main", + branchName: "paperclip/test", + }, + session: { + normalizedSessionId: "native-session", + driverKind: "codex_app_server", + protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + } as unknown as NativeExecutionInputV1; + + const acpxIdentity = (suffix: string) => ({ + providerSessionId: `record-${suffix}`, + providerBackendSessionId: `backend-${suffix}`, + providerSessionIdentity: { + kind: "acpx", + normalizedSessionId: "native-session", + acpxRecordId: `record-${suffix}`, + backendSessionId: `backend-${suffix}`, + agentSessionId: `agent-session-${suffix}`, + profileDigest: "sha256:profile", + workspaceDigest: "sha256:workspace", + requestedModel: "claude-sonnet-5", + effectiveModel: "claude-sonnet-5", + permissionMode: "approve-all", + }, + }); + + it("allows only identity-stable ACPX rotation after a governed interaction", () => { + const execution = { + ...backupExecution, + provider: { + kind: "acpx", + agent: "claude", + model: "claude-sonnet-5", + }, + session: { + ...backupExecution.session, + driverKind: "acpx_runtime", + }, + interactionResponses: [{ interactionId: "interaction-1" }], + } as unknown as NativeExecutionInputV1; + const previous = acpxIdentity("previous"); + const current = acpxIdentity("current"); + + expect( + providerSessionIdentityTransitionIsAllowed({ + execution, + previous, + current, + }), + ).toBe(true); + expect( + providerSessionIdentityTransitionIsAllowed({ + execution: { + ...execution, + interactionResponses: [], + } as unknown as NativeExecutionInputV1, + previous, + current, + }), + ).toBe(false); + expect( + providerSessionIdentityTransitionIsAllowed({ + execution, + previous, + current: { + ...current, + providerSessionIdentity: { + ...current.providerSessionIdentity, + workspaceDigest: "sha256:different-workspace", + }, + }, + }), + ).toBe(false); + expect( + providerSessionIdentityTransitionIsAllowed({ + execution, + previous, + current: { + ...current, + providerBackendSessionId: "unbound-backend", + }, + }), + ).toBe(false); + }); + + it("restores a verified continuation into an intentionally fresh non-reusable sandbox", () => { + expect( + shouldRestoreNativeHarnessBackupIntoSandbox({ + acquisitionOutcome: "created", + reusableLeaseConfigured: false, + backupAvailable: true, + }), + ).toBe(true); + expect( + shouldRestoreNativeHarnessBackupIntoSandbox({ + acquisitionOutcome: "created", + reusableLeaseConfigured: true, + backupAvailable: true, + }), + ).toBe(false); + expect( + shouldRestoreNativeHarnessBackupIntoSandbox({ + acquisitionOutcome: "created", + reusableLeaseConfigured: false, + backupAvailable: false, + }), + ).toBe(false); + }); + + it("accepts a complete digest-matched backup and rejects corruption", async () => { + const root = await mkdtemp(join(tmpdir(), "paperclip-harness-backup-")); + try { + const current = join(root, "failover-backups", "current"); + await mkdir(join(current, "runner"), { recursive: true }); + await mkdir(join(current, "codex-home", "sessions"), { recursive: true }); + await writeFile( + join(current, "runner", "runner-state.json"), + "runner-state", + ); + await writeFile( + join(current, "codex-home", "sessions", "thread.jsonl"), + "thread-state", + ); + const manifest = buildNativeHarnessBackupManifest({ + backupRoot: current, + execution: backupExecution, + runnerInstanceId: "runner-1", + providerSessionIdentity: { + providerSessionId: "thread-1", + providerBackendSessionId: "session-1", + providerSessionIdentity: null, + }, + sourceProviderLeaseId: "sandbox-1", + completedAt: "2026-08-26T00:00:00.000Z", + }); + await writeFile(join(current, "manifest.json"), JSON.stringify(manifest)); + + expect( + verifyNativeHarnessBackup({ + root, + execution: backupExecution, + runnerInstanceId: "runner-1", + }), + ).toMatchObject({ + root: current, + manifest: { + sourceProviderLeaseId: "sandbox-1", + directories: [ + expect.objectContaining({ name: "runner" }), + expect.objectContaining({ name: "codex-home" }), + ], + }, + }); + + const continuationExecution = { + ...backupExecution, + binding: { + ...backupExecution.binding, + runId: "run-2", + executionWorkspaceId: "run-2", + }, + } as NativeExecutionInputV1; + expect( + verifyNativeHarnessBackup({ + root, + execution: continuationExecution, + runnerInstanceId: "runner-1", + }), + ).not.toBeNull(); + + await writeFile( + join(current, "codex-home", "sessions", "thread.jsonl"), + "corrupt", + ); + expect( + verifyNativeHarnessBackup({ + root, + execution: backupExecution, + runnerInstanceId: "runner-1", + }), + ).toBeNull(); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + it("rejects a backup whose provider identity or harness contract changed", async () => { + const root = await mkdtemp( + join(tmpdir(), "paperclip-harness-backup-identity-"), + ); + try { + const current = join(root, "failover-backups", "current"); + await mkdir(join(current, "runner"), { recursive: true }); + await mkdir(join(current, "codex-home"), { recursive: true }); + await writeFile( + join(current, "runner", "runner-state.json"), + "runner-state", + ); + expect(() => + buildNativeHarnessBackupManifest({ + backupRoot: current, + execution: backupExecution, + runnerInstanceId: "runner-1", + providerSessionIdentity: { + providerSessionId: null, + providerBackendSessionId: null, + providerSessionIdentity: null, + }, + sourceProviderLeaseId: "sandbox-1", + }), + ).toThrow("runner_harness_state_mismatch"); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + it("verifies the lease stamp and all backup directory digests before replacement", async () => { + const stateBase = await mkdtemp(join(tmpdir(), "paperclip-harness-stamp-")); + const previousStateDirectory = process.env.PAPERCLIP_RUNNER_STATE_DIR; + process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase; + try { + const sessionRoot = join( + stateBase, + createHash("sha256").update("native-session").digest("hex"), + ); + const current = join(sessionRoot, "failover-backups", "current"); + await mkdir(join(current, "runner"), { recursive: true }); + await mkdir(join(current, "codex-home", "sessions"), { recursive: true }); + await writeFile( + join(current, "runner", "runner-state.json"), + "runner-state", + ); + await writeFile( + join(current, "codex-home", "sessions", "thread.jsonl"), + "thread-state", + ); + const manifest = buildNativeHarnessBackupManifest({ + backupRoot: current, + execution: backupExecution, + runnerInstanceId: "runner-1", + providerSessionIdentity: { + providerSessionId: "thread-1", + providerBackendSessionId: "session-1", + providerSessionIdentity: null, + }, + sourceProviderLeaseId: "sandbox-1", + }); + const manifestPath = join(current, "manifest.json"); + await writeFile(manifestPath, JSON.stringify(manifest)); + const stamp = createNativeHarnessBackupStamp({ + manifestPath, + normalizedSessionId: "native-session", + runnerInstanceId: "runner-1", + completedAt: manifest.completedAt, + }); + + expect(verifyNativeHarnessBackupStamp(stamp, "sandbox-1")).toBe(true); + expect(verifyNativeHarnessBackupStamp(stamp, "sandbox-2")).toBe(false); + await writeFile(join(current, "runner", "runner-state.json"), "corrupt"); + expect(verifyNativeHarnessBackupStamp(stamp, "sandbox-1")).toBe(false); + } finally { + if (previousStateDirectory === undefined) { + delete process.env.PAPERCLIP_RUNNER_STATE_DIR; + } else { + process.env.PAPERCLIP_RUNNER_STATE_DIR = previousStateDirectory; + } + await rm(stateBase, { recursive: true, force: true }); + } + }); +}); + +describe("remote provider checkpoint snapshots", () => { + it("excludes Codex scratch and credential state without mutating the live provider home", async () => { + const execute = vi + .fn() + .mockResolvedValueOnce({ + exitCode: 0, + timedOut: false, + stdout: "", + stderr: "", + }) + .mockResolvedValueOnce({ + exitCode: 0, + timedOut: false, + stdout: "", + stderr: "", + }) + .mockResolvedValueOnce({ + exitCode: 0, + timedOut: false, + stdout: "", + stderr: "", + }); + const syncOut = vi.fn( + async ( + _operations: Array<{ + files: Array<{ + sourcePath: string; + targetPath: string; + kind: "file" | "directory"; + mode?: number; + }>; + }>, + ) => undefined, + ); + + await syncRemoteRunnerDirectoryOut({ + runner: { execute, syncOut } as never, + sourcePath: "/remote/session/filesystem/codex-home", + targetPath: "/tmp/paperclip-checkpoint-test-codex-home", + mode: 0o700, + excludeTopLevelEntries: ["tmp", ".tmp", "auth.json", "config.toml"], + }); + + expect(execute).toHaveBeenNthCalledWith( + 1, + expect.objectContaining({ + args: ["-c", "test -d '/remote/session/filesystem/codex-home'"], + }), + ); + const snapshotCommand = String(execute.mock.calls[1]?.[0]?.args?.[1]); + expect(snapshotCommand).toContain("'--exclude=./tmp'"); + expect(snapshotCommand).toContain("'--exclude=./.tmp'"); + expect(snapshotCommand).toContain("'--exclude=./auth.json'"); + expect(snapshotCommand).toContain("'--exclude=./config.toml'"); + expect(snapshotCommand).toContain( + "-C '/remote/session/filesystem/codex-home'", + ); + expect(snapshotCommand).not.toContain( + "rm -rf -- '/remote/session/filesystem/codex-home'", + ); + + const batch = syncOut.mock.calls[0]?.[0]?.[0]; + expect(batch?.files[0]).toMatchObject({ + sourcePath: expect.stringMatching( + /^\/remote\/session\/filesystem\/\.paperclip-checkpoint-/, + ), + targetPath: "/tmp/paperclip-checkpoint-test-codex-home", + kind: "directory", + mode: 0o700, + }); + expect(String(execute.mock.calls[2]?.[0]?.args?.[1])).toMatch( + /^rm -rf -- '\/remote\/session\/filesystem\/\.paperclip-checkpoint-/, + ); + }); + + it("rejects non-top-level checkpoint exclusions", async () => { + const execute = vi.fn().mockResolvedValue({ + exitCode: 0, + timedOut: false, + stdout: "", + stderr: "", + }); + await expect( + syncRemoteRunnerDirectoryOut({ + runner: { execute, syncOut: vi.fn() } as never, + sourcePath: "/remote/codex-home", + targetPath: "/tmp/paperclip-checkpoint-invalid-codex-home", + mode: 0o700, + excludeTopLevelEntries: ["../outside"], + }), + ).rejects.toThrow("runner_remote_checkpoint_exclusion_invalid"); + }); + + it("rejects unsafe fallback archives without replacing durable state", async () => { + const root = await mkdtemp(join(tmpdir(), "paperclip-checkpoint-unsafe-")); + const archiveSource = join(root, "archive-source"); + const targetPath = join(root, "durable-target"); + try { + await mkdir(archiveSource, { recursive: true }); + await mkdir(targetPath, { recursive: true }); + await writeFile(join(targetPath, "preserved.txt"), "preserved"); + await symlink("/etc/passwd", join(archiveSource, "host-secret")); + const archive = execFileSync( + "tar", + ["-czf", "-", "-C", archiveSource, "."], + { maxBuffer: 8 * 1024 * 1024 }, + ); + const execute = vi + .fn() + .mockResolvedValueOnce({ + exitCode: 0, + timedOut: false, + stdout: "", + stderr: "", + }) + .mockResolvedValueOnce({ + exitCode: 0, + timedOut: false, + stdout: archive.toString("base64"), + stderr: "", + }); + + await expect( + syncRemoteRunnerDirectoryOut({ + runner: { execute } as never, + sourcePath: "/remote/codex-home", + targetPath, + mode: 0o700, + }), + ).rejects.toThrow("runner_remote_checkpoint_archive_unsafe_entry"); + await expect(access(join(targetPath, "preserved.txt"))).resolves.toBeUndefined(); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); + +describe("remote provider checkpoint restores", () => { + it("does not upload excluded Codex scratch trees or credentials", async () => { + const sourcePath = await mkdtemp( + join(tmpdir(), "paperclip-codex-restore-source-"), + ); + try { + await mkdir(join(sourcePath, "sessions"), { recursive: true }); + await mkdir(join(sourcePath, ".tmp"), { recursive: true }); + await writeFile( + join(sourcePath, "sessions", "thread.jsonl"), + "durable session", + ); + await writeFile( + join(sourcePath, ".tmp", "scratch.bin"), + "disposable scratch", + ); + await writeFile(join(sourcePath, "auth.json"), "credential"); + await writeFile(join(sourcePath, "config.toml"), "bearer token"); + const syncIn = vi.fn( + async ( + operations: Array<{ + files: Array<{ sourcePath: string }>; + }>, + ) => { + const stagedPath = operations[0]!.files[0]!.sourcePath; + expect(stagedPath).not.toBe(sourcePath); + await expect( + access(join(stagedPath, "sessions", "thread.jsonl")), + ).resolves.toBeUndefined(); + await expect( + access(join(stagedPath, ".tmp", "scratch.bin")), + ).rejects.toThrow(); + await expect(access(join(stagedPath, "auth.json"))).rejects.toThrow(); + await expect( + access(join(stagedPath, "config.toml")), + ).rejects.toThrow(); + }, + ); + + await stageRemoteRunnerDirectory({ + target: { + kind: "remote", + transport: "provider", + remoteCwd: "/remote", + runner: { syncIn } as never, + } as never, + runner: { syncIn } as never, + sourcePath, + targetPath: "/remote/codex-home", + mode: 0o700, + excludeTopLevelEntries: ["tmp", ".tmp", "auth.json", "config.toml"], + }); + + expect(syncIn).toHaveBeenCalledOnce(); + } finally { + await rm(sourcePath, { recursive: true, force: true }); + } + }); +}); + +describe("remote preinstalled executable discovery", () => { + it("accepts one normalized absolute executable path", () => { + expect( + parseRemoteExecutableCandidate( + "/home/daytona/.local/bin/paperclip-runnerd\n", + ), + ).toBe("/home/daytona/.local/bin/paperclip-runnerd"); + }); + + it.each([ + "paperclip-runnerd\n", + "/safe/path\n/unexpected/second-line\n", + "/safe/path with spaces\n", + "/safe/path;touch-bad\n", + ])("rejects ambiguous or shell-active output: %j", (stdout) => { + expect(parseRemoteExecutableCandidate(stdout)).toBeNull(); + }); + + it("does not accept a merely contract-compatible runnerd when a build-owned artifact is configured", () => { + expect( + mayUsePreinstalledRunnerArtifact("/artifacts/paperclip-runnerd"), + ).toBe(false); + expect(mayUsePreinstalledRunnerArtifact(" ")).toBe(true); + expect(mayUsePreinstalledRunnerArtifact(undefined)).toBe(true); + }); +}); + +describe("remote runner build metadata", () => { + const current = { + schema: "paperclip-runner/runnerd-build-metadata/v1", + binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", + binaryContractVersion: 2, + prpTransportModes: ["dial_ws_loopback", "dial_wss", "listen_ws"], + }; + + it("accepts the current contract with the required transport", () => { + expect(() => + assertRemoteRunnerBuildMetadata(current, "listen_ws"), + ).not.toThrow(); + }); + + it("fails before dispatch when a preinstalled runner uses the stale contract", () => { + expect(() => + assertRemoteRunnerBuildMetadata( + { + ...current, + binaryContractVersion: 1, + }, + "listen_ws", + ), + ).toThrow("runner_remote_artifact_contract_incompatible"); + }); + + it("requires the selected transport without falling through", () => { + expect(() => + assertRemoteRunnerBuildMetadata( + { + ...current, + prpTransportModes: ["dial_wss"], + }, + "listen_ws", + ), + ).toThrow("runner_remote_transport_capability_missing:listen_ws"); + }); +}); + + describe("runtime question fallback", () => { const questionSet = { schema: "paperclip.question_set.v1" as const, @@ -1691,6 +2502,13 @@ describe("native session bounded recovery", () => { new Error("provider_request_timeout: provider=codex stage=turn.start"), ), ).toBe("provider_request_timeout"); + expect( + nativeSessionFailureSourceCode( + new Error( + "runner_remote_provider_artifact_incompatible: OpenCode version mismatch", + ), + ), + ).toBe("runner_remote_provider_artifact_incompatible"); }); it("retries the same run twice and stops at the third failed attempt", () => { @@ -1717,6 +2535,17 @@ describe("native session bounded recovery", () => { failureCode: "native_event_replay_conflict", nextAttemptAt: null, }); + expect( + nativeSessionFailureDisposition( + 1, + now, + "runner_remote_provider_artifact_incompatible", + ), + ).toEqual({ + phase: "terminal_failure", + failureCode: "runner_remote_provider_artifact_incompatible", + nextAttemptAt: null, + }); }); it("escalates exhausted result-less sessions to board review instead of leaving the provider as its own owner", () => { @@ -1949,4 +2778,208 @@ describe("runnerd provider runtime wiring", () => { }); }); + it("uses the remote workspace for both the runner backend and native session", async () => { + const remoteCwd = "/home/daytona/paperclip-workspace"; + const remoteExecution = { + ...execution, + binding: { ...execution.binding, runId: "run-remote-workspace-test" }, + task: { + identifier: "DOT-REMOTE", + title: "Remote workspace test", + description: null, + prompt: "Verify the remote workspace.", + workMode: "standard", + }, + workspace: { + cwd: "/host/paperclip-workspace", + repoUrl: null, + repoRef: null, + branchName: null, + }, + session: { + normalizedSessionId: "remote-workspace-session", + driverKind: "codex_app_server", + protocolVersion: 2, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + provider: { + kind: "codex", + model: null, + approvalPolicy: "never", + }, + executionMode: "default", + planningContext: null, + interactionResponses: [], + credentialBindings: [], + } as unknown as NativeExecutionInputV1; + state.createBackend.mockClear(); + state.execute.mockReset().mockResolvedValue({ + result: { summary: "completed" }, + terminal: { runTerminalState: "succeeded" }, + turnId: "turn", + normalizedSessionId: "session", + providerSessionId: null, + driverKind: "test", + driverVersion: "1", + nativeEventCount: 1, + highestContiguousSourceSeq: 1, + }); + + await executePaperclipNativeSession({ + db: leaseDb(remoteExecution), + execution: remoteExecution, + runnerInstanceId: "runner", + useRunnerd: true, + runnerExecutionTarget: { + kind: "remote", + transport: "ssh", + remoteCwd, + spec: { + host: "runner.internal", + port: 22, + username: "runner", + remoteWorkspacePath: remoteCwd, + remoteCwd, + privateKey: null, + knownHosts: null, + strictHostKeyChecking: true, + }, + }, + runnerPublicUrl: "wss://paperclip.example.test", + }); + + expect(state.createBackend).toHaveBeenCalledWith( + expect.objectContaining({ + workspace: expect.objectContaining({ cwd: remoteCwd }), + }), + expect.any(Object), + ); + expect(state.execute).toHaveBeenCalledWith( + expect.objectContaining({ + input: expect.objectContaining({ + workspace: expect.objectContaining({ cwd: remoteCwd }), + }), + }), + ); + }); + + it.each([ + ["opencode", { kind: "opencode", model: null }, "opencode_server"], + [ + "acpx", + { kind: "acpx", agent: "codex", model: null }, + "acpx_runtime", + ], + ])( + "fails closed before launching the remote %s provider", + async (providerKind, provider, driverKind) => { + const remoteCwd = "/home/daytona/paperclip-workspace"; + const remoteProviderExecution = { + ...execution, + binding: { + ...execution.binding, + runId: `run-remote-${providerKind}-rejected`, + }, + session: { + ...execution.session, + normalizedSessionId: `remote-${providerKind}-rejected`, + driverKind, + }, + provider, + } as unknown as NativeExecutionInputV1; + state.createBackend.mockClear(); + + await expect( + createRunnerdBackend({ + db: leaseDb(remoteProviderExecution), + execution: remoteProviderExecution, + runnerInstanceId: "runner", + runnerExecutionTarget: { + kind: "remote", + transport: "ssh", + remoteCwd, + spec: { + host: "runner.internal", + port: 22, + username: "runner", + remoteWorkspacePath: remoteCwd, + remoteCwd, + privateKey: null, + knownHosts: null, + strictHostKeyChecking: true, + }, + }, + }), + ).rejects.toThrow( + `runner_remote_provider_artifact_incompatible: remote ${providerKind} is unavailable until runnerd provider dispatch is qualified`, + ); + expect(state.createBackend).not.toHaveBeenCalled(); + }, + ); + + it("passes the isolated ACPX runtime directory to the native backend factory", async () => { + const acpxExecution = { + ...execution, + schema: "paperclip.native-execution-input.v3", + task: { + identifier: "DOT-ACPX", + title: "ACPX task", + description: null, + prompt: "Complete the ACPX task.", + workMode: "standard", + }, + workspace: { + cwd: "/tmp/acpx-native", + repoUrl: null, + repoRef: null, + branchName: null, + }, + session: { + normalizedSessionId: "acpx-session", + driverKind: "acpx_runtime", + protocolVersion: 1, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }, + provider: { + kind: "acpx", + agent: "codex", + model: "gpt-5.6-sol", + permissionPolicy: "interactive", + profile: { + driverKind: "acpx_runtime", + protocolVersion: 1, + acpxVersion: "0.13.1", + agent: "codex", + agentProfileVersion: 1, + agentServerPackage: "@agentclientprotocol/codex-acp", + agentServerVersion: "1.6.2", + agentRuntimePackage: null, + agentRuntimeVersion: null, + commandDigest: "sha256:test", + }, + }, + executionMode: "default", + planningContext: null, + interactionResponses: [], + credentialBindings: [], + runtimeContext: {}, + } as unknown as NativeExecutionInputV1; + state.createBackend.mockClear(); + await createRunnerdBackend({ + db: leaseDb(acpxExecution), + execution: acpxExecution, + runnerInstanceId: "runner", + }); + + expect(state.createBackend).toHaveBeenCalledWith( + acpxExecution, + expect.objectContaining({ + acpxRuntimeDirectory: expect.stringContaining( + "/runtime/paperclip-runner/acpx", + ), + acpxDynamicToolHandler: expect.any(Function), + }), + ); + }); + }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index b2cbc5eb00..b1a79b2386 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,4 +1,5 @@ import { createHash, randomUUID } from "node:crypto"; +import { execFileSync } from "node:child_process"; import { chmodSync, closeSync, @@ -7,12 +8,18 @@ import { fstatSync, lstatSync, mkdirSync, + mkdtempSync, openSync, + readFileSync, + readdirSync, + readlinkSync, readSync, renameSync, + rmSync, writeFileSync, } from "node:fs"; -import { resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { join, posix, resolve } from "node:path"; import type { AdapterExecutionResult, AdapterRuntimeEvent, @@ -21,6 +28,7 @@ import type { NativeFinalizationResult } from "@paperclipai/shared"; import type { HarnessRuntimeRequestResolution, NativeExecutionInput, + NativeRuntimeContextSnapshot, NativeSession, NativeSessionBackend, PaperclipQuestionSet, @@ -31,13 +39,25 @@ import type { import { createNativeSessionBackend, createRunnerdCodexTransport, + defaultCapabilityRunnerdBinary, executeNativeSession, parsePaperclipQuestionSet, + resolveSourceCodexHome, + type RunnerProcessHandle, + type RunnerProcessLaunchSpec, } from "../../vendor/paperclip-runner/index.js"; +import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; +import { createSshCommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/ssh"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { + resolvePaperclipRunnerTransport, + type PaperclipRunnerTransport, +} from "@paperclipai/adapter-utils/runner-connectivity"; import type { Db } from "@paperclipai/db"; import { and, desc, eq, gt, inArray, or, sql } from "drizzle-orm"; import { documentRevisions, + environmentLeases, heartbeatRunEvents, heartbeatRuns, issueDocuments, @@ -48,6 +68,7 @@ import { import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js"; +import { connectRunnerPrpIngress } from "../../realtime/runner-prp-outbound.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { persistActivity, publishActivity } from "../activity-log.js"; import { commitNativeStatusDecision } from "./status-decision-committer.js"; @@ -68,6 +89,7 @@ import { type NativeRunSpanScope, type NativeRunTrace, } from "./native-run-trace.js"; +import { createNativeHarnessBackupStamp } from "./native-harness-backup-stamp.js"; type ActiveNativeSession = { session: NativeSession; @@ -89,6 +111,9 @@ export class NativeCancellationPendingRecoveryError extends Error { } const activeNativeSessions = new Map(); +const MAX_REMOTE_CHECKPOINT_ARCHIVE_BYTES = 64 * 1024 * 1024; +const MAX_REMOTE_CHECKPOINT_EXPANDED_BYTES = 64 * 1024 * 1024; +const MAX_REMOTE_CHECKPOINT_ENTRIES = 20_000; const NATIVE_DURABLE_IDENTITY_MAX_BYTES = 2 * 1024 * 1024; const NATIVE_WARM_CHECKPOINT_MAX_BYTES = 8 * 1024 * 1024; const NATIVE_SESSION_EXECUTION_LEASE_TTL_MS = 20 * 60_000; @@ -978,6 +1003,28 @@ export async function synchronizeCompletedProviderPlan(input: { }; } +class SessionToolAuthorityRouter { + #authority: PaperclipRunnerToolAuthority; + + constructor(authority: PaperclipRunnerToolAuthority) { + this.#authority = authority; + } + + bind(authority: PaperclipRunnerToolAuthority): void { + this.#authority = authority; + } + + definitions() { + return this.#authority.definitions(); + } + + execute(call: Parameters[0]) { + return this.#authority.execute(call); + } +} + +const sessionToolRouters = new Map(); + function nativeSessionKey(execution: NativeExecutionInput): string { return ( execution.session.normalizedSessionId ?? @@ -1129,6 +1176,369 @@ function nativeSessionConfigDigest(execution: NativeExecutionInput): string { .digest("hex")}`; } +function nativeHarnessEnvironmentFingerprint( + execution: NativeExecutionInput, +): string { + return `sha256:${createHash("sha256") + .update( + canonicalJson({ + companyId: execution.binding.companyId, + agentId: execution.binding.agentId, + issueId: execution.binding.issueId, + normalizedSessionId: nativeSessionKey(execution), + workspace: { + cwd: execution.workspace.cwd, + repoUrl: execution.workspace.repoUrl, + repoRef: execution.workspace.repoRef, + branchName: execution.workspace.branchName, + }, + provider: execution.provider, + driverKind: execution.session.driverKind, + }), + ) + .digest("hex")}`; +} + +type NativeProviderKind = NativeExecutionInput["provider"]["kind"]; +type NativeDriverKind = NativeExecutionInput["session"]["driverKind"]; + +export interface NativeHarnessPersistenceDirectory { + name: "runner" | "codex-home" | "opencode" | "acpx"; + location: "runner" | "filesystem"; + excludeTopLevelEntries: readonly string[]; +} + +export interface NativeHarnessPersistenceProfile { + providerKind: NativeProviderKind; + driverKind: NativeDriverKind; + directories: readonly NativeHarnessPersistenceDirectory[]; +} + +export interface NativeHarnessBackupManifest { + schema: "paperclip.native-harness-backup.v1"; + normalizedSessionId: string; + runnerInstanceId: string; + providerKind: NativeProviderKind; + driverKind: NativeDriverKind; + providerSessionIdentity: unknown; + sourceProviderLeaseId: string; + environmentFingerprint: string; + runnerContractVersion: number; + directories: Array<{ + name: string; + sha256: string; + bytes: number; + }>; + completedAt: string; +} + +export function resolveNativeHarnessPersistenceProfile( + execution: NativeExecutionInput, +): NativeHarnessPersistenceProfile { + const providerDirectory: NativeHarnessPersistenceDirectory | null = + execution.provider.kind === "codex" + ? { + name: "codex-home", + location: "filesystem", + // Codex session history lives below this home, but these files are + // launch-time material: auth.json is copied from the configured + // credential source and config.toml can contain the native MCP + // bearer token. Re-materialize both for a replacement sandbox + // instead of putting credentials into the disaster-recovery copy. + excludeTopLevelEntries: ["tmp", ".tmp", "auth.json", "config.toml"], + } + : execution.provider.kind === "opencode" + ? { + name: "opencode", + location: "filesystem", + excludeTopLevelEntries: [], + } + : execution.provider.kind === "acpx" + ? { + name: "acpx", + location: "filesystem", + excludeTopLevelEntries: [], + } + : null; + return { + providerKind: execution.provider.kind, + driverKind: execution.session.driverKind, + directories: [ + { name: "runner", location: "runner", excludeTopLevelEntries: [] }, + ...(providerDirectory ? [providerDirectory] : []), + ], + }; +} + +function canonicalJson(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`; + if (value && typeof value === "object") { + const entries = Object.entries(value as Record).sort( + ([left], [right]) => left.localeCompare(right), + ); + return `{${entries.map(([key, entry]) => `${JSON.stringify(key)}:${canonicalJson(entry)}`).join(",")}}`; + } + return JSON.stringify(value) ?? "null"; +} + +function providerSessionIdentityFromRunnerState( + state: Record, +): Record { + return { + providerSessionId: state.providerSessionId ?? null, + providerBackendSessionId: state.providerBackendSessionId ?? null, + providerSessionIdentity: state.providerSessionIdentity ?? null, + }; +} + +function providerSessionIdentityIsPresent(value: unknown): boolean { + const identity = record(value); + return ( + (identity.providerSessionId !== null && + identity.providerSessionId !== undefined) || + (identity.providerBackendSessionId !== null && + identity.providerBackendSessionId !== undefined) || + (identity.providerSessionIdentity !== null && + identity.providerSessionIdentity !== undefined) + ); +} + +export function providerSessionIdentityTransitionIsAllowed(input: { + execution: NativeExecutionInput; + previous: unknown; + current: unknown; +}): boolean { + if (canonicalJson(input.previous) === canonicalJson(input.current)) { + return true; + } + if ( + input.execution.provider.kind !== "acpx" || + input.execution.interactionResponses.length === 0 + ) { + return false; + } + + const previousOuter = record(input.previous); + const currentOuter = record(input.current); + const previous = record(previousOuter.providerSessionIdentity); + const current = record(currentOuter.providerSessionIdentity); + if (previous.kind !== "acpx" || current.kind !== "acpx") return false; + + const stableFields = [ + "normalizedSessionId", + "profileDigest", + "workspaceDigest", + "requestedModel", + "effectiveModel", + "permissionMode", + ] as const; + if ( + current.normalizedSessionId !== nativeSessionKey(input.execution) || + stableFields.some( + (field) => + typeof previous[field] !== "string" || + previous[field] !== current[field], + ) + ) { + return false; + } + + return ( + typeof previous.acpxRecordId === "string" && + typeof previous.backendSessionId === "string" && + typeof previous.agentSessionId === "string" && + typeof current.acpxRecordId === "string" && + typeof current.backendSessionId === "string" && + typeof current.agentSessionId === "string" && + previousOuter.providerSessionId === previous.acpxRecordId && + previousOuter.providerBackendSessionId === previous.backendSessionId && + currentOuter.providerSessionId === current.acpxRecordId && + currentOuter.providerBackendSessionId === current.backendSessionId + ); +} + +function digestBackupDirectory(directory: string): { + sha256: string; + bytes: number; +} { + const hash = createHash("sha256"); + let bytes = 0; + const visit = (current: string, relative: string) => { + const entries = readdirSync(current, { withFileTypes: true }).sort( + (left, right) => left.name.localeCompare(right.name), + ); + if (entries.length === 0) hash.update(`directory:${relative}\0`); + for (const entry of entries) { + const entryPath = resolve(current, entry.name); + const entryRelative = relative ? `${relative}/${entry.name}` : entry.name; + const stats = lstatSync(entryPath); + if (entry.isDirectory()) { + hash.update(`directory:${entryRelative}:${stats.mode & 0o777}\0`); + visit(entryPath, entryRelative); + } else if (entry.isSymbolicLink()) { + hash.update(`symlink:${entryRelative}:${readlinkSync(entryPath)}\0`); + } else if (entry.isFile()) { + const contents = readFileSync(entryPath); + bytes += contents.byteLength; + hash.update( + `file:${entryRelative}:${stats.mode & 0o777}:${contents.byteLength}\0`, + ); + hash.update(contents); + } else { + throw new Error( + `runner_harness_backup_unsupported_entry:${entryRelative}`, + ); + } + } + }; + visit(directory, ""); + return { sha256: `sha256:${hash.digest("hex")}`, bytes }; +} + +function harnessBackupRoot(root: string): string { + return resolve(root, "failover-backups"); +} + +function harnessBackupCandidates(root: string): string[] { + const backupRoot = harnessBackupRoot(root); + return [resolve(backupRoot, "current"), resolve(backupRoot, "previous")]; +} + +type VerifiedHarnessBackup = { + root: string; + manifest: NativeHarnessBackupManifest; + bytes: number; +}; + +export function shouldRestoreNativeHarnessBackupIntoSandbox(input: { + acquisitionOutcome: "created" | "resumed" | "replacement" | null; + reusableLeaseConfigured: boolean | null | undefined; + backupAvailable: boolean; +}): boolean { + return ( + input.acquisitionOutcome === "created" && + input.reusableLeaseConfigured === false && + input.backupAvailable + ); +} + +function compatibleNativeHarnessBackupManifests(input: { + root: string; + execution: NativeExecutionInput; + runnerInstanceId: string; +}): Array<{ root: string; manifest: NativeHarnessBackupManifest }> { + const profile = resolveNativeHarnessPersistenceProfile(input.execution); + const expectedNames = profile.directories + .map((directory) => directory.name) + .sort(); + const compatible: Array<{ + root: string; + manifest: NativeHarnessBackupManifest; + }> = []; + for (const candidateRoot of harnessBackupCandidates(input.root)) { + const manifestPath = resolve(candidateRoot, "manifest.json"); + if (!existsSync(manifestPath)) continue; + let manifest: NativeHarnessBackupManifest; + try { + manifest = JSON.parse( + readFileSync(manifestPath, "utf8"), + ) as NativeHarnessBackupManifest; + } catch { + continue; + } + if ( + manifest.schema !== "paperclip.native-harness-backup.v1" || + manifest.normalizedSessionId !== nativeSessionKey(input.execution) || + manifest.runnerInstanceId !== input.runnerInstanceId || + manifest.providerKind !== profile.providerKind || + manifest.driverKind !== profile.driverKind || + manifest.environmentFingerprint !== + nativeHarnessEnvironmentFingerprint(input.execution) || + manifest.runnerContractVersion !== RUNNERD_BINARY_CONTRACT_VERSION || + !providerSessionIdentityIsPresent(manifest.providerSessionIdentity) || + !Array.isArray(manifest.directories) + ) + continue; + const manifestNames = manifest.directories + .map((directory) => directory.name) + .sort(); + if (canonicalJson(expectedNames) !== canonicalJson(manifestNames)) continue; + compatible.push({ root: candidateRoot, manifest }); + } + return compatible; +} + +export function buildNativeHarnessBackupManifest(input: { + backupRoot: string; + execution: NativeExecutionInput; + runnerInstanceId: string; + providerSessionIdentity: unknown; + sourceProviderLeaseId: string; + completedAt?: string; +}): NativeHarnessBackupManifest { + if (!providerSessionIdentityIsPresent(input.providerSessionIdentity)) { + throw new Error("runner_harness_state_mismatch"); + } + const profile = resolveNativeHarnessPersistenceProfile(input.execution); + const directories = profile.directories.map((directory) => { + const path = resolve(input.backupRoot, directory.name); + if (!existsSync(path)) throw new Error("runner_harness_state_mismatch"); + return { name: directory.name, ...digestBackupDirectory(path) }; + }); + return { + schema: "paperclip.native-harness-backup.v1", + normalizedSessionId: nativeSessionKey(input.execution), + runnerInstanceId: input.runnerInstanceId, + providerKind: profile.providerKind, + driverKind: profile.driverKind, + providerSessionIdentity: input.providerSessionIdentity, + sourceProviderLeaseId: input.sourceProviderLeaseId, + environmentFingerprint: nativeHarnessEnvironmentFingerprint( + input.execution, + ), + runnerContractVersion: RUNNERD_BINARY_CONTRACT_VERSION, + directories, + completedAt: input.completedAt ?? new Date().toISOString(), + }; +} + +export function verifyNativeHarnessBackup(input: { + root: string; + execution: NativeExecutionInput; + runnerInstanceId: string; +}): VerifiedHarnessBackup | null { + for (const { + root: candidateRoot, + manifest, + } of compatibleNativeHarnessBackupManifests(input)) { + let bytes = 0; + let valid = true; + for (const declared of manifest.directories) { + const directoryPath = resolve(candidateRoot, declared.name); + if (!existsSync(directoryPath)) { + valid = false; + break; + } + try { + const digest = digestBackupDirectory(directoryPath); + if ( + digest.sha256 !== declared.sha256 || + digest.bytes !== declared.bytes + ) { + valid = false; + break; + } + bytes += digest.bytes; + } catch { + valid = false; + break; + } + } + if (valid) return { root: candidateRoot, manifest, bytes }; + } + return null; +} + function nativeSessionCheckpointDirectory(): string { const directory = resolve( resolvePaperclipInstanceRoot(), @@ -1260,7 +1670,9 @@ export function nativeSessionFailureDisposition( now = new Date(), sourceFailureCode?: ReturnType, ) { - const permanentFailure = sourceFailureCode === "native_event_replay_conflict"; + const permanentFailure = + sourceFailureCode === "native_event_replay_conflict" || + sourceFailureCode === "runner_remote_provider_artifact_incompatible"; const exhausted = permanentFailure || attempt >= 3; return { phase: exhausted @@ -1299,6 +1711,7 @@ export function nativeSessionRecoveryProjection(input: { export function nativeSessionFailureSourceCode( error: unknown, ): + | "runner_remote_provider_artifact_incompatible" | "provider_process_exited" | "provider_stdout_closed" | "provider_process_output_closed" @@ -1314,6 +1727,9 @@ export function nativeSessionFailureSourceCode( | "native_event_replay_conflict" | "native_session_interrupted" { const message = error instanceof Error ? error.message : String(error); + if (/runner_remote_provider_artifact_incompatible/i.test(message)) { + return "runner_remote_provider_artifact_incompatible"; + } if (/provider_process_exited/i.test(message)) { return "provider_process_exited"; } @@ -1416,6 +1832,7 @@ export async function nativeProviderRecoveryEvidence(input: { const definitelyPreSession = new Set< ReturnType >([ + "runner_remote_provider_artifact_incompatible", "provider_process_exited", "provider_stdout_closed", "provider_process_output_closed", @@ -2310,6 +2727,14 @@ export async function executePaperclipNativeSession(input: { preparationSpans?: NativeRunHistoricalSpan[]; /** Resolved adapter env; the runner transport applies a provider allowlist before spawn. */ runnerEnvironment?: NodeJS.ProcessEnv; + runnerExecutionTarget?: AdapterExecutionTarget | null; + enableRunnerPreviewIngress?: boolean; + runnerPublicUrl?: string | null; + runnerCaBundlePath?: string | null; + runnerRemoteBinaryPath?: string | null; + runnerRemoteCodexPath?: string | null; + runnerRemoteCodexNpmSpec?: string | null; + runnerRemoteProviderPackPath?: string | null; enqueueWakeup?: ( agentId: string, options: { @@ -2398,6 +2823,8 @@ export async function executePaperclipNativeSession(input: { } const leaseOwner = input.leaseOwner ?? `${effectiveRunnerInstanceId}:${randomUUID()}`; + const leaseNow = new Date(); + const leaseExpiresAt = new Date(leaseNow.getTime() + 20 * 60_000); let attempt: number; try { attempt = await trace.measure( @@ -2912,7 +3339,16 @@ export async function executePaperclipNativeSession(input: { }) : null; } - const runnerExecution = input.execution; + const runnerExecution = + input.useRunnerd && input.runnerExecutionTarget?.kind === "remote" + ? { + ...input.execution, + workspace: { + ...input.execution.workspace, + cwd: input.runnerExecutionTarget.remoteCwd, + }, + } + : input.execution; const leaseRenewal = startNativeSessionExecutionLeaseRenewal({ db: input.db, runId: input.execution.binding.runId, @@ -2950,6 +3386,20 @@ export async function executePaperclipNativeSession(input: { createNativeSessionBackend(input.execution, { runnerInstanceId: input.runnerInstanceId, onSpawn: input.onSpawn, + opencodeEnvironment: input.runnerEnvironment ?? process.env, + acpxEnvironment: input.runnerEnvironment ?? process.env, + opencodeRuntimeDirectory: resolve( + resolvePaperclipInstanceRoot(), + "runtime", + "paperclip-runner", + "opencode", + ), + acpxRuntimeDirectory: resolve( + resolvePaperclipInstanceRoot(), + "runtime", + "paperclip-runner", + "acpx", + ), }), controlPlane, runnerInstanceId: effectiveRunnerInstanceId, @@ -3447,6 +3897,732 @@ export function normalizeNativeUsage(usage: Record | null) { }; } +function processEnvironment( + environment: NodeJS.ProcessEnv, +): Record { + return Object.fromEntries( + Object.entries(environment).filter( + (entry): entry is [string, string] => typeof entry[1] === "string", + ), + ); +} + +export function parseRemoteExecutableCandidate(stdout: string): string | null { + const lines = stdout + .split(/\r?\n/) + .map((line) => line.trim()) + .filter(Boolean); + if (lines.length !== 1) return null; + const candidate = lines[0]!; + if ( + !candidate.startsWith("/") || + candidate.length > 4_096 || + !/^\/[A-Za-z0-9_./+@-]+$/.test(candidate) + ) { + return null; + } + return posix.normalize(candidate); +} + +export function mayUsePreinstalledRunnerArtifact( + configuredRemoteBinaryPath: string | null | undefined, +): boolean { + return !configuredRemoteBinaryPath?.trim(); +} + +const RUNNERD_BUILD_METADATA_SCHEMA = + "paperclip-runner/runnerd-build-metadata/v1"; +const RUNNERD_BINARY_CONTRACT_VERSION = 2; + +const REMOTE_PROVIDER_PACK_SCHEMA = "paperclip-runner/remote-provider-pack/v1"; +const REMOTE_PROVIDER_PACK_PINS = { + nodeMinimum: "24.11.0", + codex: "0.148.0", + opencode: "1.18.17", + acpx: "0.13.1", + claudeAcp: "0.70.0", + codexAcp: "1.6.2", +} as const; +const REMOTE_PROVIDER_PACK_PROFILE_DIGESTS = { + claude: + "sha256:9d73d1f0f121fb96cc8badb28c22d5bff02d8582eb2e40360a81c189e1b9422a", + codex: + "sha256:94049b3e3c3aee87de62703786e4fa81d031d7bd979f99bdf516d84f28791a79", +} as const; + +type RemoteProviderPackManifest = { + schema: typeof REMOTE_PROVIDER_PACK_SCHEMA; + digest: string; + payload: { + pins: typeof REMOTE_PROVIDER_PACK_PINS; + target: { platform: string; architecture: string }; + runnerSourceRevision: string; + distDigest: string; + bridgeDigest: string; + acpxProfileDigests: typeof REMOTE_PROVIDER_PACK_PROFILE_DIGESTS; + artifacts: { + nodeCommand: { path: string; sha256: string }; + productionLock: { path: string; sha256: string }; + opencodeCommand: { path: string; sha256: string }; + opencodeExecutable: { path: string; sha256: string }; + opencodeProxy: { path: string; sha256: string }; + acpxSidecar: { path: string; sha256: string }; + }; + }; +}; + +function sha256File(path: string): string { + return `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`; +} + +export function sha256DirectoryTree(root: string): string { + const hash = createHash("sha256"); + const visit = (directory: string, prefix = "") => { + const entries = readdirSync(directory, { withFileTypes: true }).sort( + (left, right) => left.name.localeCompare(right.name), + ); + for (const entry of entries) { + const relativePath = prefix ? `${prefix}/${entry.name}` : entry.name; + const absolutePath = join(directory, entry.name); + if (entry.isDirectory()) { + hash.update(`directory\0${relativePath}\n`); + visit(absolutePath, relativePath); + } else if (entry.isFile()) { + hash.update(`file\0${relativePath}\0${sha256File(absolutePath)}\n`); + } else if (entry.isSymbolicLink()) { + hash.update( + `symlink\0${relativePath}\0${readlinkSync(absolutePath)}\n`, + ); + } else { + throw new Error( + `runner_remote_provider_artifact_incompatible: unsupported dist entry ${relativePath}`, + ); + } + } + }; + visit(root); + return `sha256:${hash.digest("hex")}`; +} + +function providerPackRelativePath(value: unknown, field: string): string { + if ( + typeof value !== "string" || + value.length === 0 || + value.startsWith("/") || + value.includes("\\") || + posix.normalize(value) !== value || + value.split("/").includes("..") + ) { + throw new Error( + `runner_remote_provider_artifact_incompatible: invalid ${field}`, + ); + } + return value; +} + +export function readRemoteProviderPackManifest( + packRoot: string, +): RemoteProviderPackManifest { + let manifest: RemoteProviderPackManifest; + try { + manifest = JSON.parse( + readFileSync(resolve(packRoot, "provider-pack.json"), "utf8"), + ) as RemoteProviderPackManifest; + } catch (error) { + throw new Error( + "runner_remote_provider_artifact_incompatible: provider-pack.json is unreadable", + { cause: error }, + ); + } + const payload = manifest?.payload; + if ( + manifest.schema !== REMOTE_PROVIDER_PACK_SCHEMA || + !payload || + canonicalJson(payload.pins) !== canonicalJson(REMOTE_PROVIDER_PACK_PINS) || + canonicalJson(payload.acpxProfileDigests) !== + canonicalJson(REMOTE_PROVIDER_PACK_PROFILE_DIGESTS) || + typeof payload.target?.platform !== "string" || + typeof payload.target?.architecture !== "string" || + !/^[0-9a-f]{40}(?:-dirty)?$/.test(payload.runnerSourceRevision) || + !/^sha256:[0-9a-f]{64}$/.test(payload.distDigest) + ) { + throw new Error( + "runner_remote_provider_artifact_incompatible: provider pack pins or source revision do not match", + ); + } + const digest = `sha256:${createHash("sha256") + .update(canonicalJson(payload)) + .digest("hex")}`; + if (manifest.digest !== digest) { + throw new Error( + "runner_remote_provider_artifact_incompatible: provider pack manifest digest mismatch", + ); + } + const artifactEntries = [ + ["provider Node", payload.artifacts?.nodeCommand], + ["production lockfile", payload.artifacts?.productionLock], + ["OpenCode command", payload.artifacts?.opencodeCommand], + ["OpenCode executable", payload.artifacts?.opencodeExecutable], + ["OpenCode proxy", payload.artifacts?.opencodeProxy], + ["ACPX sidecar", payload.artifacts?.acpxSidecar], + ] as const; + for (const [label, artifact] of artifactEntries) { + const artifactPath = providerPackRelativePath( + artifact?.path, + `${label} path`, + ); + if ( + typeof artifact?.sha256 !== "string" || + sha256File(resolve(packRoot, artifactPath)) !== artifact.sha256 + ) { + throw new Error( + `runner_remote_provider_artifact_incompatible: ${label} digest mismatch`, + ); + } + } + if (sha256DirectoryTree(resolve(packRoot, "dist")) !== payload.distDigest) { + throw new Error( + "runner_remote_provider_artifact_incompatible: provider dist tree digest mismatch", + ); + } + const bridgeDigest = `sha256:${createHash("sha256") + .update(payload.artifacts.opencodeProxy.sha256) + .update("\n") + .update(payload.artifacts.acpxSidecar.sha256) + .update("\n") + .update(payload.distDigest) + .digest("hex")}`; + if (payload.bridgeDigest !== bridgeDigest) { + throw new Error( + "runner_remote_provider_artifact_incompatible: provider bridge digest mismatch", + ); + } + return structuredClone(manifest); +} + +export function assertRemoteRunnerBuildMetadata( + value: unknown, + requiredMode: "dial_wss" | "listen_ws", +): void { + if (!value || typeof value !== "object" || Array.isArray(value)) { + throw new Error("runner_remote_artifact_metadata_invalid"); + } + const metadata = value as Record; + if ( + metadata.schema !== RUNNERD_BUILD_METADATA_SCHEMA || + metadata.binaryName !== "paperclip-runnerd" || + metadata.packageName !== "@paperclipai/paperclip-runner" || + metadata.binaryContractVersion !== RUNNERD_BINARY_CONTRACT_VERSION + ) { + throw new Error("runner_remote_artifact_contract_incompatible"); + } + const modes = Array.isArray(metadata.prpTransportModes) + ? metadata.prpTransportModes + : []; + if (!modes.includes(requiredMode)) { + throw new Error( + `runner_remote_transport_capability_missing:${requiredMode}`, + ); + } +} + +async function stageRemoteRunnerFile(input: { + target: Extract; + runner: CommandManagedRuntimeRunner; + sourcePath: string; + targetPath: string; + mode: number; +}): Promise { + const runner = input.runner; + if (runner.syncIn) { + await runner.syncIn([ + { + operationId: `runner-stage-${randomUUID()}`, + files: [ + { + sourcePath: input.sourcePath, + targetPath: input.targetPath, + kind: "file", + mode: input.mode, + }, + ], + }, + ]); + return; + } + const bytes = readFileSync(input.sourcePath); + const directory = posix.dirname(input.targetPath); + const script = + `umask 077; mkdir -p '${directory.replaceAll("'", "'\\''")}' && ` + + `base64 -d > '${input.targetPath.replaceAll("'", "'\\''")}' && ` + + `chmod ${input.mode.toString(8)} '${input.targetPath.replaceAll("'", "'\\''")}'`; + const result = await runner.execute({ + command: "sh", + args: ["-c", script], + stdin: bytes.toString("base64"), + bypassSession: true, + }); + if (result.exitCode !== 0 || result.timedOut) { + throw new Error("runner_remote_staging_failed"); + } +} + +function archiveExcludeArgs(entries: readonly string[]): string[] { + for (const entry of entries) { + if (entry === "." || entry === ".." || !/^[A-Za-z0-9._-]+$/.test(entry)) { + throw new Error("runner_remote_checkpoint_exclusion_invalid"); + } + } + return entries.map((entry) => `--exclude=./${entry}`); +} + +export async function stageRemoteRunnerDirectory(input: { + target: Extract; + runner: CommandManagedRuntimeRunner; + sourcePath: string; + targetPath: string; + mode: number; + excludeTopLevelEntries?: readonly string[]; +}): Promise { + const excludeArgs = archiveExcludeArgs(input.excludeTopLevelEntries ?? []); + if (input.runner.syncIn) { + let stagingRoot: string | null = null; + let sourcePath = input.sourcePath; + try { + if (excludeArgs.length > 0) { + stagingRoot = mkdtempSync(join(tmpdir(), "paperclip-runner-restore-")); + const archive = execFileSync( + "tar", + [...excludeArgs, "-czf", "-", "-C", input.sourcePath, "."], + { encoding: "buffer", maxBuffer: 64 * 1024 * 1024 }, + ); + execFileSync("tar", ["-xzf", "-", "-C", stagingRoot], { + input: archive, + maxBuffer: 64 * 1024 * 1024, + }); + sourcePath = stagingRoot; + } + await input.runner.syncIn([ + { + operationId: `runner-stage-dir-${randomUUID()}`, + files: [ + { + sourcePath, + targetPath: input.targetPath, + kind: "directory", + mode: input.mode, + }, + ], + }, + ]); + } finally { + if (stagingRoot) rmSync(stagingRoot, { recursive: true, force: true }); + } + return; + } + const archive = execFileSync( + "tar", + [...excludeArgs, "-czf", "-", "-C", input.sourcePath, "."], + { encoding: "buffer", maxBuffer: 64 * 1024 * 1024 }, + ); + const escapedTarget = input.targetPath.replaceAll("'", "'\\''"); + const script = + `umask 077; mkdir -p '${escapedTarget}' && ` + + `base64 -d | tar -xzf - -C '${escapedTarget}' && ` + + `chmod ${input.mode.toString(8)} '${escapedTarget}'`; + const result = await input.runner.execute({ + command: "sh", + args: ["-c", script], + stdin: archive.toString("base64"), + bypassSession: true, + }); + if (result.exitCode !== 0 || result.timedOut) { + throw new Error("runner_remote_directory_staging_failed"); + } +} + +async function remoteRunnerPathExists(input: { + runner: CommandManagedRuntimeRunner; + path: string; + kind: "file" | "directory"; +}): Promise { + const escapedPath = input.path.replaceAll("'", "'\\''"); + const result = await input.runner.execute({ + command: "sh", + args: ["-c", `test -${input.kind === "file" ? "f" : "d"} '${escapedPath}'`], + bypassSession: true, + timeoutMs: 10_000, + }); + return result.exitCode === 0 && !result.timedOut; +} + +function assertSafeRemoteCheckpointArchive(archive: Buffer): void { + if (archive.length > MAX_REMOTE_CHECKPOINT_ARCHIVE_BYTES) { + throw new Error("runner_remote_checkpoint_archive_too_large"); + } + let names: string[]; + let verboseEntries: string[]; + try { + names = execFileSync("tar", ["-tzf", "-"], { + input: archive, + encoding: "utf8", + maxBuffer: 8 * 1024 * 1024, + timeout: 30_000, + }) + .split("\n") + .filter((line) => line.length > 0); + verboseEntries = execFileSync("tar", ["-tvzf", "-"], { + input: archive, + encoding: "utf8", + maxBuffer: 8 * 1024 * 1024, + timeout: 30_000, + }) + .split("\n") + .filter((line) => line.length > 0); + } catch { + throw new Error("runner_remote_checkpoint_archive_invalid"); + } + if ( + names.length === 0 || + names.length > MAX_REMOTE_CHECKPOINT_ENTRIES || + verboseEntries.length !== names.length + ) { + throw new Error("runner_remote_checkpoint_archive_invalid"); + } + for (const name of names) { + const normalized = name.replace(/^\.\//, ""); + if ( + name.includes("\0") || + name.startsWith("/") || + normalized.split("/").some((part) => part === "..") + ) { + throw new Error("runner_remote_checkpoint_archive_unsafe_path"); + } + } + for (const entry of verboseEntries) { + const type = entry.trimStart()[0]; + if (type !== "-" && type !== "d") { + throw new Error("runner_remote_checkpoint_archive_unsafe_entry"); + } + } + try { + execFileSync("tar", ["-xOzf", "-"], { + input: archive, + stdio: ["pipe", "pipe", "pipe"], + maxBuffer: MAX_REMOTE_CHECKPOINT_EXPANDED_BYTES, + timeout: 60_000, + }); + } catch { + throw new Error("runner_remote_checkpoint_archive_expanded_too_large"); + } +} + +function assertSafeExtractedCheckpoint(root: string): void { + const pending = [root]; + let entries = 0; + let totalBytes = 0; + while (pending.length > 0) { + const directory = pending.pop()!; + for (const entry of readdirSync(directory, { withFileTypes: true })) { + entries += 1; + if (entries > MAX_REMOTE_CHECKPOINT_ENTRIES) { + throw new Error("runner_remote_checkpoint_too_many_entries"); + } + const path = join(directory, entry.name); + const metadata = lstatSync(path); + if ( + metadata.isSymbolicLink() || + (!metadata.isDirectory() && !metadata.isFile()) + ) { + throw new Error("runner_remote_checkpoint_unsafe_entry"); + } + if (metadata.isDirectory()) { + pending.push(path); + } else { + totalBytes += metadata.size; + if (totalBytes > MAX_REMOTE_CHECKPOINT_EXPANDED_BYTES) { + throw new Error("runner_remote_checkpoint_expanded_too_large"); + } + } + } + } +} + +/** + * Copy a remote runner directory into durable local state. + * + * Some provider homes contain process-local scratch trees (for example Codex's + * `tmp/arg0` executable aliases). Those aliases can point outside the directory + * and are neither portable nor required to resume a provider session. When a + * provider supplies a native syncOut implementation, create a sibling snapshot + * first so the live runtime remains untouched and the provider's archive safety + * checks still apply to every persisted entry. + */ +export async function syncRemoteRunnerDirectoryOut(input: { + runner: CommandManagedRuntimeRunner; + sourcePath: string; + targetPath: string; + mode: number; + excludeTopLevelEntries?: readonly string[]; +}): Promise { + if ( + !(await remoteRunnerPathExists({ + runner: input.runner, + path: input.sourcePath, + kind: "directory", + })) + ) + return; + mkdirSync(resolve(input.targetPath, ".."), { recursive: true, mode: 0o700 }); + const excluded = input.excludeTopLevelEntries ?? []; + const excludeArgs = archiveExcludeArgs(excluded) + .map((argument) => `'${argument}'`) + .join(" "); + if (input.runner.syncOut) { + let syncSourcePath = input.sourcePath; + let snapshotPath: string | null = null; + if (excluded.length > 0) { + const checkpointId = randomUUID(); + snapshotPath = posix.join( + posix.dirname(input.sourcePath), + `.paperclip-checkpoint-${checkpointId}`, + ); + const archivePath = `${snapshotPath}.tar`; + const escapedSource = input.sourcePath.replaceAll("'", "'\\''"); + const escapedSnapshot = snapshotPath.replaceAll("'", "'\\''"); + const escapedArchive = archivePath.replaceAll("'", "'\\''"); + const snapshotResult = await input.runner.execute({ + command: "sh", + args: [ + "-c", + `set -e; umask 077; mkdir -p '${escapedSnapshot}'; ` + + `trap "rm -f '${escapedArchive}'" EXIT; ` + + `tar ${excludeArgs} -cf '${escapedArchive}' -C '${escapedSource}' .; ` + + `tar -xf '${escapedArchive}' -C '${escapedSnapshot}'`, + ], + bypassSession: true, + timeoutMs: 120_000, + }); + if (snapshotResult.exitCode !== 0 || snapshotResult.timedOut) { + throw new Error("runner_remote_checkpoint_snapshot_failed"); + } + syncSourcePath = snapshotPath; + } + try { + await input.runner.syncOut([ + { + operationId: `runner-checkpoint-dir-${randomUUID()}`, + files: [ + { + sourcePath: syncSourcePath, + targetPath: input.targetPath, + kind: "directory", + mode: input.mode, + }, + ], + }, + ]); + } finally { + if (snapshotPath) { + const escapedSnapshot = snapshotPath.replaceAll("'", "'\\''"); + await input.runner + .execute({ + command: "sh", + args: ["-c", `rm -rf -- '${escapedSnapshot}'`], + bypassSession: true, + timeoutMs: 30_000, + }) + .catch(() => undefined); + } + } + return; + } + const escapedSource = input.sourcePath.replaceAll("'", "'\\''"); + const result = await input.runner.execute({ + command: "sh", + args: ["-c", `tar ${excludeArgs} -czf - -C '${escapedSource}' . | base64`], + bypassSession: true, + timeoutMs: 120_000, + }); + if (result.exitCode !== 0 || result.timedOut) { + throw new Error("runner_remote_checkpoint_failed"); + } + const archive = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64"); + assertSafeRemoteCheckpointArchive(archive); + const parent = resolve(input.targetPath, ".."); + const stagingRoot = mkdtempSync(join(parent, ".paperclip-checkpoint-")); + const stagedTarget = join(stagingRoot, "payload"); + const previousTarget = join( + parent, + `.paperclip-checkpoint-previous-${randomUUID()}`, + ); + let previousMoved = false; + let replacementInstalled = false; + try { + mkdirSync(stagedTarget, { recursive: true, mode: input.mode }); + execFileSync( + "tar", + [ + "--no-same-owner", + "--no-same-permissions", + "-xzf", + "-", + "-C", + stagedTarget, + ], + { input: archive, maxBuffer: 8 * 1024 * 1024, timeout: 60_000 }, + ); + assertSafeExtractedCheckpoint(stagedTarget); + chmodSync(stagedTarget, input.mode); + if (existsSync(input.targetPath)) { + renameSync(input.targetPath, previousTarget); + previousMoved = true; + } + renameSync(stagedTarget, input.targetPath); + replacementInstalled = true; + if (previousMoved) { + rmSync(previousTarget, { recursive: true, force: true }); + previousMoved = false; + } + } catch (error) { + if ( + previousMoved && + !replacementInstalled && + !existsSync(input.targetPath) + ) { + try { + renameSync(previousTarget, input.targetPath); + previousMoved = false; + } catch { + // Leave the last durable checkpoint at previousTarget. Deleting it in + // finally would turn a failed replacement into irreversible data loss. + } + } + throw error; + } finally { + if (previousMoved && replacementInstalled) { + rmSync(previousTarget, { recursive: true, force: true }); + } + rmSync(stagingRoot, { recursive: true, force: true }); + } +} + +async function readRemoteRunnerState(input: { + runner: CommandManagedRuntimeRunner; + stateDirectory: string; +}): Promise> { + const statePath = posix.join(input.stateDirectory, "runner-state.json"); + const escapedPath = statePath.replaceAll("'", "'\\''"); + const result = await input.runner.execute({ + command: "sh", + args: ["-c", `test -f '${escapedPath}' && base64 < '${escapedPath}'`], + bypassSession: true, + timeoutMs: 10_000, + }); + if (result.exitCode !== 0 || result.timedOut) { + throw new Error("runner_remote_state_unavailable"); + } + return record( + JSON.parse( + Buffer.from(result.stdout.replace(/\s+/g, ""), "base64").toString("utf8"), + ), + ); +} + +function createRemoteRunnerProcessLauncher(input: { + target: Extract; + runner: CommandManagedRuntimeRunner; + remoteBinary: string; + runnerInstanceId: string; + ensureArtifact?: () => Promise; + onSpawn?: (meta: { + pid: number; + processGroupId: number | null; + startedAt: string; + }) => Promise; + onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise; + trace?: NativeRunTrace; + onRunnerProcessSpawned?: () => void; +}): (spec: RunnerProcessLaunchSpec) => RunnerProcessHandle { + const runner = input.runner; + return (spec) => { + const child: RunnerProcessHandle["child"] = { + pid: undefined, + exitCode: null, + signalCode: null, + kill: () => { + const pattern = `--runner-id ${input.runnerInstanceId}`; + void runner.execute({ + command: "pkill", + args: ["-f", pattern], + bypassSession: true, + timeoutMs: 10_000, + }); + return true; + }, + }; + const completion = (async () => { + if (input.ensureArtifact) { + if (input.trace) { + await input.trace.measure( + "runner.runtime.stage", + input.ensureArtifact, + { parentName: "runner.session.startup" }, + ); + } else { + await input.ensureArtifact(); + } + } + const launchStartedAtMs = Date.now(); + // The provider's onSpawn callback is optional and some sandbox command + // runners cannot report a remote pid until after the command has begun + // streaming. Signal as soon as staging is complete and the launch RPC is + // dispatched; this is late enough to avoid preview retries during staging + // and early enough to avoid a callback-dependent deadlock. + input.onRunnerProcessSpawned?.(); + await input.trace?.record({ + name: "runner.process.dispatch", + parentName: "runner.session.startup", + startedAtMs: launchStartedAtMs, + endedAtMs: Date.now(), + attributes: { target: "remote" }, + }); + const result = await runner.execute({ + command: input.remoteBinary, + args: [...spec.args], + cwd: input.target.remoteCwd, + env: processEnvironment(spec.environment), + timeoutMs: 62 * 60_000, + useSession: true, + onLog: input.onLog, + onSpawn: async (meta) => { + child.pid = meta.pid; + await input.onSpawn?.({ + pid: meta.pid, + processGroupId: null, + startedAt: new Date().toISOString(), + }); + await input.trace?.record({ + name: "runner.process.launch", + parentName: "runner.session.startup", + startedAtMs: launchStartedAtMs, + endedAtMs: Date.now(), + }); + }, + }); + child.exitCode = result.exitCode; + return { + code: result.exitCode, + signal: null, + stdout: result.stdout, + stderr: result.stderr, + }; + })(); + return { child, completion }; + }; +} + +/** Production runnerd backend seam, exported so provider wiring can be regression tested. */ export async function createRunnerdBackend(input: { db: Db; execution: NativeExecutionInput; @@ -3458,6 +4634,14 @@ export async function createRunnerdBackend(input: { startedAt: string; }) => Promise; runnerEnvironment?: NodeJS.ProcessEnv; + runnerExecutionTarget?: AdapterExecutionTarget | null; + enableRunnerPreviewIngress?: boolean; + runnerPublicUrl?: string | null; + runnerCaBundlePath?: string | null; + runnerRemoteBinaryPath?: string | null; + runnerRemoteCodexPath?: string | null; + runnerRemoteCodexNpmSpec?: string | null; + runnerRemoteProviderPackPath?: string | null; trace?: NativeRunTrace; onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise; enqueueWakeup?: ( @@ -3474,8 +4658,11 @@ export async function createRunnerdBackend(input: { }, ) => Promise; }): Promise { - if (input.execution.provider.kind !== "codex") { - throw new Error("paperclip_runner_provider_unsupported"); + const target = input.runnerExecutionTarget ?? { kind: "local" as const }; + if (target.kind === "remote" && input.execution.provider.kind !== "codex") { + throw new Error( + `runner_remote_provider_artifact_incompatible: remote ${input.execution.provider.kind} is unavailable until runnerd provider dispatch is qualified`, + ); } const authority = new PaperclipRunnerToolAuthority(input.db, { companyId: input.execution.binding.companyId, @@ -3486,86 +4673,1628 @@ export async function createRunnerdBackend(input: { workMode: input.execution.task.workMode, enqueueWakeup: input.enqueueWakeup, }); - const dynamicTools = await authority.definitions(); + const sessionScopeId = nativeSessionScopeKey(input.execution); + const existingRouter = sessionToolRouters.get(sessionScopeId); + const authorityRouter = + existingRouter ?? new SessionToolAuthorityRouter(authority); + authorityRouter.bind(authority); + sessionToolRouters.set(sessionScopeId, authorityRouter); + const dynamicTools = await authorityRouter.definitions(); const root = runnerdStateRoot(input.execution); mkdirSync(root, { recursive: true, mode: 0o700 }); - const environment = input.runnerEnvironment ?? process.env; - - const archiveContinuityState = async () => { - const archiveRoot = resolve( - root, - "continuity-breaks", - `${Date.now()}-${randomUUID()}`, + const remoteTarget = target.kind === "remote" ? target : null; + const remoteCommandRunner = remoteTarget + ? remoteTarget.transport === "ssh" + ? createSshCommandManagedRuntimeRunner({ + spec: remoteTarget.spec, + defaultCwd: remoteTarget.remoteCwd, + }) + : remoteTarget.runner + : null; + if (remoteTarget && !remoteCommandRunner) { + throw new Error( + "runner_transport_ineligible: remote process runner is unavailable", ); - mkdirSync(archiveRoot, { recursive: true, mode: 0o700 }); - for (const name of ["control-plane", "runner", "codex-home"]) { - const source = resolve(root, name); - if (existsSync(source)) renameSync(source, resolve(archiveRoot, name)); + } + const remoteRuntimeRoot = remoteTarget + ? posix.join( + remoteTarget.remoteCwd, + ".paperclip-runtime", + "paperclip-runner", + ) + : null; + const requiresRemoteProviderPack = + remoteTarget !== null && + (input.execution.provider.kind === "opencode" || + input.execution.provider.kind === "acpx"); + const configuredProviderPackRoot = + input.runnerRemoteProviderPackPath?.trim() || null; + let expectedProviderPackManifest: RemoteProviderPackManifest | null = null; + if (requiresRemoteProviderPack) { + if ( + !configuredProviderPackRoot || + !existsSync(configuredProviderPackRoot) || + !lstatSync(configuredProviderPackRoot).isDirectory() + ) { + throw new Error( + "runner_remote_provider_artifact_incompatible: configure PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH with the build-owned provider pack", + ); + } + expectedProviderPackManifest = readRemoteProviderPackManifest( + configuredProviderPackRoot, + ); + } + const stagedRemoteProviderPackRoot = remoteRuntimeRoot + ? posix.join(remoteRuntimeRoot, "provider-pack") + : null; + let activeRemoteProviderPackRoot: string | null = null; + const remoteBinary = remoteRuntimeRoot + ? posix.join(remoteRuntimeRoot, "bin", "paperclip-runnerd") + : null; + const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null; + const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null; + if (explicitRemoteCodex && remoteCodexNpmSpec) { + throw new Error("runner_remote_codex_source_conflict"); + } + const remoteCodexBinary = + remoteRuntimeRoot && input.execution.provider.kind === "codex" + ? remoteCodexNpmSpec + ? posix.join( + remoteRuntimeRoot, + "harnesses", + "codex", + "node_modules", + ".bin", + "codex", + ) + : posix.join(remoteRuntimeRoot, "bin", "codex") + : null; + const remoteSessionDigest = createHash("sha256") + .update(nativeSessionKey(input.execution)) + .digest("hex"); + const remoteSessionRoot = remoteRuntimeRoot + ? posix.join(remoteRuntimeRoot, "sessions", remoteSessionDigest) + : null; + const remoteStateDirectory = remoteSessionRoot + ? posix.join(remoteSessionRoot, "runner") + : undefined; + const remoteRunnerFilesystemRoot = remoteSessionRoot + ? posix.join(remoteSessionRoot, "filesystem") + : null; + const persistenceProfile = resolveNativeHarnessPersistenceProfile( + input.execution, + ); + const sandboxLeaseAcquisition = + remoteTarget?.transport === "sandbox" + ? (remoteTarget.sandboxLeaseAcquisition ?? null) + : null; + const remotePersistencePath = ( + directory: NativeHarnessPersistenceDirectory, + ): string | null => + directory.location === "runner" + ? (remoteStateDirectory ?? null) + : remoteRunnerFilesystemRoot + ? posix.join(remoteRunnerFilesystemRoot, directory.name) + : null; + const sourceRuntimeContext = + "runtimeContext" in input.execution ? input.execution.runtimeContext : null; + const remoteRuntimeContext: NativeRuntimeContextSnapshot | null = + remoteRunnerFilesystemRoot && sourceRuntimeContext + ? { + ...sourceRuntimeContext, + instructions: { + ...sourceRuntimeContext.instructions, + bundle: { + ...sourceRuntimeContext.instructions.bundle, + rootPath: posix.join( + remoteRunnerFilesystemRoot, + "context", + "instructions", + ), + }, + }, + skills: sourceRuntimeContext.skills.map((skill, index) => ({ + ...skill, + bundle: { + ...skill.bundle, + rootPath: posix.join( + remoteRunnerFilesystemRoot, + "context", + "skills", + `${index}-${skill.bundle.digest.slice(0, 12)}`, + ), + }, + })), + } + : sourceRuntimeContext; + let remotePrepared = false; + let selectedRemoteMode: "dial_wss" | "listen_ws" | null = null; + let remoteCaBundleMapping: { sourcePath: string; targetPath: string } | null = + null; + let resolveRemoteRunnerProcessSpawned: (() => void) | null = null; + const remoteRunnerProcessSpawned = remoteTarget + ? new Promise((resolveSpawned) => { + resolveRemoteRunnerProcessSpawned = resolveSpawned; + }) + : Promise.resolve(); + + const verifyRemoteRunner = async ( + requiredMode: "dial_wss" | "listen_ws", + executable = remoteBinary, + ) => { + if (!remoteTarget || !remoteCommandRunner || !executable) return; + const metadataResult = await remoteCommandRunner.execute({ + command: executable, + args: ["--build-metadata"], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 30_000, + }); + if (metadataResult.exitCode !== 0 || metadataResult.timedOut) { + throw new Error("runner_remote_artifact_verification_failed"); + } + let metadata: Record; + try { + metadata = JSON.parse(metadataResult.stdout) as Record; + } catch (error) { + throw new Error("runner_remote_artifact_metadata_invalid", { + cause: error, + }); + } + assertRemoteRunnerBuildMetadata(metadata, requiredMode); + }; + + const verifyRemoteCodex = async (executable = remoteCodexBinary) => { + if (!remoteTarget || !remoteCommandRunner || !executable) return; + const versionResult = await remoteCommandRunner.execute({ + command: executable, + args: ["--version"], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 30_000, + }); + if (versionResult.exitCode !== 0 || versionResult.timedOut) { + throw new Error("runner_remote_codex_artifact_verification_failed"); + } + const versionOutput = `${versionResult.stdout}\n${versionResult.stderr}`; + const version = versionOutput.match(/\bcodex-cli\s+(\d+\.\d+\.\d+)\b/)?.[1]; + if (version !== REMOTE_PROVIDER_PACK_PINS.codex) { + throw new Error( + `runner_remote_provider_artifact_incompatible: expected Codex ${REMOTE_PROVIDER_PACK_PINS.codex}, received ${version ?? "an unrecognized version"}`, + ); } }; - const backend = createNativeSessionBackend(input.execution, { + const verifyRemoteProviderPack = async (packRoot: string) => { + if (!remoteTarget || !remoteCommandRunner || !expectedProviderPackManifest) + return; + const expected = Buffer.from( + canonicalJson(expectedProviderPackManifest), + "utf8", + ).toString("base64"); + const providerNodeCommand = posix.join( + packRoot, + expectedProviderPackManifest.payload.artifacts.nodeCommand.path, + ); + const verifyScript = [ + "const fs=require('node:fs')", + "const crypto=require('node:crypto')", + "const path=require('node:path')", + "const root=process.argv[1]", + "const expected=Buffer.from(process.argv[2],'base64').toString('utf8')", + "const actual=fs.readFileSync(path.join(root,'provider-pack.json'),'utf8').trim()", + "const canonical=(v)=>Array.isArray(v)?'['+v.map(canonical).join(',')+']':v&&typeof v==='object'?'{'+Object.keys(v).sort().map(k=>JSON.stringify(k)+':'+canonical(v[k])).join(',')+'}':JSON.stringify(v)", + "const manifest=JSON.parse(actual)", + "if(canonical(manifest)!==expected)throw new Error('manifest mismatch')", + "const hash=(p)=>'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(path.join(root,p))).digest('hex')", + "const tree=(treeRoot)=>{const digest=crypto.createHash('sha256');const visit=(directory,prefix='')=>{for(const entry of fs.readdirSync(directory,{withFileTypes:true}).sort((a,b)=>a.name.localeCompare(b.name))){const relative=prefix?prefix+'/'+entry.name:entry.name;const absolute=path.join(directory,entry.name);if(entry.isDirectory()){digest.update('directory\\0'+relative+'\\n');visit(absolute,relative)}else if(entry.isFile()){digest.update('file\\0'+relative+'\\0'+'sha256:'+crypto.createHash('sha256').update(fs.readFileSync(absolute)).digest('hex')+'\\n')}else if(entry.isSymbolicLink()){digest.update('symlink\\0'+relative+'\\0'+fs.readlinkSync(absolute)+'\\n')}else throw new Error('unsupported dist entry '+relative)}};visit(treeRoot);return 'sha256:'+digest.digest('hex')}", + "for(const name of ['nodeCommand','productionLock','opencodeCommand','opencodeExecutable','opencodeProxy','acpxSidecar']){const artifact=manifest.payload.artifacts[name];if(hash(artifact.path)!==artifact.sha256)throw new Error(name+' digest mismatch')}", + "if(tree(path.join(root,'dist'))!==manifest.payload.distDigest)throw new Error('dist tree digest mismatch')", + "const version=process.versions.node.split('.').map(Number)", + "const minimum=manifest.payload.pins.nodeMinimum.split('.').map(Number)", + "if(version[0]JSON.parse(fs.readFileSync(path.join(root,'node_modules',...pkg.split('/'),'package.json'),'utf8')).version", + "const expectedPackages={acpx:manifest.payload.pins.acpx,'@agentclientprotocol/claude-agent-acp':manifest.payload.pins.claudeAcp,'@agentclientprotocol/codex-acp':manifest.payload.pins.codexAcp,'opencode-ai':manifest.payload.pins.opencode}", + "for(const [pkg,version] of Object.entries(expectedPackages))if(packageVersion(pkg)!==version)throw new Error(pkg+' version mismatch')", + ].join(";"); + const verified = await remoteCommandRunner.execute({ + command: providerNodeCommand, + args: ["-e", verifyScript, packRoot, expected], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 30_000, + }); + if (verified.exitCode !== 0 || verified.timedOut) { + throw new Error( + `runner_remote_provider_artifact_incompatible: provider pack verification failed (${verified.stderr.trim().slice(-1_024)})`, + ); + } + const opencodeCommand = posix.join( + packRoot, + expectedProviderPackManifest.payload.artifacts.opencodeCommand.path, + ); + const opencodeVersion = await remoteCommandRunner.execute({ + command: opencodeCommand, + args: ["--version"], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 30_000, + }); + if ( + opencodeVersion.exitCode !== 0 || + opencodeVersion.timedOut || + opencodeVersion.stdout.trim() !== REMOTE_PROVIDER_PACK_PINS.opencode + ) { + throw new Error( + "runner_remote_provider_artifact_incompatible: OpenCode version mismatch", + ); + } + }; + + const discoverPreinstalledProviderPack = async () => { + if (!remoteTarget || !remoteCommandRunner) return null; + const result = await remoteCommandRunner.execute({ + command: "sh", + args: [ + "-c", + 'for candidate in /opt/paperclip-runner/provider-pack "$HOME/.local/share/paperclip-runner/provider-pack"; do if [ -f "$candidate/provider-pack.json" ]; then printf \'%s\\n\' "$candidate"; break; fi; done', + ], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 10_000, + }); + if (result.exitCode !== 0 || result.timedOut) return null; + return parseRemoteExecutableCandidate(result.stdout); + }; + + const discoverPreinstalledExecutable = async ( + name: "paperclip-runnerd" | "codex", + ) => { + if (!remoteTarget || !remoteCommandRunner) return null; + const result = await remoteCommandRunner.execute({ + command: "sh", + args: [ + "-c", + `candidate="$HOME/.local/bin/${name}"; ` + + `if [ -x "$candidate" ]; then printf '%s\\n' "$candidate"; ` + + `else command -v ${name} 2>/dev/null || true; fi`, + ], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 10_000, + }); + if (result.exitCode !== 0 || result.timedOut) return null; + return parseRemoteExecutableCandidate(result.stdout); + }; + + const linkPreinstalledExecutable = async ( + sourcePath: string, + targetPath: string, + ) => { + if (!remoteTarget || !remoteCommandRunner) return; + const escapedSource = sourcePath.replaceAll("'", "'\\''"); + const escapedTarget = targetPath.replaceAll("'", "'\\''"); + const escapedDirectory = posix.dirname(targetPath).replaceAll("'", "'\\''"); + const result = await remoteCommandRunner.execute({ + command: "sh", + args: [ + "-c", + `umask 077; mkdir -p '${escapedDirectory}' && ` + + `ln -sfn '${escapedSource}' '${escapedTarget}'`, + ], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 10_000, + }); + if (result.exitCode !== 0 || result.timedOut) { + throw new Error("runner_remote_preinstalled_link_failed"); + } + }; + + const prepareRemoteRunner = async ( + requiredMode: "dial_wss" | "listen_ws", + ) => { + if ( + !remoteTarget || + !remoteCommandRunner || + !remoteBinary || + remotePrepared + ) + return; + selectedRemoteMode = requiredMode; + let usedPreinstalledRunner = false; + const explicitRemoteBinary = input.runnerRemoteBinaryPath?.trim() || null; + if (mayUsePreinstalledRunnerArtifact(explicitRemoteBinary)) { + const preinstalledRunner = await measureNativeRunnerSpan( + input.trace, + "runner.artifact.discover", + () => discoverPreinstalledExecutable("paperclip-runnerd"), + ); + if (preinstalledRunner) { + try { + await measureNativeRunnerSpan( + input.trace, + "runner.artifact.verify_preinstalled", + () => verifyRemoteRunner(requiredMode, preinstalledRunner), + ); + await measureNativeRunnerSpan( + input.trace, + "runner.artifact.link", + () => linkPreinstalledExecutable(preinstalledRunner, remoteBinary), + ); + usedPreinstalledRunner = true; + await input.onLog?.( + "stderr", + "[paperclip-runner] using preinstalled runnerd from the sandbox image\n", + ); + } catch { + usedPreinstalledRunner = false; + } + } + } + if (!usedPreinstalledRunner) { + const sourceBinary = + explicitRemoteBinary ?? defaultCapabilityRunnerdBinary(); + if (!existsSync(sourceBinary)) { + throw new Error("runner_remote_artifact_unavailable"); + } + if (!explicitRemoteBinary) { + const platform = await remoteCommandRunner.execute({ + command: "sh", + args: ["-c", "uname -s; uname -m"], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 10_000, + }); + const [remoteOs = "", remoteArch = ""] = platform.stdout + .trim() + .split(/\r?\n/); + const localOs = + process.platform === "darwin" + ? "Darwin" + : process.platform === "linux" + ? "Linux" + : process.platform; + const localArch = + process.arch === "x64" + ? "x86_64" + : process.arch === "arm64" + ? "aarch64" + : process.arch; + const archMatches = + remoteArch === localArch || + (localArch === "aarch64" && remoteArch === "arm64"); + if ( + platform.exitCode !== 0 || + platform.timedOut || + remoteOs !== localOs || + !archMatches + ) { + throw new Error( + "runner_remote_artifact_platform_mismatch: configure PAPERCLIP_RUNNER_REMOTE_BINARY_PATH for the remote OS and architecture", + ); + } + } + await stageRemoteRunnerFile({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: sourceBinary, + targetPath: remoteBinary, + mode: 0o700, + }); + } + await measureNativeRunnerSpan(input.trace, "runner.artifact.verify", () => + verifyRemoteRunner(requiredMode), + ); + if (remoteCodexBinary && explicitRemoteCodex) { + if (!existsSync(explicitRemoteCodex)) { + throw new Error("runner_remote_codex_artifact_unavailable"); + } + await stageRemoteRunnerFile({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: explicitRemoteCodex, + targetPath: remoteCodexBinary, + mode: 0o700, + }); + await verifyRemoteCodex(); + } + if (remoteCodexBinary && remoteCodexNpmSpec) { + let usedPreinstalledCodex = false; + const preinstalledCodex = await measureNativeRunnerSpan( + input.trace, + "harness.artifact.discover", + () => discoverPreinstalledExecutable("codex"), + ); + if (preinstalledCodex) { + try { + await measureNativeRunnerSpan( + input.trace, + "harness.artifact.verify_preinstalled", + () => verifyRemoteCodex(preinstalledCodex), + ); + await measureNativeRunnerSpan( + input.trace, + "harness.artifact.link", + () => + linkPreinstalledExecutable(preinstalledCodex, remoteCodexBinary), + ); + usedPreinstalledCodex = true; + await input.onLog?.( + "stderr", + "[paperclip-runner] using preinstalled Codex from the sandbox image\n", + ); + } catch { + usedPreinstalledCodex = false; + } + } + if (!usedPreinstalledCodex) { + const installRoot = posix.join( + remoteRuntimeRoot!, + "harnesses", + "codex", + ); + const installResult = await remoteCommandRunner.execute({ + command: "npm", + args: [ + "install", + "--prefix", + installRoot, + "--no-audit", + "--no-fund", + remoteCodexNpmSpec, + ], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 180_000, + }); + if (installResult.exitCode !== 0 || installResult.timedOut) { + throw new Error("runner_remote_codex_install_failed"); + } + } + await measureNativeRunnerSpan( + input.trace, + "harness.artifact.verify", + () => verifyRemoteCodex(), + ); + } + if (remoteCodexBinary && !explicitRemoteCodex && !remoteCodexNpmSpec) { + const preinstalledCodex = await measureNativeRunnerSpan( + input.trace, + "harness.artifact.discover", + () => discoverPreinstalledExecutable("codex"), + ); + if (!preinstalledCodex) { + throw new Error( + "runner_remote_codex_artifact_unavailable: install codex in the sandbox image or configure PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC", + ); + } + await measureNativeRunnerSpan( + input.trace, + "harness.artifact.verify_preinstalled", + () => verifyRemoteCodex(preinstalledCodex), + ); + await measureNativeRunnerSpan(input.trace, "harness.artifact.link", () => + linkPreinstalledExecutable(preinstalledCodex, remoteCodexBinary), + ); + await measureNativeRunnerSpan( + input.trace, + "harness.artifact.verify", + () => verifyRemoteCodex(), + ); + await input.onLog?.( + "stderr", + "[paperclip-runner] using preinstalled Codex from the sandbox image\n", + ); + } + if ( + requiresRemoteProviderPack && + configuredProviderPackRoot && + stagedRemoteProviderPackRoot + ) { + let preinstalledProviderPack = await discoverPreinstalledProviderPack(); + if (preinstalledProviderPack) { + try { + await measureNativeRunnerSpan( + input.trace, + "provider_pack.verify_preinstalled", + () => verifyRemoteProviderPack(preinstalledProviderPack!), + ); + const escapedSource = preinstalledProviderPack.replaceAll( + "'", + "'\\''", + ); + const escapedTarget = stagedRemoteProviderPackRoot.replaceAll( + "'", + "'\\''", + ); + const escapedParent = posix + .dirname(stagedRemoteProviderPackRoot) + .replaceAll("'", "'\\''"); + const linked = await remoteCommandRunner.execute({ + command: "sh", + args: [ + "-c", + `umask 077; mkdir -p '${escapedParent}' && rm -rf '${escapedTarget}' && ln -s '${escapedSource}' '${escapedTarget}'`, + ], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 10_000, + }); + if (linked.exitCode !== 0 || linked.timedOut) { + throw new Error( + "runner_remote_provider_artifact_incompatible: preinstalled provider pack could not be linked", + ); + } + activeRemoteProviderPackRoot = stagedRemoteProviderPackRoot; + await input.onLog?.( + "stderr", + "[paperclip-runner] using manifest-matched provider pack from the sandbox image\n", + ); + } catch { + preinstalledProviderPack = null; + } + } + if (!preinstalledProviderPack) { + if (!remoteCommandRunner.syncIn) { + throw new Error( + "runner_remote_provider_artifact_incompatible: this remote transport cannot stage a provider pack; preinstall the exact manifest-matched pack", + ); + } + const escapedPackRoot = stagedRemoteProviderPackRoot.replaceAll( + "'", + "'\\''", + ); + const cleared = await remoteCommandRunner.execute({ + command: "sh", + args: ["-c", `rm -rf '${escapedPackRoot}'`], + cwd: remoteTarget.remoteCwd, + bypassSession: true, + timeoutMs: 10_000, + }); + if (cleared.exitCode !== 0 || cleared.timedOut) { + throw new Error( + "runner_remote_provider_artifact_incompatible: stale provider pack could not be replaced", + ); + } + await stageRemoteRunnerDirectory({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: configuredProviderPackRoot, + targetPath: stagedRemoteProviderPackRoot, + mode: 0o700, + }); + await measureNativeRunnerSpan(input.trace, "provider_pack.verify", () => + verifyRemoteProviderPack(stagedRemoteProviderPackRoot), + ); + activeRemoteProviderPackRoot = stagedRemoteProviderPackRoot; + } + } + remotePrepared = true; + }; + + const inspectRemoteHarnessState = async (): Promise<{ + complete: boolean; + runnerState: Record | null; + }> => { + if (!remoteCommandRunner || !remoteStateDirectory) { + return { complete: false, runnerState: null }; + } + const requirements = persistenceProfile.directories.flatMap((directory) => { + const path = remotePersistencePath(directory); + if (!path) return []; + const escaped = path.replaceAll("'", "'\\''"); + return directory.name === "runner" + ? [`test -f '${escaped}/runner-state.json'`] + : [`test -d '${escaped}'`]; + }); + const escapedRunnerState = posix + .join(remoteStateDirectory, "runner-state.json") + .replaceAll("'", "'\\''"); + const inspected = await remoteCommandRunner.execute({ + command: "sh", + args: [ + "-c", + `${requirements.join(" && ")} && base64 < '${escapedRunnerState}'`, + ], + bypassSession: true, + timeoutMs: 10_000, + }); + if (inspected.exitCode !== 0 || inspected.timedOut) { + return { complete: false, runnerState: null }; + } + let runnerState: Record; + try { + runnerState = record( + JSON.parse( + Buffer.from(inspected.stdout.replace(/\s+/g, ""), "base64").toString( + "utf8", + ), + ), + ); + } catch { + throw new Error("runner_harness_state_mismatch"); + } + if ( + runnerState.runnerInstanceId !== input.runnerInstanceId || + runnerState.normalizedSessionId !== nativeSessionKey(input.execution) + ) { + throw new Error("runner_harness_state_mismatch"); + } + const providerSessionIdentity = + providerSessionIdentityFromRunnerState(runnerState); + if (!providerSessionIdentityIsPresent(providerSessionIdentity)) { + throw new Error("runner_harness_state_mismatch"); + } + const previousManifest = compatibleNativeHarnessBackupManifests({ + root, + execution: input.execution, + runnerInstanceId: input.runnerInstanceId, + })[0]?.manifest; + if ( + previousManifest && + !providerSessionIdentityTransitionIsAllowed({ + execution: input.execution, + previous: previousManifest.providerSessionIdentity, + current: providerSessionIdentity, + }) + ) { + throw new Error("runner_harness_state_mismatch"); + } + return { complete: true, runnerState }; + }; + + const recordInPlaceHarnessReuse = async ( + runnerState: Record, + startedAtMs = Date.now(), + ) => { + const now = Date.now(); + const attributes = { + provider: input.execution.provider.kind, + harness: input.execution.session.driverKind, + lifecycleMode: input.execution.session.lifecyclePolicy.mode, + stateSource: "sandbox_filesystem", + bytesTransferred: 0, + }; + await input.trace?.record({ + name: "harness_state.reuse", + startedAtMs, + endedAtMs: now, + attributes, + }); + await input.trace?.record({ + name: "provider.session.resume", + startedAtMs: now, + endedAtMs: now, + attributes: { + provider: input.execution.provider.kind, + harness: input.execution.session.driverKind, + identityPresent: providerSessionIdentityIsPresent( + providerSessionIdentityFromRunnerState(runnerState), + ), + }, + }); + }; + + const recordHarnessBackupStampForCurrentLease = async ( + backup: VerifiedHarnessBackup, + ) => { + if (remoteTarget?.transport !== "sandbox" || !remoteTarget.leaseId) { + return; + } + const leaseRow = await input.db + .select({ metadata: environmentLeases.metadata }) + .from(environmentLeases) + .where(eq(environmentLeases.id, remoteTarget.leaseId)) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!leaseRow) throw new Error("runner_harness_backup_lease_missing"); + const stamp = createNativeHarnessBackupStamp({ + manifestPath: resolve(backup.root, "manifest.json"), + normalizedSessionId: backup.manifest.normalizedSessionId, + runnerInstanceId: backup.manifest.runnerInstanceId, + completedAt: backup.manifest.completedAt, + }); + const updated = await input.db + .update(environmentLeases) + .set({ + metadata: { + ...(leaseRow.metadata ?? {}), + nativeHarnessBackup: stamp, + }, + updatedAt: new Date(), + }) + .where(eq(environmentLeases.id, remoteTarget.leaseId)) + .returning({ id: environmentLeases.id }) + .then((rows) => rows[0] ?? null); + if (!updated) throw new Error("runner_harness_backup_lease_missing"); + }; + + const restoreVerifiedHarnessBackup = async () => { + if (!remoteTarget || !remoteCommandRunner) { + throw new Error("runner_harness_backup_unavailable"); + } + const backup = verifyNativeHarnessBackup({ + root, + execution: input.execution, + runnerInstanceId: input.runnerInstanceId, + }); + if (!backup) throw new Error("runner_harness_backup_unavailable"); + await measureNativeRunnerSpan( + input.trace, + "harness_state.failover_restore", + async () => { + for (const directory of persistenceProfile.directories) { + const targetPath = remotePersistencePath(directory); + if (!targetPath) throw new Error("runner_harness_state_mismatch"); + await stageRemoteRunnerDirectory({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: resolve(backup.root, directory.name), + targetPath, + mode: 0o700, + }); + } + }, + { + attributes: { + provider: input.execution.provider.kind, + harness: input.execution.session.driverKind, + lifecycleMode: input.execution.session.lifecyclePolicy.mode, + stateSource: "verified_failover_backup", + bytesTransferred: backup.bytes, + }, + }, + ); + const restored = await inspectRemoteHarnessState(); + if ( + !restored.complete || + !restored.runnerState || + canonicalJson( + providerSessionIdentityFromRunnerState(restored.runnerState), + ) !== canonicalJson(backup.manifest.providerSessionIdentity) + ) { + throw new Error("runner_harness_state_mismatch"); + } + // A deliberately non-reusable environment receives a fresh provider lease + // for every turn. Stamp that new lease as soon as the verified host backup + // has been restored so a later provider/bootstrap failure can still clean + // the ephemeral sandbox up without discarding the only durable copy. + await recordHarnessBackupStampForCurrentLease(backup); + return backup; + }; + + const materializeRemoteHarnessLaunchState = async () => { + if (!remoteTarget || !remoteCommandRunner) return; + for (const directory of persistenceProfile.directories) { + if (directory.location !== "filesystem") continue; + const targetPath = remotePersistencePath(directory); + if (!targetPath) throw new Error("runner_harness_state_mismatch"); + const escapedTarget = targetPath.replaceAll("'", "'\\''"); + const created = await remoteCommandRunner.execute({ + command: "sh", + args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`], + bypassSession: true, + timeoutMs: 10_000, + }); + if (created.exitCode !== 0 || created.timedOut) { + throw new Error("runner_remote_directory_staging_failed"); + } + + // Codex launch credentials are intentionally excluded from failover + // backups. Re-materialize only those launch-time files into a fresh or + // replacement sandbox after the durable history has been restored. + if (directory.name !== "codex-home") continue; + const localDirectory = resolve(root, directory.name); + for (const name of ["auth.json", "config.toml"] as const) { + const sourcePath = resolve(localDirectory, name); + if (!existsSync(sourcePath)) continue; + await stageRemoteRunnerFile({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath, + targetPath: posix.join(targetPath, name), + mode: 0o600, + }); + } + } + }; + + const ensureRemoteRunner = async () => { + await measureNativeRunnerSpan( + input.trace, + "stage.sync", + async () => { + if (!selectedRemoteMode) { + throw new Error("runner_remote_transport_mode_unresolved"); + } + try { + await verifyRemoteRunner(selectedRemoteMode); + await verifyRemoteCodex(); + if (requiresRemoteProviderPack) { + if (!activeRemoteProviderPackRoot) { + throw new Error( + "runner_remote_provider_artifact_incompatible: provider pack was not prepared", + ); + } + await verifyRemoteProviderPack(activeRemoteProviderPackRoot); + } + } catch { + remotePrepared = false; + await prepareRemoteRunner(selectedRemoteMode); + } + await measureNativeRunnerSpan(input.trace, "stage.asset.home", () => + measureNativeRunnerSpan( + input.trace, + "session.checkpoint.restore", + async () => { + if ( + remoteTarget?.transport === "sandbox" && + remoteCommandRunner + ) { + const acquisitionRecordedAtMs = Date.now(); + const backupAvailable = harnessBackupCandidates(root).some( + (candidate) => + existsSync(resolve(candidate, "manifest.json")), + ); + const restoreIntoCreatedSandbox = + shouldRestoreNativeHarnessBackupIntoSandbox({ + acquisitionOutcome: + sandboxLeaseAcquisition?.outcome ?? null, + reusableLeaseConfigured: + remoteTarget.reusableLeaseConfigured, + backupAvailable, + }); + await input.trace?.record({ + name: "sandbox.lease.acquisition", + startedAtMs: acquisitionRecordedAtMs, + endedAtMs: acquisitionRecordedAtMs, + attributes: { + provider: remoteTarget.providerKey ?? "sandbox", + harness: input.execution.session.driverKind, + lifecycleMode: input.execution.session.lifecyclePolicy.mode, + outcome: sandboxLeaseAcquisition?.outcome ?? "unknown", + stateSource: + sandboxLeaseAcquisition?.outcome === "replacement" || + restoreIntoCreatedSandbox + ? "verified_failover_backup" + : sandboxLeaseAcquisition?.outcome === "resumed" + ? "sandbox_filesystem" + : "new_sandbox", + bytesTransferred: 0, + }, + }); + if (sandboxLeaseAcquisition?.outcome === "resumed") { + const reuseStartedAtMs = Date.now(); + const state = await measureNativeRunnerSpan( + input.trace, + "sandbox.lease.resume", + inspectRemoteHarnessState, + { + attributes: { + provider: remoteTarget.providerKey ?? "sandbox", + harness: input.execution.session.driverKind, + lifecycleMode: + input.execution.session.lifecyclePolicy.mode, + outcome: "resumed", + }, + }, + ); + if (!state.complete || !state.runnerState) { + throw new Error("runner_harness_state_mismatch"); + } + await recordInPlaceHarnessReuse( + state.runnerState, + reuseStartedAtMs, + ); + } else if (sandboxLeaseAcquisition?.outcome === "replacement") { + await measureNativeRunnerSpan( + input.trace, + "sandbox.lease.replacement", + restoreVerifiedHarnessBackup, + { + attributes: { + provider: remoteTarget.providerKey ?? "sandbox", + harness: input.execution.session.driverKind, + lifecycleMode: + input.execution.session.lifecyclePolicy.mode, + outcome: "replacement", + reason: sandboxLeaseAcquisition.reason ?? "unknown", + }, + }, + ); + } else if (restoreIntoCreatedSandbox) { + await measureNativeRunnerSpan( + input.trace, + "sandbox.lease.replacement", + restoreVerifiedHarnessBackup, + { + attributes: { + provider: remoteTarget.providerKey ?? "sandbox", + harness: input.execution.session.driverKind, + lifecycleMode: + input.execution.session.lifecyclePolicy.mode, + outcome: "created", + reason: "reuse_disabled", + }, + }, + ); + } else { + const reuseStartedAtMs = Date.now(); + const state = await inspectRemoteHarnessState(); + if (state.complete && state.runnerState) { + // Re-entry while this newly-created lease is already running (for + // example a transport reconnect) still uses the in-place state. + await recordInPlaceHarnessReuse( + state.runnerState, + reuseStartedAtMs, + ); + } else if (backupAvailable) { + // A continuation that has a durable backup but no recorded reusable + // lease was not provider-confirmed lost. Never silently create a new + // provider session from that ambiguous state. + throw new Error("runner_harness_state_mismatch"); + } + } + await materializeRemoteHarnessLaunchState(); + } else if (remoteTarget && remoteCommandRunner) { + // Local and generic SSH execution retain their existing checkpoint + // behavior. The manifest-only failover gate applies to managed sandbox + // replacement, where provider lease provenance is available. + for (const directory of persistenceProfile.directories) { + const localDirectory = resolve(root, directory.name); + const remoteDirectory = remotePersistencePath(directory); + if ( + remoteDirectory && + existsSync(localDirectory) && + !(await remoteRunnerPathExists({ + runner: remoteCommandRunner, + path: + directory.name === "runner" + ? posix.join(remoteDirectory, "runner-state.json") + : remoteDirectory, + kind: directory.name === "runner" ? "file" : "directory", + })) + ) { + await stageRemoteRunnerDirectory({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: localDirectory, + targetPath: remoteDirectory, + mode: 0o700, + excludeTopLevelEntries: directory.excludeTopLevelEntries, + }); + } + } + } + }, + { + attributes: { + mode: remoteTarget?.transport ?? "local", + lifecycleMode: input.execution.session.lifecyclePolicy.mode, + }, + }, + ), + ); + if ( + remoteTarget && + remoteCommandRunner && + remoteRunnerFilesystemRoot && + sourceRuntimeContext && + remoteRuntimeContext + ) { + await measureNativeRunnerSpan( + input.trace, + "stage.asset.runtime_context", + async () => { + await stageRemoteRunnerDirectory({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: sourceRuntimeContext.instructions.bundle.rootPath, + targetPath: remoteRuntimeContext.instructions.bundle.rootPath, + mode: 0o555, + }); + for ( + let index = 0; + index < sourceRuntimeContext.skills.length; + index += 1 + ) { + await stageRemoteRunnerDirectory({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: + sourceRuntimeContext.skills[index]!.bundle.rootPath, + targetPath: + remoteRuntimeContext.skills[index]!.bundle.rootPath, + mode: 0o555, + }); + } + await stageRemoteRunnerFile({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: resolve(root, "runtime-context.json"), + targetPath: posix.join( + remoteRunnerFilesystemRoot, + "runtime-context.json", + ), + mode: 0o600, + }); + }, + ); + } + if (remoteTarget && remoteCommandRunner && remoteCaBundleMapping) { + const caBundleMapping = remoteCaBundleMapping; + await measureNativeRunnerSpan( + input.trace, + "stage.asset.ca_bundle", + () => + stageRemoteRunnerFile({ + target: remoteTarget, + runner: remoteCommandRunner, + sourcePath: caBundleMapping.sourcePath, + targetPath: caBundleMapping.targetPath, + mode: 0o600, + }), + ); + } + }, + { + attributes: { + target: remoteTarget?.transport ?? "local", + lifecycleMode: input.execution.session.lifecyclePolicy.mode, + }, + }, + ); + }; + + const checkpointRemoteRunner = async () => { + if ( + !remoteCommandRunner || + !remoteStateDirectory || + !remoteRunnerFilesystemRoot + ) + return; + // Transport release also runs when provider bootstrap failed. In that case + // runnerd has no durable provider identity (and may not have created the + // provider persistence directory at all), so attempting a failover backup + // would replace the original provider error with + // `runner_harness_state_mismatch`. Only checkpoint a harness that runnerd + // has proved complete. A malformed or identity-conflicting state still + // throws from inspectRemoteHarnessState and therefore fails closed. + const checkpointable = await inspectRemoteHarnessState(); + if (!checkpointable.complete) return; + const backupSpanAttributes = { + provider: input.execution.provider.kind, + harness: input.execution.session.driverKind, + lifecycleMode: input.execution.session.lifecyclePolicy.mode, + stateSource: "sandbox_filesystem", + bytesTransferred: 0, + }; + await measureNativeRunnerSpan( + input.trace, + "session.checkpoint.persist", + () => + measureNativeRunnerSpan( + input.trace, + "harness_state.backup.persist", + async () => { + const runnerState = await readRemoteRunnerState({ + runner: remoteCommandRunner, + stateDirectory: remoteStateDirectory, + }); + if ( + runnerState.runnerInstanceId !== input.runnerInstanceId || + runnerState.normalizedSessionId !== + nativeSessionKey(input.execution) + ) { + throw new Error("runner_harness_state_mismatch"); + } + const providerSessionIdentity = + providerSessionIdentityFromRunnerState(runnerState); + if (!providerSessionIdentityIsPresent(providerSessionIdentity)) { + throw new Error("runner_harness_state_mismatch"); + } + + const backupRoot = harnessBackupRoot(root); + mkdirSync(backupRoot, { recursive: true, mode: 0o700 }); + const pendingRoot = resolve(backupRoot, `.pending-${randomUUID()}`); + mkdirSync(pendingRoot, { recursive: true, mode: 0o700 }); + try { + for (const directory of persistenceProfile.directories) { + const sourcePath = remotePersistencePath(directory); + if (!sourcePath) + throw new Error("runner_harness_state_mismatch"); + const targetPath = resolve(pendingRoot, directory.name); + await syncRemoteRunnerDirectoryOut({ + runner: remoteCommandRunner, + sourcePath, + targetPath, + mode: 0o700, + excludeTopLevelEntries: directory.excludeTopLevelEntries, + }); + if (!existsSync(targetPath)) { + throw new Error("runner_harness_state_mismatch"); + } + } + const manifest = buildNativeHarnessBackupManifest({ + backupRoot: pendingRoot, + execution: input.execution, + runnerInstanceId: input.runnerInstanceId, + providerSessionIdentity, + sourceProviderLeaseId: + sandboxLeaseAcquisition?.providerLeaseId ?? + remoteTarget?.leaseId ?? + input.durableEnvironmentLeaseId ?? + "unknown", + }); + backupSpanAttributes.bytesTransferred = + manifest.directories.reduce( + (total, directory) => total + directory.bytes, + 0, + ); + const temporaryManifest = resolve( + pendingRoot, + "manifest.json.tmp", + ); + const manifestPath = resolve(pendingRoot, "manifest.json"); + writeFileSync(temporaryManifest, JSON.stringify(manifest), { + encoding: "utf8", + mode: 0o600, + }); + renameSync(temporaryManifest, manifestPath); + + const currentRoot = resolve(backupRoot, "current"); + const previousRoot = resolve(backupRoot, "previous"); + rmSync(previousRoot, { recursive: true, force: true }); + let movedCurrent = false; + if (existsSync(currentRoot)) { + renameSync(currentRoot, previousRoot); + movedCurrent = true; + } + try { + renameSync(pendingRoot, currentRoot); + if ( + remoteTarget?.transport === "sandbox" && + remoteTarget.leaseId + ) { + await recordHarnessBackupStampForCurrentLease({ + root: currentRoot, + manifest, + bytes: backupSpanAttributes.bytesTransferred, + }); + } + } catch (error) { + if (existsSync(currentRoot)) { + rmSync(currentRoot, { recursive: true, force: true }); + } + if ( + movedCurrent && + existsSync(previousRoot) && + !existsSync(currentRoot) + ) { + renameSync(previousRoot, currentRoot); + } + throw error; + } + rmSync(previousRoot, { recursive: true, force: true }); + } finally { + rmSync(pendingRoot, { recursive: true, force: true }); + } + }, + { + attributes: backupSpanAttributes, + }, + ), + { parentName: "task.settle" }, + ); + }; + + const remoteProcessLauncher = + remoteTarget && remoteCommandRunner && remoteBinary + ? createRemoteRunnerProcessLauncher({ + target: remoteTarget, + runner: remoteCommandRunner, + remoteBinary, + runnerInstanceId: input.runnerInstanceId, + ensureArtifact: ensureRemoteRunner, + onSpawn: input.onSpawn, + onLog: input.onLog, + trace: input.trace, + onRunnerProcessSpawned: () => resolveRemoteRunnerProcessSpawned?.(), + }) + : undefined; + const runnerExecution: NativeExecutionInput = remoteTarget + ? { + ...input.execution, + workspace: { + ...input.execution.workspace, + cwd: remoteTarget.remoteCwd, + }, + } + : input.execution; + const effectiveRunnerEnvironment: NodeJS.ProcessEnv = remoteRuntimeRoot + ? { + ...(input.runnerEnvironment ?? process.env), + HOME: remoteTarget!.remoteCwd, + CODEX_HOME: posix.join(remoteTarget!.remoteCwd, ".codex"), + } + : (input.runnerEnvironment ?? process.env); + const archiveContinuityState = async () => { + const archiveToken = `${Date.now()}-${randomUUID()}`; + const archiveRoot = resolve(root, "continuity-breaks", archiveToken); + mkdirSync(archiveRoot, { recursive: true, mode: 0o700 }); + for (const name of [ + "control-plane", + "runner", + "codex-home", + "opencode", + "acpx", + ]) { + const source = resolve(root, name); + if (existsSync(source)) renameSync(source, resolve(archiveRoot, name)); + } + if (remoteCommandRunner && remoteSessionRoot) { + const escapedSource = remoteSessionRoot.replaceAll("'", "'\\''"); + const archivedRemote = `${remoteSessionRoot}.continuity-break-${archiveToken}`; + const escapedArchive = archivedRemote.replaceAll("'", "'\\''"); + const result = await remoteCommandRunner.execute({ + command: "sh", + args: [ + "-c", + `if test -d '${escapedSource}'; then mv '${escapedSource}' '${escapedArchive}'; fi`, + ], + bypassSession: true, + timeoutMs: 30_000, + }); + if (result.exitCode !== 0 || result.timedOut) { + throw new Error("runner_continuity_break_archive_failed"); + } + } + remotePrepared = false; + }; + const backend = createNativeSessionBackend(runnerExecution, { runnerInstanceId: input.runnerInstanceId, onSpawn: input.onSpawn, dynamicTools, - dynamicToolHandler: (call) => authority.execute(call), + dynamicToolHandler: (call) => authorityRouter.execute(call), + acpxDynamicToolHandler: (call) => authorityRouter.execute(call), + opencodeRuntimeDirectory: resolve( + resolvePaperclipInstanceRoot(), + "runtime", + "paperclip-runner", + "opencode", + ), + acpxRuntimeDirectory: resolve( + resolvePaperclipInstanceRoot(), + "runtime", + "paperclip-runner", + "acpx", + ), codexTransportFactory: (recoveryContext) => createRunnerdCodexTransport({ - runnerBinary: resolvePaperclipRunnerBinary(), + provider: + input.execution.provider.kind === "codex" + ? "codex" + : input.execution.provider.kind === "opencode" + ? "opencode" + : input.execution.provider.kind === "acpx" + ? "acpx" + : undefined, + ...(input.execution.provider.kind === "acpx" + ? { + acpxAgent: input.execution.provider.agent, + acpxPermissionMode: input.execution.provider.permissionMode, + acpxPermissionModePinned: + input.execution.schema === + "paperclip.native-execution-input.v4", + acpxRuntimeDirectory: remoteRunnerFilesystemRoot + ? posix.join(remoteRunnerFilesystemRoot, "acpx") + : resolve( + resolvePaperclipInstanceRoot(), + "runtime", + "paperclip-runner", + "acpx", + ), + } + : {}), + ...(expectedProviderPackManifest && stagedRemoteProviderPackRoot + ? { + providerNodeCommand: posix.join( + stagedRemoteProviderPackRoot, + expectedProviderPackManifest.payload.artifacts.nodeCommand.path, + ), + opencodeCommand: posix.join( + stagedRemoteProviderPackRoot, + expectedProviderPackManifest.payload.artifacts + .opencodeExecutable.path, + ), + opencodeProxyPath: posix.join( + stagedRemoteProviderPackRoot, + expectedProviderPackManifest.payload.artifacts.opencodeProxy + .path, + ), + acpxSidecarPath: posix.join( + stagedRemoteProviderPackRoot, + expectedProviderPackManifest.payload.artifacts.acpxSidecar.path, + ), + } + : {}), stateDirectory: root, - environment, + runnerStateDirectory: remoteStateDirectory, + readRunnerState: + remoteStateDirectory && remoteCommandRunner + ? () => + readRemoteRunnerState({ + runner: remoteCommandRunner, + stateDirectory: remoteStateDirectory, + }) + : undefined, + runnerBinary: remoteBinary ?? resolvePaperclipRunnerBinary(), + codexCommand: remoteCodexBinary ?? undefined, + sourceCodexHome: remoteTarget + ? resolveSourceCodexHome(input.runnerEnvironment ?? process.env) + : undefined, + runnerProcessLauncher: remoteProcessLauncher, + runnerReconnectGraceMs: remoteTarget ? 120_000 : undefined, + environment: effectiveRunnerEnvironment, lifecyclePolicy: input.execution.session.lifecyclePolicy, runtimeContext: "runtimeContext" in input.execution ? input.execution.runtimeContext : null, + runnerRuntimeContext: remoteRuntimeContext, + runnerFilesystemRoot: remoteRunnerFilesystemRoot ?? undefined, + opencodeRuntimeDirectory: remoteRunnerFilesystemRoot + ? posix.join(remoteRunnerFilesystemRoot, "opencode") + : undefined, resumeDynamicTools: dynamicTools, - providerRecoveryPolicy: recoveryContext?.providerRecoveryPolicy, + providerRecoveryPolicy: + recoveryContext?.providerRecoveryPolicy ?? + (input.execution.provider.kind === "acpx" && + input.execution.interactionResponses.length > 0 + ? "allow_replacement_after_governed_wait" + : undefined), prpIdentity: { runnerInstanceId: input.runnerInstanceId, environmentLeaseId: input.durableEnvironmentLeaseId ?? input.execution.binding.executionWorkspaceId, runId: input.execution.binding.runId, - normalizedSessionId: nativeSessionKey(input.execution), + normalizedSessionId: + input.execution.session.normalizedSessionId ?? + `session-${input.execution.binding.runId}`, turnId: `turn-${input.execution.binding.runId}`, itemId: `item-${input.execution.binding.runId}`, }, - controlPlaneRegistration: async (controlPlaneAuthority) => { - const selectedAtMs = Date.now(); - await input.trace?.record({ - name: "runner.transport.selected", - parentName: "runner.transport.connect", - startedAtMs: selectedAtMs, - endedAtMs: selectedAtMs, - attributes: { - mode: "local_loopback", - connectionOwner: "runnerd", - }, - }); - await input.onLog?.( - "stderr", - "[paperclip-runner] transport mode=local_loopback state=connecting\n", - ); - const registration = await measureNativeRunnerSpan( + controlPlaneRegistration: (authority) => + measureNativeRunnerSpan( input.trace, - "runner.prp.route.register", - () => - registerRunnerPrpAuthority({ - companyId: input.execution.binding.companyId, - runId: input.execution.binding.runId, - authority: controlPlaneAuthority, - }), - { parentName: "runner.transport.connect" }, - ); - return { - ...registration, - startupFailureCode: "runner_local_connect_failed" as const, - }; - }, + "runner.transport.connect", + async () => { + if (target.kind === "local") { + const selectedAtMs = Date.now(); + await input.trace?.record({ + name: "runner.transport.selected", + parentName: "runner.transport.connect", + startedAtMs: selectedAtMs, + endedAtMs: selectedAtMs, + attributes: { + mode: "local_loopback", + connectionOwner: "runnerd", + }, + }); + await input.onLog?.( + "stderr", + "[paperclip-runner] transport mode=local_loopback state=connecting\n", + ); + const registration = await measureNativeRunnerSpan( + input.trace, + "runner.prp.route.register", + () => + registerRunnerPrpAuthority({ + companyId: input.execution.binding.companyId, + runId: input.execution.binding.runId, + authority, + }), + ); + return { + ...registration, + startupFailureCode: "runner_local_connect_failed" as const, + }; + } + + const requiredMode = + target.transport === "sandbox" && + target.effectiveCapabilities?.runnerWebSocketIngress === true + ? "listen_ws" + : "dial_wss"; + if ( + requiredMode === "listen_ws" && + input.enableRunnerPreviewIngress !== true + ) { + throw new Error("runner_ingress_unavailable"); + } + let transport: PaperclipRunnerTransport; + if (requiredMode === "dial_wss") { + // Validate eligibility before staging any artifact. + transport = await measureNativeRunnerSpan( + input.trace, + "runner.transport.resolve", + () => + resolvePaperclipRunnerTransport({ + target, + runId: input.execution.binding.runId, + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: input.runnerPublicUrl, + runnerCaBundlePath: input.runnerCaBundlePath, + enableRunnerPreviewIngress: + input.enableRunnerPreviewIngress === true, + }), + ); + if ( + transport.mode === "direct_outbound" && + transport.caBundlePath && + !existsSync(transport.caBundlePath) + ) { + throw new Error( + "runner_direct_wss_failed: configured runner CA bundle is unavailable", + ); + } + await measureNativeRunnerSpan( + input.trace, + "runner.artifact.prepare", + () => prepareRemoteRunner(requiredMode), + ); + } else { + await measureNativeRunnerSpan( + input.trace, + "runner.artifact.prepare", + () => prepareRemoteRunner(requiredMode), + ); + // Provider endpoint acquisition happens only after runnerd is staged + // and its listener capability has been verified. + transport = await measureNativeRunnerSpan( + input.trace, + "runner.ingress.acquire", + () => + resolvePaperclipRunnerTransport({ + target, + runId: input.execution.binding.runId, + localConnectUrl: "ws://127.0.0.1/unused", + runnerPublicUrl: input.runnerPublicUrl, + runnerCaBundlePath: input.runnerCaBundlePath, + enableRunnerPreviewIngress: true, + }), + ); + } + + const selectedAtMs = Date.now(); + await input.trace?.record({ + name: "runner.transport.selected", + parentName: "runner.transport.connect", + startedAtMs: selectedAtMs, + endedAtMs: selectedAtMs, + attributes: { + mode: transport.mode, + connectionOwner: + transport.mode === "provider_ingress" + ? "paperclip" + : "runnerd", + }, + }); + + await input.onLog?.( + "stderr", + `[paperclip-runner] transport mode=${transport.mode} state=connecting\n`, + ); + + if (transport.mode === "direct_outbound") { + const inbound = await measureNativeRunnerSpan( + input.trace, + "runner.prp.route.register", + () => + registerRunnerPrpAuthority({ + companyId: input.execution.binding.companyId, + runId: input.execution.binding.runId, + authority, + }), + { parentName: "runner.transport.connect" }, + ); + let caBundlePath = transport.caBundlePath; + if (caBundlePath && remoteBinary) { + const remoteCaBundlePath = posix.join( + posix.dirname(remoteBinary), + "runner-ca-bundle.pem", + ); + remoteCaBundleMapping = { + sourcePath: caBundlePath, + targetPath: remoteCaBundlePath, + }; + caBundlePath = remoteCaBundlePath; + } + return { + connection: { + mode: "connect" as const, + connectUrl: transport.connectUrl, + ...(caBundlePath ? { caBundlePath } : {}), + }, + startupFailureCode: "runner_direct_wss_failed" as const, + release: async () => { + await inbound.release(); + await checkpointRemoteRunner(); + }, + }; + } + + if (transport.mode !== "provider_ingress") { + throw new Error("runner_transport_mode_changed_after_dispatch"); + } + let outbound: ReturnType | null = + null; + let activation: Promise | null = null; + return { + connection: { + mode: "listen" as const, + listenAddress: transport.listenAddress, + listenPort: transport.listenPort, + listenPath: transport.listenPath, + }, + activate: () => { + activation = measureNativeRunnerSpan( + input.trace, + "runner.transport.activation", + async () => { + await measureNativeRunnerSpan( + input.trace, + "runner.ingress.wait_for_process", + () => remoteRunnerProcessSpawned, + ); + outbound = connectRunnerPrpIngress({ + authority, + endpoint: transport.ingress, + onStateChange: (state, failureCode) => { + void input.onLog?.( + "stderr", + `[paperclip-runner] transport mode=provider_ingress state=${state}${failureCode ? ` failure=${failureCode}` : ""}\n`, + ); + }, + }); + }, + { parentName: "runner.session.startup" }, + ); + }, + ready: async () => { + await measureNativeRunnerSpan( + input.trace, + "runner.transport.ready", + async () => { + await activation; + if (!outbound) + throw new Error("runner_ingress_unavailable"); + await measureNativeRunnerSpan( + input.trace, + "runner.prp.authenticate", + () => outbound!.ready, + ); + }, + { parentName: "runner.session.startup" }, + ); + }, + get failure() { + return outbound?.failure; + }, + startupFailureCode: "runner_ingress_unavailable" as const, + release: async () => { + if (outbound) await outbound.close(); + else await transport.ingress.close(); + await checkpointRemoteRunner(); + }, + }; + }, + ), }).transport, }); - return { descriptor: () => backend.descriptor(), openSession: (sessionInput) => backend.openSession(sessionInput), diff --git a/server/src/services/native-runtime/native-session-resume.test.ts b/server/src/services/native-runtime/native-session-resume.test.ts index 8277a07282..a6d95541bf 100644 --- a/server/src/services/native-runtime/native-session-resume.test.ts +++ b/server/src/services/native-runtime/native-session-resume.test.ts @@ -20,6 +20,7 @@ function execution(runId: string, cwd = "/workspace") { agentId, workspace: { id: runId, cwd, repoUrl: null, repoRef: null, branchName: null }, normalizedSessionId, + provider: "codex", completionContract: { id: "70000000-0000-4000-8000-000000000007", sha256: `sha256:${"a".repeat(64)}`, @@ -43,6 +44,7 @@ function planningExecution(runId: string, revisionId: string) { agentId, workspace: { id: runId, cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, normalizedSessionId, + provider: "codex", executionMode: "plan", planningContext: { documentId: "80000000-0000-4000-8000-000000000008", @@ -189,7 +191,7 @@ describe("rebindNativeSessionCheckpoint", () => { }); describe("buildNativeExecutionInput wake projection", () => { - it("writes native v4 and pins the complete Codex configuration", () => { + it("writes native v4 and pins every provider's complete effective configuration", () => { const common = { companyId, runId: currentRunId, @@ -203,14 +205,36 @@ describe("buildNativeExecutionInput wake projection", () => { } as const; const codex = buildNativeExecutionInput({ ...common, + provider: "codex", codexApprovalPolicy: "on-request", }); + const opencode = buildNativeExecutionInput({ + ...common, + provider: "opencode", + model: "openrouter/z-ai/glm-5.2", + opencodePermissionMode: "ask", + }); + const acpx = buildNativeExecutionInput({ + ...common, + provider: "acpx", + acpxAgent: "claude", + model: "claude-sonnet-5", + acpxPermissionMode: "deny-all", + }); expect(codex).toMatchObject({ schema: "paperclip.native-execution-input.v4", provider: { kind: "codex", approvalPolicy: "on-request" }, }); - expect(JSON.stringify(codex)) + expect(opencode).toMatchObject({ + schema: "paperclip.native-execution-input.v4", + provider: { kind: "opencode", permissionMode: "ask" }, + }); + expect(acpx).toMatchObject({ + schema: "paperclip.native-execution-input.v4", + provider: { kind: "acpx", permissionMode: "deny-all" }, + }); + expect(JSON.stringify([codex, opencode, acpx])) .not.toMatch(/OPENAI_API_KEY|ANTHROPIC_API_KEY|AWS_SECRET_ACCESS_KEY|PAPERCLIP_API_KEY/); }); @@ -257,6 +281,8 @@ describe("buildNativeExecutionInput wake projection", () => { branchName: null, }, normalizedSessionId, + provider: "opencode", + model: "openrouter/z-ai/glm-5.2", completionContract: { id: "70000000-0000-4000-8000-000000000007", sha256: `sha256:${"a".repeat(64)}`, diff --git a/server/src/services/native-runtime/runtime-mode.test.ts b/server/src/services/native-runtime/runtime-mode.test.ts index 376aa6f4af..50ac8154e0 100644 --- a/server/src/services/native-runtime/runtime-mode.test.ts +++ b/server/src/services/native-runtime/runtime-mode.test.ts @@ -172,6 +172,21 @@ describe("resolveNativeRuntimeMode", () => { })); }); + it("keeps a persisted OpenCode recovery on its immutable driver", () => { + expect(resolveHeartbeatNativeRuntimeMode({ + ...eligible, + enabled: false, + persisted: { + runtimeMode: "native", + runtimeModeReason: "eligible_opt_in", + runtimeModeResolvedAt: new Date(), + driverKind: "opencode_server", + }, + })).toEqual(expect.objectContaining({ + profile: { mode: "native", backend: "opencode_server", protocolVersion: 1 }, + })); + }); + it("rejects an explicit native profile outside the approved boundary", () => { expect(resolveNativeRuntimeMode({ ...eligible, agent: { ...eligible.agent, adapterType: "claude_local" } })) .toEqual(expect.objectContaining({ kind: "legacy", reason: "direct_adapter" })); @@ -179,16 +194,19 @@ describe("resolveNativeRuntimeMode", () => { .toThrow(NativeRuntimeEligibilityError); }); - it("rejects remote targets for fresh paperclip_runner starts", () => { - expect(() => resolveNativeRuntimeMode({ + it("admits remote targets only through paperclip_runner", () => { + expect(resolveNativeRuntimeMode({ ...eligible, target: { kind: "remote" }, runtimeConfig: {}, adapterConfig: { provider: "codex" }, agent: { ...eligible.agent, adapterType: "paperclip_runner" }, - })).toThrow(expect.objectContaining({ - code: "paperclip_runner_environment_unsupported", - })); + })).toMatchObject({ kind: "native" }); + expect(resolveNativeRuntimeMode({ + ...eligible, + target: { kind: "remote" }, + agent: { ...eligible.agent, adapterType: "codex_local" }, + })).toMatchObject({ kind: "legacy", reason: "direct_adapter" }); }); it("allows paperclip_runner to use a transient local workspace for projectless issues", () => { diff --git a/server/src/services/native-runtime/runtime-mode.ts b/server/src/services/native-runtime/runtime-mode.ts index aa1741fe8d..532da664ec 100644 --- a/server/src/services/native-runtime/runtime-mode.ts +++ b/server/src/services/native-runtime/runtime-mode.ts @@ -40,7 +40,7 @@ export type NativeRuntimeResolution = reason: "eligible_opt_in"; profile: { mode: "native"; - backend: "codex_app_server"; + backend: "codex_app_server" | "opencode_server" | "acpx_runtime"; protocolVersion: 1; }; authorityDecision: NativeStatusDecision; @@ -129,12 +129,6 @@ export function resolveNativeRuntimeMode(input: { "Paperclip Runner requires a standard, planning, or ask task.", ); } - if (!input.target || input.target.kind !== "local") { - throw ineligible( - "paperclip_runner_environment_unsupported", - "Paperclip Runner currently requires a local execution environment.", - ); - } const rollout = resolveNativeMigrationStatus({ facts: { applicationEnabled: true }, priorIssueStatus: "in_progress", @@ -267,13 +261,17 @@ export function resolveHeartbeatNativeRuntimeMode(input: { ); } const driverKind = input.persisted.driverKind; - const backend = driverKind === null - || driverKind === undefined - || driverKind === "codex" - || driverKind === "codex_app_server" - ? "codex_app_server" - : null; - if (!backend) { + const backend = driverKind === "opencode_server" + ? "opencode_server" + : driverKind === "acpx_runtime" + ? "acpx_runtime" + : driverKind === null + || driverKind === undefined + || driverKind === "codex" + || driverKind === "codex_app_server" + ? "codex_app_server" + : null; + if (backend === null) { throw ineligible( "paperclip_runner_driver_unsupported", `Persisted Paperclip Runner driver is unsupported: ${driverKind}`, diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index a8c56cddab..b3ac878531 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -26,10 +26,13 @@ export type { ControlPlanePort, HarnessRuntimeRequestKind, HarnessRuntimeRequestResolution, + NativeAcpxAgent, + NativeAcpxPermissionMode, NativeCodexApprovalPolicy, NativeExecutionInput, NativeExecutionInputV4, NativeInteractionResponseEnvelope, + NativeOpenCodePermissionMode, NativePlanningContext, NativeRunEvent, NativeRunResult, @@ -45,8 +48,12 @@ export type { PrpStructuredRunResult, PrpTerminalState, PrpVerificationReasonCode, + PrpWireConnection, ReplayControlPlaneEventsInput, + RunnerProcessHandle, + RunnerProcessLaunchSpec, StrictCompletionContractInput, + TransportCloseReason, } from "@paperclipai/paperclip-runner"; export type DurablePrpControlPlane = import("@paperclipai/paperclip-runner").DurablePrpControlPlane; @@ -90,6 +97,8 @@ export const parseNativeExecutionInput = runner.parseNativeExecutionInput; export const parseNativeRuntimeContext = runner.parseNativeRuntimeContext; export const parsePaperclipQuestionSet = runner.parsePaperclipQuestionSet; export const parsePaperclipQuestionResponse = runner.parsePaperclipQuestionResponse; +export const resolveQualifiedAcpxProfile = runner.resolveQualifiedAcpxProfile; +export const resolveSourceCodexHome = runner.resolveSourceCodexHome; export const validatePrpEvent = runner.validatePrpEvent; export const validatePrpStructuredRunResult = runner.validatePrpStructuredRunResult; diff --git a/ui/src/pages/InstanceExperimentalSettings.test.tsx b/ui/src/pages/InstanceExperimentalSettings.test.tsx index 5d4b49d1bc..ca60ef52c3 100644 --- a/ui/src/pages/InstanceExperimentalSettings.test.tsx +++ b/ui/src/pages/InstanceExperimentalSettings.test.tsx @@ -73,6 +73,7 @@ function defaultExperimentalSettings(): InstanceExperimentalSettingsPayload { return { enableEnvironments: false, enableNativeRunner: false, + enableRunnerPreviewIngress: false, enableManagedSandboxOnly: false, enableIsolatedWorkspaces: false, enableStreamlinedLeftNavigation: true,