feat(runner): add remote execution substrate (#12638)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner gives native runs a durable and governed execution path. > - The current native path runs on the control-plane host. > - Remote environments need an authenticated execution-target contract. > - The contract must not change direct adapters or enable new runtimes by default. > - This pull request adds the remote execution substrate and Daytona ingress. > - The benefit is a bounded base for later remote runner transport work. ## Linked Issues or Issue Description Refs #12616. Refs #12352. **Subsystem affected** Cross-cutting. This change touches runner transport, server orchestration, plugin contracts, and shared settings. **Problem or motivation** Native execution cannot resolve an authenticated runner ingress through a remote environment. The server also lacks one provider-neutral contract for remote execution targets. **Proposed solution** Add a default-off runner preview ingress capability. Add transport-neutral runner connectivity. Add remote execution target and lifecycle handling. Add a Daytona ingress implementation with redacted credentials. **Alternatives considered** A provider-specific server path would duplicate orchestration and authorization. A public endpoint without an environment contract would weaken the trust boundary. **Roadmap alignment** This work supports the Cloud and Sandbox agents milestone. It also supports self-healing runs and governed tool access. ## What Changed - Added execution-target traits for local, SSH, and sandbox environments. - Added plugin RPC contracts for runner ingress endpoints. - Added authenticated Daytona preview ingress. - Added transport-neutral PRP outbound connections. - Added remote runner artifact verification and fail-closed provider selection. - Added bounded native session resume, cancellation, and lifecycle recovery. - Preserved Codex-only selection for fresh experimental runner starts. - Preserved all direct adapter execution and finalization paths. - Removed stale Pi provider-pack requirements that security review rejected. - Kept the rollout controls off by default. - Did not change pnpm-lock.yaml, Cargo, database migrations, or GitHub workflows. ## Verification - GitHub Actions will run the repository test, typecheck, build, security, and policy gates. - Focused tests cover ingress validation, redaction, execution targets, remote lifecycle, cancellation, resume, and legacy adapter selection. - Local tests were not run. The requested verification policy uses GitHub Actions for this series. - `git diff --check origin/master...HEAD` passes. - The diff contains 52 files. ## Risks - Remote execution crosses a trust boundary. - The implementation validates target capabilities, artifact digests, provider-pack pins, and connection metadata. - The feature remains default-off. - Fresh native selection remains Codex-only. - Existing direct adapters remain on the legacy path. - This PR does not yet make remote Codex runnable. The next PR adds the Rust WSS and TLS transport. ## Model Used OpenAI Codex with GPT-5.6. The work used high-reasoning agent mode, repository tools, GitHub tools, and parallel code-audit agents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with Fixes: / Closes / Refs OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
GitHub
parent
209680409d
commit
0a422fda52
|
|
@ -0,0 +1,41 @@
|
|||
# Paperclip Runner in Daytona sandboxes
|
||||
|
||||
## Intended topology
|
||||
|
||||
Paperclip creates the run and a short-lived, one-use runner bootstrap ticket. The Daytona sandbox starts only `paperclip-runnerd`; runnerd then starts the configured provider (Codex initially). Runnerd listens on a fixed sandbox-local port and Paperclip connects through Daytona's authenticated preview WebSocket ingress. The provider never receives a Paperclip API credential or Daytona preview credential.
|
||||
|
||||
```text
|
||||
Paperclip control plane
|
||||
|
|
||||
| outbound WSS + X-Daytona-Preview-Token
|
||||
v
|
||||
Daytona preview proxy -> paperclip-runnerd:43127 -> Codex app-server
|
||||
```
|
||||
|
||||
The PRP identity binds company, issue, agent, run, environment lease, runner instance, normalized session, artifact version, artifact digest, and catalog digest. After the one-use ticket challenge succeeds, runnerd receives a renewable connection lease. Revocation, expiry, replay cursors, and the durable outbox continue to work across transient network loss and sandbox/provider restarts.
|
||||
|
||||
## Network policy
|
||||
|
||||
- Allow Paperclip to make outbound TCP 443 connections to the configured Daytona preview origin. The Paperclip host does not need public ingress.
|
||||
- Allow the sandbox provider's private preview proxy to reach runnerd on TCP 43127. Do not expose the port with a public sandbox or signed URL.
|
||||
- Allow the sandbox outbound access only to destinations explicitly required by the provider runtime.
|
||||
- Use Daytona's `wss://` preview URL with normal certificate and hostname validation. The preview token is sent only as the `X-Daytona-Preview-Token` header by Paperclip.
|
||||
- Do not put the bootstrap ticket in argv, files, provider environment, logs, or model context. Inject it into runnerd's initial environment/secret channel; runnerd already removes it from its environment immediately.
|
||||
- Runnerd is the only process allowed to reach PRP. The provider communicates with runnerd over inherited pipes.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
1. Paperclip allocates the Daytona environment and persists its environment lease.
|
||||
2. Paperclip creates a runner ticket with a very short expiry and binds it to the run, environment lease, runner digest/version, and allowed catalog digest.
|
||||
3. The sandbox startup command launches the pinned runnerd artifact in listener mode on `0.0.0.0:43127` and the run-specific PRP path. The ticket is supplied separately as secret environment material.
|
||||
4. Paperclip acquires an authenticated Daytona preview endpoint and dials runnerd. After PRP authentication, Paperclip sends `run.prepare`; only then may runnerd start Codex and advertise the run-authorized tools.
|
||||
5. Suspend/drain/revoke commands stop new turns and durably flush terminal events. Paperclip revokes the lease before destroying or recycling the sandbox.
|
||||
6. A recovered sandbox reuses its durable runner state and an unexpired connection lease; it must not mint a second provider session when a resumable one exists.
|
||||
|
||||
## Deployment-mode boundary
|
||||
|
||||
Daytona always selects provider ingress for `paperclip_runner`; it does not fall
|
||||
back to direct outbound WSS or a legacy callback bridge. Same-host native runs
|
||||
keep loopback `ws://`, and other remote providers may use direct outbound
|
||||
`wss://` only when an operator explicitly configures a reachable Paperclip
|
||||
runner URL. Legacy adapters retain their existing transport paths.
|
||||
|
|
@ -19,9 +19,41 @@ All environment variables that Paperclip uses for server configuration.
|
|||
| `PAPERCLIP_DEPLOYMENT_MODE` | `local_trusted` | Runtime mode override |
|
||||
| `PAPERCLIP_DEPLOYMENT_EXPOSURE` | `private` | Exposure policy when deployment mode is `authenticated` |
|
||||
| `PAPERCLIP_API_URL` | (auto-derived) | Paperclip API base URL. When set externally (e.g., via Kubernetes ConfigMap, load balancer, or reverse proxy), the server preserves the value instead of deriving it from the listen host and port. Useful for deployments where the public-facing URL differs from the local bind address. |
|
||||
| `PAPERCLIP_RUNNER_PUBLIC_URL` | (unset) | Explicit `wss://` base URL used only when a remote `paperclip_runner` target dials Paperclip directly. Paperclip appends `/api/runner/v1/connect/<runId>`; the reverse proxy must forward WebSocket upgrades for that route. This value is never inferred from request headers. Daytona ignores it and uses provider ingress. |
|
||||
| `PAPERCLIP_RUNNER_CA_BUNDLE_PATH` | (unset) | Optional PEM CA bundle for direct runner WSS. Platform roots remain enabled. There is no insecure TLS bypass. |
|
||||
| `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` | (host build) | Host-local path to a `paperclip-runnerd` artifact built for the remote target OS and architecture. Required when Paperclip and the remote sandbox do not share a compatible platform; build metadata and the required transport mode are verified before launch. |
|
||||
| `PAPERCLIP_RUNNER_REMOTE_CODEX_PATH` | (unset) | Optional host-local path to a Codex executable built for the remote target OS and architecture. For remote Codex-backed runners, Paperclip stages and verifies this executable beside `paperclip-runnerd`. |
|
||||
| `PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC` | (unset) | Optional pinned npm package spec (for example, `@openai/codex@0.148.0`) installed inside each fresh remote lease when its Codex harness is not baked into the sandbox image. Mutually exclusive with `PAPERCLIP_RUNNER_REMOTE_CODEX_PATH`; Paperclip verifies the installed executable before starting `runnerd`. |
|
||||
| `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` | (unset) | Host-local path to the immutable provider pack built by `pnpm --filter @paperclipai/paperclip-runner build:provider-pack`. The pack includes its target-built Node 24.11 runtime, locked production dependencies, OpenCode proxy/executable, and ACPX sidecar. Remote OpenCode and ACPX fail closed without it. A preinstalled pack is accepted only when its complete digested manifest matches this build-owned pack; otherwise Paperclip stages this pack into the sandbox. |
|
||||
| `PAPERCLIP_HIDDEN_SETTINGS` | (unset) | Comma-separated settings surfaces to hide from the UI and floor at the API, for operators hosting Paperclip for others (managed cloud, internal shared server). See [Hiding settings surfaces](#hiding-settings-surfaces). |
|
||||
| `PAPERCLIP_SETTING_DEFAULTS` | (unset) | JSON object replacing the schema default of selected instance settings, for hosting operators. See [Operator setting defaults](#operator-setting-defaults). |
|
||||
|
||||
Daytona connectivity for `paperclip_runner` uses authenticated preview WSS and
|
||||
is gated by the instance experimental setting `enableRunnerPreviewIngress`
|
||||
(default `false`). The setting has no effect on legacy adapters or callback
|
||||
bridges.
|
||||
|
||||
### Preinstalled remote runner images
|
||||
|
||||
Remote sandbox images may preinstall `paperclip-runnerd`, `codex`, and the
|
||||
provider pack at `/opt/paperclip-runner/provider-pack` instead of
|
||||
paying the upload and npm-install cost on every fresh lease. Put both executable
|
||||
names on the sandbox user's `PATH`; `$HOME/.local/bin` is checked explicitly
|
||||
before `PATH`. Paperclip verifies runner build metadata, the selected PRP
|
||||
transport capability, Codex startup, the provider-pack digest, exact harness
|
||||
pins, Node compatibility, and packaged bridge digests before linking artifacts
|
||||
into the run-specific runtime directory. A missing or incompatible executable falls back
|
||||
to `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and
|
||||
`PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC` (or
|
||||
`PAPERCLIP_RUNNER_REMOTE_CODEX_PATH`) without changing the selected transport.
|
||||
OpenCode and ACPX instead fall back only to
|
||||
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH`; they never start a provider
|
||||
process on the Paperclip host for a remote target.
|
||||
The Daytona environment editor's **Configure image** action can create this
|
||||
image without a separate container registry: install the executables in its
|
||||
setup sandbox, finish setup, and Paperclip captures and promotes the resulting
|
||||
Daytona snapshot for future leases.
|
||||
|
||||
### Hiding settings surfaces
|
||||
|
||||
`PAPERCLIP_HIDDEN_SETTINGS` takes keys from the registry in
|
||||
|
|
|
|||
|
|
@ -2965,6 +2965,7 @@ describe("sandbox adapter execution targets", () => {
|
|||
incrementalSessionOutput: false,
|
||||
concurrentSyncOperations: false,
|
||||
duplexCommandStream,
|
||||
runnerWebSocketIngress: false,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -6364,6 +6365,7 @@ describe("EffectiveSandboxCapabilities deprecated alias", () => {
|
|||
incrementalSessionOutput: false,
|
||||
concurrentSyncOperations: false,
|
||||
duplexCommandStream: false,
|
||||
runnerWebSocketIngress: false,
|
||||
};
|
||||
const aliased: EffectiveSandboxCapabilities = snapshot;
|
||||
expect(aliased).toEqual(snapshot);
|
||||
|
|
|
|||
|
|
@ -80,6 +80,7 @@ import {
|
|||
} from "./acpx-engine/startup-timing.js";
|
||||
import type { RuntimeProgressSink, RuntimeStatusSink } from "./runtime-progress.js";
|
||||
import type { LocalProcessSandboxOptions } from "./local-process-sandbox.js";
|
||||
import type { RunnerIngressEndpoint } from "./runner-connectivity.js";
|
||||
|
||||
export type { RuntimeProgressSink } from "./runtime-progress.js";
|
||||
|
||||
|
|
@ -145,6 +146,8 @@ export interface EffectiveExecutionCapabilities {
|
|||
readonly incrementalSessionOutput: boolean;
|
||||
readonly concurrentSyncOperations: boolean;
|
||||
readonly duplexCommandStream: boolean;
|
||||
/** Provider can expose a private authenticated WebSocket endpoint for runnerd. */
|
||||
readonly runnerWebSocketIngress: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -153,6 +156,13 @@ export interface EffectiveExecutionCapabilities {
|
|||
*/
|
||||
export interface EffectiveSandboxCapabilities extends EffectiveExecutionCapabilities {}
|
||||
|
||||
export interface SandboxLeaseAcquisition {
|
||||
outcome: "created" | "resumed" | "replacement";
|
||||
providerLeaseId: string;
|
||||
previousProviderLeaseId?: string;
|
||||
reason?: "not_found" | "expired" | "identity_mismatch" | "resume_failed";
|
||||
}
|
||||
|
||||
export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWorkspaceMetadata {
|
||||
kind: "remote";
|
||||
transport: "sandbox";
|
||||
|
|
@ -171,12 +181,27 @@ export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWor
|
|||
* environment. Absent means no grant.
|
||||
*/
|
||||
readonly enableSandboxDuplexBridge?: boolean;
|
||||
/** Host-owned lifecycle override for paperclip_runner in this environment. */
|
||||
readonly runnerLifecyclePolicy?:
|
||||
| { mode: "per_turn"; idleTimeoutMs: null }
|
||||
| { mode: "warm"; idleTimeoutMs: number }
|
||||
| null;
|
||||
/** Whether this environment is configured to reuse its provider lease. */
|
||||
readonly reusableLeaseConfigured?: boolean;
|
||||
/** Host-observed provenance for this exact sandbox acquisition. */
|
||||
readonly sandboxLeaseAcquisition?: SandboxLeaseAcquisition | null;
|
||||
shellCommand?: "bash" | "sh" | null;
|
||||
environmentId?: string | null;
|
||||
leaseId?: string | null;
|
||||
remoteCwd: string;
|
||||
timeoutMs?: number | null;
|
||||
runner?: CommandManagedRuntimeRunner;
|
||||
/** Host-only provider operation. It is never serialized into the sandbox. */
|
||||
getRunnerIngressEndpoint?: (input: {
|
||||
leaseId: string;
|
||||
port: number;
|
||||
path: string;
|
||||
}) => Promise<RunnerIngressEndpoint>;
|
||||
/**
|
||||
* Sandbox-backed adapter runs stream the agent CLI's stdout/stderr
|
||||
* incrementally via a log-tail loop beside the callback bridge instead of
|
||||
|
|
@ -346,6 +371,7 @@ function parseEffectiveExecutionCapabilities(value: unknown): EffectiveExecution
|
|||
incrementalSessionOutput: parsed.incrementalSessionOutput === true,
|
||||
concurrentSyncOperations: parsed.concurrentSyncOperations === true,
|
||||
duplexCommandStream: parsed.duplexCommandStream === true,
|
||||
runnerWebSocketIngress: parsed.runnerWebSocketIngress === true,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -119,8 +119,22 @@ export {
|
|||
isPaperclipRunnerProvider,
|
||||
resolvePaperclipRunnerPermissionMode,
|
||||
} from "./paperclip-runner-permissions.js";
|
||||
export {
|
||||
PAPERCLIP_RUNNER_INGRESS_PORT,
|
||||
PAPERCLIP_RUNNER_CONNECT_PATH_PREFIX,
|
||||
PaperclipRunnerTransportError,
|
||||
buildDirectRunnerConnectUrl,
|
||||
resolvePaperclipRunnerTransport,
|
||||
} from "./runner-connectivity.js";
|
||||
export type {
|
||||
SecretHeader,
|
||||
RunnerIngressEndpoint,
|
||||
PaperclipRunnerTransport,
|
||||
} from "./runner-connectivity.js";
|
||||
export type {
|
||||
AcpxPermissionMode,
|
||||
CodexPermissionMode,
|
||||
OpenCodePermissionMode,
|
||||
PaperclipRunnerPermissionCapability,
|
||||
PaperclipRunnerPermissionMode,
|
||||
PaperclipRunnerPermissionOption,
|
||||
|
|
|
|||
|
|
@ -1,7 +1,16 @@
|
|||
export type PaperclipRunnerProvider = "codex";
|
||||
export type PaperclipRunnerProvider =
|
||||
| "codex"
|
||||
| "opencode"
|
||||
| "acpx";
|
||||
|
||||
export type CodexPermissionMode = "never" | "on-request" | "untrusted";
|
||||
export type PaperclipRunnerPermissionMode = CodexPermissionMode;
|
||||
export type OpenCodePermissionMode = "allow" | "ask" | "deny";
|
||||
export type AcpxPermissionMode = "approve-all" | "approve-reads" | "deny-all";
|
||||
|
||||
export type PaperclipRunnerPermissionMode =
|
||||
| CodexPermissionMode
|
||||
| OpenCodePermissionMode
|
||||
| AcpxPermissionMode;
|
||||
|
||||
export interface PaperclipRunnerPermissionOption<TMode extends string = string> {
|
||||
value: TMode;
|
||||
|
|
@ -11,7 +20,7 @@ export interface PaperclipRunnerPermissionOption<TMode extends string = string>
|
|||
|
||||
export interface PaperclipRunnerPermissionCapability {
|
||||
configurable: true;
|
||||
configKey: "codexPermissionMode";
|
||||
configKey: "codexPermissionMode" | "opencodePermissionMode" | "acpxPermissionMode";
|
||||
defaultMode: PaperclipRunnerPermissionMode;
|
||||
options: readonly PaperclipRunnerPermissionOption<PaperclipRunnerPermissionMode>[];
|
||||
description: string;
|
||||
|
|
@ -34,6 +43,28 @@ export const PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES = {
|
|||
{ value: "untrusted", label: "Ask for untrusted operations", description: "Prompt for operations Codex does not classify as trusted." },
|
||||
],
|
||||
},
|
||||
opencode: {
|
||||
configurable: true,
|
||||
configKey: "opencodePermissionMode",
|
||||
defaultMode: "allow",
|
||||
description: "Controls OpenCode tool permissions inside the assigned Paperclip environment.",
|
||||
options: [
|
||||
{ value: "allow", label: "Full auto (allow)", description: "Allow OpenCode operations without approval pauses." },
|
||||
{ value: "ask", label: "Ask for permission", description: "Prompt before protected OpenCode operations." },
|
||||
{ value: "deny", label: "Deny operations", description: "Reject protected OpenCode operations." },
|
||||
],
|
||||
},
|
||||
acpx: {
|
||||
configurable: true,
|
||||
configKey: "acpxPermissionMode",
|
||||
defaultMode: "approve-all",
|
||||
description: "Controls ACPX agent operations inside the assigned Paperclip environment.",
|
||||
options: [
|
||||
{ value: "approve-all", label: "Full auto (approve all)", description: "Approve ACPX operations without approval pauses." },
|
||||
{ value: "approve-reads", label: "Ask for mutations", description: "Approve reads and prompt for writes, edits, and execution." },
|
||||
{ value: "deny-all", label: "Deny all", description: "Reject harness permission requests." },
|
||||
],
|
||||
},
|
||||
} as const satisfies Record<PaperclipRunnerProvider, PaperclipRunnerPermissionCapability>;
|
||||
|
||||
export function isPaperclipRunnerProvider(value: unknown): value is PaperclipRunnerProvider {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,182 @@
|
|||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { AdapterExecutionTarget } from "./execution-target.js";
|
||||
import {
|
||||
buildDirectRunnerConnectUrl,
|
||||
resolvePaperclipRunnerTransport,
|
||||
type RunnerIngressEndpoint,
|
||||
} from "./runner-connectivity.js";
|
||||
|
||||
const capabilities = {
|
||||
reusableLeases: false,
|
||||
nativeSyncIn: false,
|
||||
nativeSyncOut: false,
|
||||
persistentProcessSessions: true,
|
||||
independentControlCommands: true,
|
||||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: false,
|
||||
duplexCommandStream: false,
|
||||
runnerWebSocketIngress: true,
|
||||
} as const;
|
||||
|
||||
function ingress(): RunnerIngressEndpoint {
|
||||
const endpoint: RunnerIngressEndpoint = {
|
||||
kind: "authenticated_websocket",
|
||||
websocketUrl:
|
||||
"wss://43127-sandbox.proxy.daytona.test/api/runner/v1/connect/00000000-0000-4000-8000-000000000001",
|
||||
secretHeaders: [{ name: "X-Daytona-Preview-Token", value: "secret" }],
|
||||
generation: "generation-1",
|
||||
refresh: async () => endpoint,
|
||||
close: async () => undefined,
|
||||
};
|
||||
return endpoint;
|
||||
}
|
||||
|
||||
describe("paperclip runner transport routing", () => {
|
||||
it("keeps same-host runnerd on plaintext loopback", async () => {
|
||||
const result = await resolvePaperclipRunnerTransport({
|
||||
target: { kind: "local" },
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
localConnectUrl:
|
||||
"ws://127.0.0.1:3100/api/runner/v1/connect/00000000-0000-4000-8000-000000000001",
|
||||
enableRunnerPreviewIngress: false,
|
||||
});
|
||||
expect(result.mode).toBe("local_loopback");
|
||||
});
|
||||
|
||||
it("selects provider ingress for Daytona-style capability even when a public URL exists", async () => {
|
||||
const getRunnerIngressEndpoint = vi.fn(async () => ingress());
|
||||
const target: AdapterExecutionTarget = {
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
providerKey: "daytona",
|
||||
remoteCwd: "/workspace",
|
||||
leaseId: "lease-1",
|
||||
effectiveCapabilities: capabilities,
|
||||
getRunnerIngressEndpoint,
|
||||
};
|
||||
const result = await resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: "wss://paperclip.example.test",
|
||||
enableRunnerPreviewIngress: true,
|
||||
});
|
||||
expect(result.mode).toBe("provider_ingress");
|
||||
expect(getRunnerIngressEndpoint).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("does not request preview ingress while the new-runner rollout flag is off", async () => {
|
||||
const getRunnerIngressEndpoint = vi.fn(async () => ingress());
|
||||
const target: AdapterExecutionTarget = {
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
providerKey: "daytona",
|
||||
remoteCwd: "/workspace",
|
||||
leaseId: "lease-1",
|
||||
effectiveCapabilities: capabilities,
|
||||
getRunnerIngressEndpoint,
|
||||
};
|
||||
await expect(
|
||||
resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: "wss://paperclip.example.test",
|
||||
enableRunnerPreviewIngress: false,
|
||||
}),
|
||||
).rejects.toMatchObject({ code: "runner_ingress_unavailable" });
|
||||
expect(getRunnerIngressEndpoint).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("selects direct WSS only for a remote target with an explicit URL", async () => {
|
||||
const target: AdapterExecutionTarget = {
|
||||
kind: "remote",
|
||||
transport: "ssh",
|
||||
remoteCwd: "/workspace",
|
||||
spec: {
|
||||
host: "runner.internal",
|
||||
port: 22,
|
||||
username: "runner",
|
||||
remoteWorkspacePath: "/workspace",
|
||||
remoteCwd: "/workspace",
|
||||
privateKey: null,
|
||||
knownHosts: null,
|
||||
strictHostKeyChecking: true,
|
||||
},
|
||||
};
|
||||
const result = await resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: "wss://paperclip.example.test/runner-base/",
|
||||
runnerCaBundlePath: "/etc/paperclip/runner-ca.pem",
|
||||
enableRunnerPreviewIngress: false,
|
||||
});
|
||||
expect(result).toEqual({
|
||||
mode: "direct_outbound",
|
||||
connectUrl:
|
||||
"wss://paperclip.example.test/runner-base/api/runner/v1/connect/00000000-0000-4000-8000-000000000001",
|
||||
caBundlePath: "/etc/paperclip/runner-ca.pem",
|
||||
});
|
||||
});
|
||||
|
||||
it("fails the selected ingress mode without falling through to direct WSS", async () => {
|
||||
const target: AdapterExecutionTarget = {
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
providerKey: "daytona",
|
||||
remoteCwd: "/workspace",
|
||||
leaseId: "lease-1",
|
||||
effectiveCapabilities: capabilities,
|
||||
getRunnerIngressEndpoint: async () => {
|
||||
throw new Error("preview unavailable");
|
||||
},
|
||||
};
|
||||
await expect(
|
||||
resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: "wss://paperclip.example.test",
|
||||
enableRunnerPreviewIngress: true,
|
||||
}),
|
||||
).rejects.toThrow("preview unavailable");
|
||||
});
|
||||
|
||||
it("never routes Daytona through direct outbound when ingress capability is unavailable", async () => {
|
||||
const target: AdapterExecutionTarget = {
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
providerKey: "daytona",
|
||||
remoteCwd: "/workspace",
|
||||
leaseId: "lease-1",
|
||||
effectiveCapabilities: {
|
||||
...capabilities,
|
||||
runnerWebSocketIngress: false,
|
||||
},
|
||||
};
|
||||
await expect(
|
||||
resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: "wss://paperclip.example.test",
|
||||
enableRunnerPreviewIngress: true,
|
||||
}),
|
||||
).rejects.toMatchObject({ code: "runner_ingress_unavailable" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
"ws://paperclip.example.test",
|
||||
"wss://user@paperclip.example.test",
|
||||
"wss://paperclip.example.test?token=secret",
|
||||
"wss://paperclip.example.test#fragment",
|
||||
])("rejects unsafe direct runner URL %s", (runnerPublicUrl) => {
|
||||
expect(() =>
|
||||
buildDirectRunnerConnectUrl({
|
||||
runnerPublicUrl,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
}),
|
||||
).toThrow();
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
import type { AdapterExecutionTarget } from "./execution-target.js";
|
||||
|
||||
export const PAPERCLIP_RUNNER_INGRESS_PORT = 43_127;
|
||||
export const PAPERCLIP_RUNNER_CONNECT_PATH_PREFIX = "/api/runner/v1/connect";
|
||||
|
||||
export interface SecretHeader {
|
||||
readonly name: string;
|
||||
readonly value: string;
|
||||
}
|
||||
|
||||
export interface RunnerIngressEndpoint {
|
||||
readonly kind: "authenticated_websocket";
|
||||
readonly websocketUrl: string;
|
||||
readonly secretHeaders: readonly SecretHeader[];
|
||||
readonly generation: string;
|
||||
refresh(): Promise<RunnerIngressEndpoint>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export type PaperclipRunnerTransport =
|
||||
| {
|
||||
readonly mode: "local_loopback";
|
||||
readonly connectUrl: string;
|
||||
}
|
||||
| {
|
||||
readonly mode: "direct_outbound";
|
||||
readonly connectUrl: string;
|
||||
readonly caBundlePath?: string;
|
||||
}
|
||||
| {
|
||||
readonly mode: "provider_ingress";
|
||||
readonly listenAddress: "0.0.0.0";
|
||||
readonly listenPort: number;
|
||||
readonly listenPath: string;
|
||||
readonly ingress: RunnerIngressEndpoint;
|
||||
};
|
||||
|
||||
export class PaperclipRunnerTransportError extends Error {
|
||||
readonly code:
|
||||
| "runner_transport_ineligible"
|
||||
| "runner_direct_wss_failed"
|
||||
| "runner_ingress_unavailable";
|
||||
|
||||
constructor(
|
||||
code: PaperclipRunnerTransportError["code"],
|
||||
message: string,
|
||||
options?: ErrorOptions,
|
||||
) {
|
||||
super(`${code}: ${message}`, options);
|
||||
this.name = "PaperclipRunnerTransportError";
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
function connectPath(runId: string): string {
|
||||
if (!runId || runId.includes("/") || runId.includes("?") || runId.includes("#")) {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_transport_ineligible",
|
||||
"Runner run id is not safe for a WebSocket route.",
|
||||
);
|
||||
}
|
||||
return `${PAPERCLIP_RUNNER_CONNECT_PATH_PREFIX}/${encodeURIComponent(runId)}`;
|
||||
}
|
||||
|
||||
export function buildDirectRunnerConnectUrl(input: {
|
||||
runnerPublicUrl: string;
|
||||
runId: string;
|
||||
}): string {
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(input.runnerPublicUrl);
|
||||
} catch (error) {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_direct_wss_failed",
|
||||
"The configured runner public URL is invalid.",
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
if (url.protocol !== "wss:") {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_direct_wss_failed",
|
||||
"Remote runner connectivity requires a wss: runner public URL.",
|
||||
);
|
||||
}
|
||||
if (url.username || url.password || url.search || url.hash) {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_direct_wss_failed",
|
||||
"Runner public URLs cannot contain userinfo, a query string, or a fragment.",
|
||||
);
|
||||
}
|
||||
url.pathname = `${url.pathname.replace(/\/$/, "")}${connectPath(input.runId)}`;
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
export async function resolvePaperclipRunnerTransport(input: {
|
||||
target: AdapterExecutionTarget;
|
||||
runId: string;
|
||||
localConnectUrl: string;
|
||||
runnerPublicUrl?: string | null;
|
||||
runnerCaBundlePath?: string | null;
|
||||
enableRunnerPreviewIngress: boolean;
|
||||
getRunnerIngressEndpoint?: (input: {
|
||||
leaseId: string;
|
||||
port: number;
|
||||
path: string;
|
||||
}) => Promise<RunnerIngressEndpoint>;
|
||||
}): Promise<PaperclipRunnerTransport> {
|
||||
if (input.target.kind === "local") {
|
||||
return { mode: "local_loopback", connectUrl: input.localConnectUrl };
|
||||
}
|
||||
|
||||
if (
|
||||
input.target.transport === "sandbox" &&
|
||||
input.target.providerKey === "daytona" &&
|
||||
input.target.effectiveCapabilities?.runnerWebSocketIngress !== true
|
||||
) {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_ingress_unavailable",
|
||||
"Daytona runner execution requires provider WebSocket ingress capability.",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
input.target.transport === "sandbox" &&
|
||||
input.target.effectiveCapabilities?.runnerWebSocketIngress === true
|
||||
) {
|
||||
if (!input.enableRunnerPreviewIngress) {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_ingress_unavailable",
|
||||
"Runner preview ingress is disabled for this Paperclip instance.",
|
||||
);
|
||||
}
|
||||
const getRunnerIngressEndpoint =
|
||||
input.getRunnerIngressEndpoint ?? input.target.getRunnerIngressEndpoint;
|
||||
if (!input.target.leaseId || !getRunnerIngressEndpoint) {
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_ingress_unavailable",
|
||||
"The sandbox runner ingress provider is unavailable for this lease.",
|
||||
);
|
||||
}
|
||||
const path = connectPath(input.runId);
|
||||
const ingress = await getRunnerIngressEndpoint({
|
||||
leaseId: input.target.leaseId,
|
||||
port: PAPERCLIP_RUNNER_INGRESS_PORT,
|
||||
path,
|
||||
});
|
||||
return {
|
||||
mode: "provider_ingress",
|
||||
listenAddress: "0.0.0.0",
|
||||
listenPort: PAPERCLIP_RUNNER_INGRESS_PORT,
|
||||
listenPath: path,
|
||||
ingress,
|
||||
};
|
||||
}
|
||||
|
||||
if (input.runnerPublicUrl) {
|
||||
return {
|
||||
mode: "direct_outbound",
|
||||
connectUrl: buildDirectRunnerConnectUrl({
|
||||
runnerPublicUrl: input.runnerPublicUrl,
|
||||
runId: input.runId,
|
||||
}),
|
||||
...(input.runnerCaBundlePath
|
||||
? { caBundlePath: input.runnerCaBundlePath }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
throw new PaperclipRunnerTransportError(
|
||||
"runner_transport_ineligible",
|
||||
"The remote execution target has neither runner WebSocket ingress nor an explicit runner public URL.",
|
||||
);
|
||||
}
|
||||
|
|
@ -741,34 +741,35 @@ export interface DurablePrpControlPlaneStore {
|
|||
readonly state: StoredCoreState;
|
||||
}
|
||||
|
||||
class PrpWebSocketConnection {
|
||||
class RawWebSocketWireConnection implements PrpWireConnection {
|
||||
readonly socket: Duplex;
|
||||
pendingChallenge: PendingChallenge | null = null;
|
||||
secureChannel: SecureChannel | null = null;
|
||||
lease: ConnectionLeaseRecord | null = null;
|
||||
connectionId: string | null = null;
|
||||
#buffer = Buffer.alloc(0);
|
||||
#closed = false;
|
||||
#onText: (text: string) => void | Promise<void>;
|
||||
#onClose: () => void;
|
||||
#processing = Promise.resolve();
|
||||
#onJson: (value: unknown) => void = () => undefined;
|
||||
#onClose: (reason: TransportCloseReason) => void = () => undefined;
|
||||
|
||||
constructor(
|
||||
socket: Duplex,
|
||||
onText: (text: string) => void | Promise<void>,
|
||||
onClose: () => void,
|
||||
) {
|
||||
constructor(socket: Duplex) {
|
||||
this.socket = socket;
|
||||
this.#onText = onText;
|
||||
this.#onClose = onClose;
|
||||
socket.on("data", (chunk: Buffer) => this.#consume(chunk));
|
||||
socket.on("close", () => {
|
||||
if (!this.#closed) {
|
||||
this.#closed = true;
|
||||
this.#onClose();
|
||||
this.#onClose({ message: "socket_closed" });
|
||||
}
|
||||
});
|
||||
socket.on("error", () => this.close());
|
||||
socket.on("error", (error) => {
|
||||
if (this.#closed) return;
|
||||
this.#closed = true;
|
||||
this.#onClose({ message: "socket_error", error });
|
||||
});
|
||||
}
|
||||
|
||||
onJson(listener: (value: unknown) => void): void {
|
||||
this.#onJson = listener;
|
||||
}
|
||||
|
||||
onClose(listener: (reason: TransportCloseReason) => void): void {
|
||||
this.#onClose = listener;
|
||||
}
|
||||
|
||||
acceptInitialData(data: Buffer<ArrayBufferLike>): void {
|
||||
|
|
@ -776,11 +777,7 @@ class PrpWebSocketConnection {
|
|||
}
|
||||
|
||||
sendJson(value: unknown): void {
|
||||
const wire =
|
||||
this.secureChannel === null
|
||||
? value
|
||||
: encryptSecureJson(this.secureChannel, value);
|
||||
this.sendText(JSON.stringify(wire));
|
||||
this.sendText(JSON.stringify(value));
|
||||
}
|
||||
|
||||
sendText(text: string): void {
|
||||
|
|
@ -803,13 +800,13 @@ class PrpWebSocketConnection {
|
|||
this.socket.write(Buffer.concat([Buffer.from(header), payload]));
|
||||
}
|
||||
|
||||
close(): void {
|
||||
close(_code?: number): void {
|
||||
if (this.#closed) {
|
||||
return;
|
||||
}
|
||||
this.#closed = true;
|
||||
this.socket.destroy();
|
||||
this.#onClose();
|
||||
this.#onClose({ message: "local_close" });
|
||||
}
|
||||
|
||||
#consume(chunk: Buffer): void {
|
||||
|
|
@ -850,10 +847,14 @@ class PrpWebSocketConnection {
|
|||
payload[index] = payload[index]! ^ mask[index % 4]!;
|
||||
}
|
||||
if (opcode === 0x1) {
|
||||
const text = payload.toString("utf8");
|
||||
this.#processing = this.#processing
|
||||
.then(() => this.#onText(text))
|
||||
.catch(() => this.close());
|
||||
try {
|
||||
this.#onJson(JSON.parse(payload.toString("utf8")) as unknown);
|
||||
} catch (error) {
|
||||
this.#closed = true;
|
||||
this.socket.destroy();
|
||||
this.#onClose({ message: "invalid_json", error });
|
||||
return;
|
||||
}
|
||||
} else if (opcode === 0x8) {
|
||||
this.close();
|
||||
return;
|
||||
|
|
@ -874,6 +875,49 @@ class PrpWebSocketConnection {
|
|||
}
|
||||
}
|
||||
|
||||
class AuthorityConnection {
|
||||
pendingChallenge: PendingChallenge | null = null;
|
||||
secureChannel: SecureChannel | null = null;
|
||||
lease: ConnectionLeaseRecord | null = null;
|
||||
connectionId: string | null = null;
|
||||
readonly wire: PrpWireConnection;
|
||||
#closed = false;
|
||||
#onClose: () => void;
|
||||
|
||||
constructor(input: {
|
||||
wire: PrpWireConnection;
|
||||
onJson: (value: unknown) => void;
|
||||
onClose: () => void;
|
||||
}) {
|
||||
this.wire = input.wire;
|
||||
this.#onClose = input.onClose;
|
||||
this.wire.onJson(input.onJson);
|
||||
this.wire.onClose(() => this.#markClosed());
|
||||
}
|
||||
|
||||
sendJson(value: unknown): void {
|
||||
if (this.#closed) return;
|
||||
this.wire.sendJson(
|
||||
this.secureChannel === null
|
||||
? value
|
||||
: encryptSecureJson(this.secureChannel, value),
|
||||
);
|
||||
}
|
||||
|
||||
close(code?: number): void {
|
||||
if (this.#closed) return;
|
||||
this.#closed = true;
|
||||
this.wire.close(code);
|
||||
this.#onClose();
|
||||
}
|
||||
|
||||
#markClosed(): void {
|
||||
if (this.#closed) return;
|
||||
this.#closed = true;
|
||||
this.#onClose();
|
||||
}
|
||||
}
|
||||
|
||||
/** Authenticated, replay-safe PRP transport authority. Business operations are caller supplied. */
|
||||
export class DurablePrpControlPlane {
|
||||
readonly #identity: DurableRecoveryIdentity;
|
||||
|
|
@ -881,7 +925,7 @@ export class DurablePrpControlPlane {
|
|||
#expectedRunnerVersion: string;
|
||||
#expectedRunnerDigest: string;
|
||||
#server: Server | null = null;
|
||||
#connections = new Set<PrpWebSocketConnection>();
|
||||
#connections = new Set<AuthorityConnection>();
|
||||
#pendingSemanticCalls = new Set<string>();
|
||||
#port: number | null = null;
|
||||
#onSemanticToolInput?: DurablePrpControlPlaneOptions["onSemanticToolInput"];
|
||||
|
|
@ -1110,23 +1154,36 @@ export class DurablePrpControlPlane {
|
|||
"\r\n",
|
||||
].join("\r\n"),
|
||||
);
|
||||
let connection!: PrpWebSocketConnection;
|
||||
connection = new PrpWebSocketConnection(
|
||||
socket,
|
||||
(text): Promise<void> => this.#handleText(connection, text),
|
||||
() => this.#connections.delete(connection),
|
||||
);
|
||||
this.#connections.add(connection);
|
||||
connection.acceptInitialData(head);
|
||||
const wire = new RawWebSocketWireConnection(socket);
|
||||
this.attachWireConnection(wire);
|
||||
wire.acceptInitialData(head);
|
||||
}
|
||||
|
||||
async #handleText(
|
||||
connection: PrpWebSocketConnection,
|
||||
text: string,
|
||||
/** Attach either an accepted inbound WebSocket or a Paperclip-opened peer. */
|
||||
attachWireConnection(wire: PrpWireConnection): PrpWireAttachment {
|
||||
let connection!: AuthorityConnection;
|
||||
let processing = Promise.resolve();
|
||||
connection = new AuthorityConnection({
|
||||
wire,
|
||||
onJson: (value) => {
|
||||
processing = processing
|
||||
.then(() => this.#handleJson(connection, value))
|
||||
.catch(() => connection.close());
|
||||
},
|
||||
onClose: () => this.#connections.delete(connection),
|
||||
});
|
||||
this.#connections.add(connection);
|
||||
return {
|
||||
isAuthenticated: () => connection.secureChannel !== null,
|
||||
};
|
||||
}
|
||||
|
||||
async #handleJson(
|
||||
connection: AuthorityConnection,
|
||||
wire: unknown,
|
||||
): Promise<void> {
|
||||
let envelope: Record<string, unknown>;
|
||||
try {
|
||||
const wire = JSON.parse(text) as unknown;
|
||||
envelope =
|
||||
connection.secureChannel === null
|
||||
? (wire as Record<string, unknown>)
|
||||
|
|
@ -1306,7 +1363,7 @@ export class DurablePrpControlPlane {
|
|||
}
|
||||
|
||||
#authHello(
|
||||
connection: PrpWebSocketConnection,
|
||||
connection: AuthorityConnection,
|
||||
envelope: Record<string, unknown>,
|
||||
): void {
|
||||
if (connection.pendingChallenge !== null) {
|
||||
|
|
@ -1371,7 +1428,7 @@ export class DurablePrpControlPlane {
|
|||
}
|
||||
|
||||
#authResponse(
|
||||
connection: PrpWebSocketConnection,
|
||||
connection: AuthorityConnection,
|
||||
envelope: Record<string, unknown>,
|
||||
): void {
|
||||
const pending = connection.pendingChallenge;
|
||||
|
|
@ -1451,7 +1508,7 @@ export class DurablePrpControlPlane {
|
|||
}
|
||||
|
||||
#welcome(
|
||||
connection: PrpWebSocketConnection,
|
||||
connection: AuthorityConnection,
|
||||
leaseToken: string | null,
|
||||
): void {
|
||||
const lease = connection.lease;
|
||||
|
|
@ -1522,7 +1579,7 @@ export class DurablePrpControlPlane {
|
|||
}
|
||||
|
||||
#controlEnvelope(
|
||||
connection: PrpWebSocketConnection,
|
||||
connection: AuthorityConnection,
|
||||
envelopeId: string,
|
||||
kind: string,
|
||||
payload: Record<string, unknown>,
|
||||
|
|
@ -1550,7 +1607,7 @@ export class DurablePrpControlPlane {
|
|||
};
|
||||
}
|
||||
|
||||
#sendNextCommand(connection: PrpWebSocketConnection): void {
|
||||
#sendNextCommand(connection: AuthorityConnection): void {
|
||||
const [command] = this.#nextPendingCommand();
|
||||
if (command === undefined) return;
|
||||
this.#store.state.commandDeliveryCounts[command.commandId] =
|
||||
|
|
@ -1567,7 +1624,7 @@ export class DurablePrpControlPlane {
|
|||
}
|
||||
|
||||
#commandResult(
|
||||
connection: PrpWebSocketConnection,
|
||||
connection: AuthorityConnection,
|
||||
envelope: Record<string, unknown>,
|
||||
): void {
|
||||
const result = envelope.payload as Record<string, unknown> | undefined;
|
||||
|
|
@ -1609,7 +1666,7 @@ export class DurablePrpControlPlane {
|
|||
}
|
||||
|
||||
async #event(
|
||||
connection: PrpWebSocketConnection,
|
||||
connection: AuthorityConnection,
|
||||
envelope: Record<string, unknown>,
|
||||
): Promise<void> {
|
||||
const validated = validatePrpEvent(envelope.payload);
|
||||
|
|
|
|||
|
|
@ -11,7 +11,9 @@ const PLUGIN_ID = "paperclip.daytona-sandbox-provider";
|
|||
// neutral `supportsLoginPty`.
|
||||
// 0.1.4 adds the `concurrentSyncOperations` sandbox capability to the driver.
|
||||
// 0.1.5 adds the `duplexCommandStream` sandbox capability to the driver.
|
||||
const PLUGIN_VERSION = "0.1.5";
|
||||
// 0.1.6 adds private authenticated WebSocket ingress for paperclip_runner.
|
||||
// 0.1.7 exposes host-owned warm/cold runner lifecycle controls.
|
||||
const PLUGIN_VERSION = "0.1.7";
|
||||
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
id: PLUGIN_ID,
|
||||
|
|
@ -58,6 +60,7 @@ const manifest: PaperclipPluginManifestV1 = {
|
|||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: true,
|
||||
duplexCommandStream: true,
|
||||
runnerWebSocketIngress: true,
|
||||
},
|
||||
supportsInteractiveSetup: true,
|
||||
interactiveSetupConnectionTypes: ["ssh"],
|
||||
|
|
@ -169,6 +172,21 @@ const manifest: PaperclipPluginManifestV1 = {
|
|||
"Whether to stop and later resume the sandbox across runs instead of deleting it on release.",
|
||||
default: false,
|
||||
},
|
||||
runnerLifecycleMode: {
|
||||
type: "string",
|
||||
enum: ["inherit", "per_turn", "warm"],
|
||||
description:
|
||||
"paperclip_runner lifecycle for this environment. Inherit uses the agent setting; warm keeps runnerd and the sandbox available between turns.",
|
||||
default: "inherit",
|
||||
},
|
||||
runnerIdleTimeoutMs: {
|
||||
type: "integer",
|
||||
description:
|
||||
"How long an idle warm paperclip_runner stays alive before it checkpoints and suspends.",
|
||||
minimum: 1000,
|
||||
maximum: 86400000,
|
||||
default: 300000,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@ function createMockSandbox(overrides: {
|
|||
recoverable?: boolean;
|
||||
workDir?: string;
|
||||
autoDestroyAt?: string | null;
|
||||
updatedAt?: string;
|
||||
} = {}) {
|
||||
return {
|
||||
id: overrides.id ?? "sandbox-123",
|
||||
|
|
@ -57,6 +58,7 @@ function createMockSandbox(overrides: {
|
|||
// A configured provider TTL populates `autoDestroyAt` after `setTtl` +
|
||||
// `refreshData`. The default mock leaves it unset (no TTL configured).
|
||||
autoDestroyAt: overrides.autoDestroyAt ?? undefined,
|
||||
updatedAt: overrides.updatedAt,
|
||||
getWorkDir: vi.fn().mockResolvedValue(overrides.workDir ?? "/home/daytona"),
|
||||
getUserHomeDir: vi.fn().mockResolvedValue("/home/daytona"),
|
||||
start: vi.fn().mockResolvedValue(undefined),
|
||||
|
|
@ -75,6 +77,10 @@ function createMockSandbox(overrides: {
|
|||
token: "ssh-token-secret",
|
||||
command: "ssh ssh-token-secret@ssh.app.daytona.io",
|
||||
}),
|
||||
getPreviewLink: vi.fn().mockResolvedValue({
|
||||
url: "https://43127-sandbox-123.proxy.daytona.test",
|
||||
token: "preview-token-secret",
|
||||
}),
|
||||
_experimental_createSnapshot: vi.fn().mockResolvedValue(undefined),
|
||||
fs: {
|
||||
createFolder: vi.fn().mockResolvedValue(undefined),
|
||||
|
|
@ -161,11 +167,90 @@ describe("Daytona sandbox provider plugin", () => {
|
|||
expect(plugin.definition.onDuplexChannelClose).toBeTypeOf("function");
|
||||
});
|
||||
|
||||
it("declares and returns private authenticated runner WebSocket ingress", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox();
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
const base = {
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { image: "node:20", timeoutMs: 300_000, reuseLease: false },
|
||||
};
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
...base,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
});
|
||||
const endpoint = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.({
|
||||
...base,
|
||||
lease: lease!,
|
||||
port: 43_127,
|
||||
path: "/api/runner/v1/connect/00000000-0000-4000-8000-000000000001",
|
||||
});
|
||||
|
||||
expect(manifest.environmentDrivers?.[0]?.sandboxCapabilities).toMatchObject({
|
||||
runnerWebSocketIngress: true,
|
||||
});
|
||||
expect(endpoint).toMatchObject({
|
||||
kind: "authenticated_websocket",
|
||||
websocketUrl:
|
||||
"wss://43127-sandbox-123.proxy.daytona.test/api/runner/v1/connect/00000000-0000-4000-8000-000000000001",
|
||||
secretHeaders: [
|
||||
{ name: "X-Daytona-Preview-Token", value: "preview-token-secret" },
|
||||
],
|
||||
});
|
||||
expect(endpoint?.websocketUrl).not.toContain("preview-token-secret");
|
||||
expect(endpoint?.websocketUrl).not.toContain("host-key");
|
||||
});
|
||||
|
||||
it("keeps ingress generation independent of token rotation and changes it after a sandbox lifecycle revision", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ updatedAt: "2026-08-25T10:00:00.000Z" });
|
||||
sandbox.getPreviewLink
|
||||
.mockResolvedValueOnce({
|
||||
url: "https://43127-sandbox-123.proxy.daytona.test",
|
||||
token: "preview-token-1",
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
url: "https://43127-sandbox-123.proxy.daytona.test",
|
||||
token: "preview-token-2",
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
url: "https://43127-sandbox-123.proxy.daytona.test",
|
||||
token: "preview-token-3",
|
||||
});
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
const base = {
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { image: "node:20", timeoutMs: 300_000, reuseLease: false },
|
||||
};
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
...base,
|
||||
runId: "00000000-0000-4000-8000-000000000001",
|
||||
});
|
||||
const request = {
|
||||
...base,
|
||||
lease: lease!,
|
||||
port: 43_127,
|
||||
path: "/api/runner/v1/connect/00000000-0000-4000-8000-000000000001",
|
||||
};
|
||||
const first = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.(request);
|
||||
const second = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.(request);
|
||||
expect(second?.generation).toBe(first?.generation);
|
||||
expect(second?.secretHeaders).not.toEqual(first?.secretHeaders);
|
||||
|
||||
sandbox.updatedAt = "2026-08-25T10:05:00.000Z";
|
||||
const restarted = await plugin.definition.onEnvironmentRunnerIngressEndpoint?.(request);
|
||||
expect(restarted?.generation).not.toBe(first?.generation);
|
||||
});
|
||||
|
||||
it("bumps the plugin version so the server reconciles the stored manifest", () => {
|
||||
// The bundled-plugin boot reconcile refreshes the stored manifest for an
|
||||
// existing install only when the version changes. The duplex capability needs
|
||||
// the bump to reach an existing install.
|
||||
expect(manifest.version).toBe("0.1.5");
|
||||
expect(manifest.version).toBe("0.1.7");
|
||||
});
|
||||
|
||||
it("opens a duplex channel, forwards a host write, and closes it on lease release", async () => {
|
||||
|
|
@ -1074,7 +1159,7 @@ describe("Daytona sandbox provider plugin", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("deletes the sandbox if resume setup throws after the sandbox starts", async () => {
|
||||
it("preserves the sandbox if resume setup throws after the sandbox starts", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ id: "sandbox-resume", state: "stopped" });
|
||||
sandbox.getWorkDir.mockRejectedValue(new Error("workdir lookup failed"));
|
||||
|
|
@ -1094,7 +1179,7 @@ describe("Daytona sandbox provider plugin", () => {
|
|||
).rejects.toThrow("workdir lookup failed");
|
||||
|
||||
expect(sandbox.start).toHaveBeenCalled();
|
||||
expect(sandbox.delete).toHaveBeenCalledTimes(1);
|
||||
expect(sandbox.delete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("marks missing reusable leases as expired on resume", async () => {
|
||||
|
|
|
|||
|
|
@ -23,6 +23,8 @@ import type {
|
|||
PluginEnvironmentDestroyLeaseParams,
|
||||
PluginEnvironmentExecuteParams,
|
||||
PluginEnvironmentExecuteResult,
|
||||
PluginEnvironmentRunnerIngressEndpointParams,
|
||||
PluginEnvironmentRunnerIngressEndpoint,
|
||||
PluginEnvironmentGetInteractiveSetupParams,
|
||||
PluginEnvironmentInteractiveSetupSession,
|
||||
PluginEnvironmentLease,
|
||||
|
|
@ -1304,6 +1306,34 @@ const sandboxHandleCache = (() => {
|
|||
return { get, seed, clear, reset, markFresh, findByProviderLeaseId };
|
||||
})();
|
||||
|
||||
// Preview credentials can rotate without a sandbox restart, so they must not
|
||||
// define endpoint generation. Daytona's lifecycle revision does: refreshData
|
||||
// updates `updatedAt` after stop/start. When Daytona does not expose a
|
||||
// lifecycle revision, the in-memory generation remains stable for the worker.
|
||||
const runnerIngressGenerationStore = (() => {
|
||||
const entries = new Map<
|
||||
string,
|
||||
{ revision: string | null; generation: string }
|
||||
>();
|
||||
|
||||
function get(sandbox: Sandbox): string {
|
||||
const revision = sandbox.updatedAt ?? sandbox.createdAt ?? null;
|
||||
const current = entries.get(sandbox.id);
|
||||
if (current && current.revision === revision) return current.generation;
|
||||
const generation = createHash("sha256")
|
||||
.update(`${sandbox.id}\0${revision ?? randomUUID()}`)
|
||||
.digest("hex");
|
||||
entries.set(sandbox.id, { revision, generation });
|
||||
return generation;
|
||||
}
|
||||
|
||||
function reset(): void {
|
||||
entries.clear();
|
||||
}
|
||||
|
||||
return { get, reset };
|
||||
})();
|
||||
|
||||
// Advisory writable-set store. It holds, per lease scope, the sandbox
|
||||
// directories that a sync operation declared read-write (`access: "rw"`). The
|
||||
// store is advisory and best-effort in-memory state: it adds no security (the
|
||||
|
|
@ -1422,6 +1452,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void {
|
|||
sandboxHandleLeaseAdmissionStates.reset();
|
||||
sandboxHandleWritableDirs.reset();
|
||||
sandboxHandleSessionStore.reset();
|
||||
runnerIngressGenerationStore.reset();
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -2204,7 +2235,10 @@ const plugin = definePlugin({
|
|||
};
|
||||
} catch (error) {
|
||||
evictSandboxHandle(scope);
|
||||
await sandbox.delete(toTimeoutSeconds(config.timeoutMs)).catch(() => undefined);
|
||||
// A timeout, rate limit, or provider 5xx does not prove this sandbox is
|
||||
// lost. Preserve the exact resource and let the host retry its recorded
|
||||
// lease; replacement is permitted only after an explicit not-found or
|
||||
// an immutable workspace identity mismatch.
|
||||
throw error;
|
||||
}
|
||||
}, { allowClosed: true });
|
||||
|
|
@ -2632,7 +2666,6 @@ const plugin = definePlugin({
|
|||
},
|
||||
});
|
||||
const getDurationMs = timingNow() - getStart;
|
||||
await ensureSandboxStarted(sandbox, toTimeoutSeconds(resolveTimeoutMs(params.timeoutMs, config)));
|
||||
const scope: SandboxScope = {
|
||||
driverKey: params.driverKey,
|
||||
companyId: params.companyId,
|
||||
|
|
@ -2640,6 +2673,13 @@ const plugin = definePlugin({
|
|||
providerLeaseId,
|
||||
config,
|
||||
};
|
||||
if (sandbox.state !== "started") {
|
||||
// A provider restart destroys Daytona process sessions. Drop the stale
|
||||
// session id before starting the sandbox so runnerd recovery opens a
|
||||
// new session instead of retrying a dead one for its whole grace.
|
||||
sandboxHandleSessionStore.clear(scope);
|
||||
}
|
||||
await ensureSandboxStarted(sandbox, toTimeoutSeconds(resolveTimeoutMs(params.timeoutMs, config)));
|
||||
// Dispatch the command. A normal command runs in the persistent session:
|
||||
// the provider opens the one session on a cache miss and runs every command
|
||||
// in it. The provider never falls back to a one-shot command to open a
|
||||
|
|
@ -2668,6 +2708,70 @@ const plugin = definePlugin({
|
|||
});
|
||||
},
|
||||
|
||||
async onEnvironmentRunnerIngressEndpoint(
|
||||
params: PluginEnvironmentRunnerIngressEndpointParams,
|
||||
): Promise<PluginEnvironmentRunnerIngressEndpoint> {
|
||||
if (params.port !== 43_127) {
|
||||
throw new Error("Daytona runner ingress must use fixed port 43127.");
|
||||
}
|
||||
if (!/^\/api\/runner\/v1\/connect\/[^/?#]+$/.test(params.path)) {
|
||||
throw new Error("Daytona runner ingress path is invalid.");
|
||||
}
|
||||
const providerLeaseId = params.lease.providerLeaseId;
|
||||
if (!providerLeaseId) {
|
||||
throw new Error("Daytona runner ingress requires a provider lease id.");
|
||||
}
|
||||
const config = parseDriverConfig(params.config);
|
||||
return await withSandboxActivityGate(
|
||||
{
|
||||
driverKey: params.driverKey,
|
||||
companyId: params.companyId,
|
||||
environmentId: params.environmentId,
|
||||
providerLeaseId,
|
||||
config,
|
||||
},
|
||||
async () => {
|
||||
const sandbox = await getSandbox({
|
||||
driverKey: params.driverKey,
|
||||
companyId: params.companyId,
|
||||
environmentId: params.environmentId,
|
||||
providerLeaseId,
|
||||
config,
|
||||
});
|
||||
await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs));
|
||||
await withLivenessTimeout(
|
||||
"sandbox.refreshData",
|
||||
config.livenessTimeoutMs,
|
||||
() => sandbox.refreshData(),
|
||||
);
|
||||
const preview = await sandbox.getPreviewLink(params.port);
|
||||
if (typeof preview.url !== "string" || typeof preview.token !== "string") {
|
||||
throw new Error("Daytona returned an incomplete private preview endpoint.");
|
||||
}
|
||||
const url = new URL(preview.url);
|
||||
if (
|
||||
url.protocol !== "https:" ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
url.search ||
|
||||
url.hash
|
||||
) {
|
||||
throw new Error("Daytona returned an invalid private preview URL.");
|
||||
}
|
||||
url.protocol = "wss:";
|
||||
url.pathname = `${url.pathname.replace(/\/$/, "")}${params.path}`;
|
||||
return {
|
||||
kind: "authenticated_websocket",
|
||||
websocketUrl: url.toString(),
|
||||
secretHeaders: [
|
||||
{ name: "X-Daytona-Preview-Token", value: preview.token },
|
||||
],
|
||||
generation: runnerIngressGenerationStore.get(sandbox),
|
||||
};
|
||||
},
|
||||
);
|
||||
},
|
||||
|
||||
// Opt-in native inbound transfer. Defining this hook (with onEnvironmentSyncOut)
|
||||
// makes the worker advertise `environmentSyncIn`/`environmentSyncOut`, so the
|
||||
// host runner routes Daytona workspace/asset transfers through the SDK's batch
|
||||
|
|
@ -2915,6 +3019,7 @@ const plugin = definePlugin({
|
|||
await entry.session.close().catch(() => undefined);
|
||||
}
|
||||
sandboxHandleCache.reset();
|
||||
runnerIngressGenerationStore.reset();
|
||||
},
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -55,6 +55,8 @@ import type {
|
|||
PluginEnvironmentDestroyLeaseParams,
|
||||
PluginEnvironmentExecuteParams,
|
||||
PluginEnvironmentExecuteResult,
|
||||
PluginEnvironmentRunnerIngressEndpointParams,
|
||||
PluginEnvironmentRunnerIngressEndpoint,
|
||||
PluginEnvironmentSyncInParams,
|
||||
PluginEnvironmentSyncOutParams,
|
||||
PluginEnvironmentSyncResult,
|
||||
|
|
@ -399,6 +401,11 @@ export interface PluginDefinition {
|
|||
params: PluginEnvironmentExecuteParams,
|
||||
): Promise<PluginEnvironmentExecuteResult>;
|
||||
|
||||
/** Return an authenticated private WebSocket ingress for runnerd. */
|
||||
onEnvironmentRunnerIngressEndpoint?(
|
||||
params: PluginEnvironmentRunnerIngressEndpointParams,
|
||||
): Promise<PluginEnvironmentRunnerIngressEndpoint>;
|
||||
|
||||
/**
|
||||
* Optional, opt-in: called before execution to place host files/directories at
|
||||
* target sandbox paths using a provider-native transport instead of the default
|
||||
|
|
|
|||
|
|
@ -188,6 +188,8 @@ export type {
|
|||
PluginEnvironmentRealizeWorkspaceResult,
|
||||
PluginEnvironmentExecuteParams,
|
||||
PluginEnvironmentExecuteResult,
|
||||
PluginEnvironmentRunnerIngressEndpointParams,
|
||||
PluginEnvironmentRunnerIngressEndpoint,
|
||||
PluginSyncFileMapping,
|
||||
PluginPostUploadCommand,
|
||||
PluginSyncOperation,
|
||||
|
|
|
|||
|
|
@ -614,6 +614,21 @@ export interface PluginEnvironmentLease {
|
|||
expiresAt?: string | null;
|
||||
}
|
||||
|
||||
/** Serializable provider result. The host adds refresh/close lifecycle methods. */
|
||||
export interface PluginEnvironmentRunnerIngressEndpoint {
|
||||
kind: "authenticated_websocket";
|
||||
websocketUrl: string;
|
||||
secretHeaders: Array<{ name: string; value: string }>;
|
||||
generation: string;
|
||||
}
|
||||
|
||||
export interface PluginEnvironmentRunnerIngressEndpointParams
|
||||
extends PluginEnvironmentDriverBaseParams {
|
||||
lease: PluginEnvironmentLease;
|
||||
port: number;
|
||||
path: string;
|
||||
}
|
||||
|
||||
export interface PluginEnvironmentAcquireLeaseParams extends PluginEnvironmentDriverBaseParams {
|
||||
runId: string;
|
||||
workspaceMode?: string;
|
||||
|
|
@ -1350,6 +1365,10 @@ export interface HostToWorkerMethods {
|
|||
params: PluginEnvironmentExecuteParams,
|
||||
result: PluginEnvironmentExecuteResult,
|
||||
];
|
||||
environmentRunnerIngressEndpoint: [
|
||||
params: PluginEnvironmentRunnerIngressEndpointParams,
|
||||
result: PluginEnvironmentRunnerIngressEndpoint,
|
||||
];
|
||||
environmentSyncIn: [
|
||||
params: PluginEnvironmentSyncInParams,
|
||||
result: PluginEnvironmentSyncResult,
|
||||
|
|
@ -1440,6 +1459,7 @@ export const HOST_TO_WORKER_OPTIONAL_METHODS: readonly HostToWorkerMethodName[]
|
|||
"environmentDestroyLease",
|
||||
"environmentRealizeWorkspace",
|
||||
"environmentExecute",
|
||||
"environmentRunnerIngressEndpoint",
|
||||
"environmentSyncIn",
|
||||
"environmentSyncOut",
|
||||
"environmentStartInteractiveSetup",
|
||||
|
|
|
|||
|
|
@ -88,6 +88,7 @@ import type {
|
|||
PluginEnvironmentAcquireLeaseParams,
|
||||
PluginEnvironmentDestroyLeaseParams,
|
||||
PluginEnvironmentExecuteParams,
|
||||
PluginEnvironmentRunnerIngressEndpointParams,
|
||||
PluginEnvironmentSyncInParams,
|
||||
PluginEnvironmentSyncOutParams,
|
||||
PluginEnvironmentRealizeWorkspaceParams,
|
||||
|
|
@ -1640,6 +1641,11 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
|
|||
case "environmentExecute":
|
||||
return handleEnvironmentExecute(params as PluginEnvironmentExecuteParams);
|
||||
|
||||
case "environmentRunnerIngressEndpoint":
|
||||
return handleEnvironmentRunnerIngressEndpoint(
|
||||
params as PluginEnvironmentRunnerIngressEndpointParams,
|
||||
);
|
||||
|
||||
case "environmentSyncIn":
|
||||
return handleEnvironmentSyncIn(params as PluginEnvironmentSyncInParams);
|
||||
|
||||
|
|
@ -1729,6 +1735,9 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
|
|||
if (plugin.definition.onEnvironmentDestroyLease) supportedMethods.push("environmentDestroyLease");
|
||||
if (plugin.definition.onEnvironmentRealizeWorkspace) supportedMethods.push("environmentRealizeWorkspace");
|
||||
if (plugin.definition.onEnvironmentExecute) supportedMethods.push("environmentExecute");
|
||||
if (plugin.definition.onEnvironmentRunnerIngressEndpoint) {
|
||||
supportedMethods.push("environmentRunnerIngressEndpoint");
|
||||
}
|
||||
if (plugin.definition.onEnvironmentSyncIn) supportedMethods.push("environmentSyncIn");
|
||||
if (plugin.definition.onEnvironmentSyncOut) supportedMethods.push("environmentSyncOut");
|
||||
if (plugin.definition.onEnvironmentStartInteractiveSetup) supportedMethods.push("environmentStartInteractiveSetup");
|
||||
|
|
@ -2043,6 +2052,15 @@ export function startWorkerRpcHost(options: WorkerRpcHostOptions): WorkerRpcHost
|
|||
return plugin.definition.onEnvironmentExecute(params);
|
||||
}
|
||||
|
||||
async function handleEnvironmentRunnerIngressEndpoint(
|
||||
params: PluginEnvironmentRunnerIngressEndpointParams,
|
||||
) {
|
||||
if (!plugin.definition.onEnvironmentRunnerIngressEndpoint) {
|
||||
throw methodNotImplemented("environmentRunnerIngressEndpoint");
|
||||
}
|
||||
return plugin.definition.onEnvironmentRunnerIngressEndpoint(params);
|
||||
}
|
||||
|
||||
async function handleEnvironmentSyncIn(params: PluginEnvironmentSyncInParams) {
|
||||
if (!plugin.definition.onEnvironmentSyncIn) {
|
||||
throw methodNotImplemented("environmentSyncIn");
|
||||
|
|
|
|||
|
|
@ -269,6 +269,14 @@ export const INSTANCE_FEATURE_CATALOG: Record<InstanceFeatureKey, FeatureCatalog
|
|||
cloudDefault: false,
|
||||
selfHostedDefault: false,
|
||||
},
|
||||
enableRunnerPreviewIngress: {
|
||||
title: "Runner Preview Ingress",
|
||||
description:
|
||||
"Let paperclip_runner connect through an authenticated sandbox-provider WebSocket ingress. Legacy adapters never use this path.",
|
||||
tier: "managed",
|
||||
cloudDefault: false,
|
||||
selfHostedDefault: false,
|
||||
},
|
||||
enableWorktreeRunExecution: {
|
||||
title: "Worktree Run Execution",
|
||||
description:
|
||||
|
|
|
|||
|
|
@ -30,6 +30,10 @@ export interface FakeSandboxEnvironmentConfig {
|
|||
reuseLease: boolean;
|
||||
/** Stream agent CLI stdout/stderr during sandbox runs (bridge log-tail loop). */
|
||||
streamRunLogs?: boolean;
|
||||
/** Override the paperclip_runner lifecycle for this environment. */
|
||||
runnerLifecycleMode?: "inherit" | "per_turn" | "warm";
|
||||
/** Warm runner idle timeout in milliseconds when runnerLifecycleMode is warm. */
|
||||
runnerIdleTimeoutMs?: number;
|
||||
/**
|
||||
* Archive the sandbox on lease release instead of deleting it, so operators
|
||||
* can inspect it from the provider dashboard. Injected by test/probe paths;
|
||||
|
|
@ -44,6 +48,10 @@ export interface PluginSandboxEnvironmentConfig {
|
|||
timeoutMs?: number;
|
||||
/** Stream agent CLI stdout/stderr during sandbox runs (bridge log-tail loop). */
|
||||
streamRunLogs?: boolean;
|
||||
/** Override the paperclip_runner lifecycle for this environment. */
|
||||
runnerLifecycleMode?: "inherit" | "per_turn" | "warm";
|
||||
/** Warm runner idle timeout in milliseconds when runnerLifecycleMode is warm. */
|
||||
runnerIdleTimeoutMs?: number;
|
||||
/**
|
||||
* Archive the sandbox on lease release instead of deleting it, so operators
|
||||
* can inspect it from the provider dashboard. Injected by test/probe paths;
|
||||
|
|
|
|||
|
|
@ -101,6 +101,8 @@ export interface InstanceExperimentalSettings {
|
|||
* redeploy.
|
||||
*/
|
||||
enableSandboxDuplexBridge: boolean;
|
||||
/** Default-off rollout gate for paperclip_runner provider WebSocket ingress. */
|
||||
enableRunnerPreviewIngress: boolean;
|
||||
/**
|
||||
* Worktree preview instances (`PAPERCLIP_IN_WORKTREE=true`) suppress the
|
||||
* heartbeat run engine by default so previews never self-execute tasks. When
|
||||
|
|
|
|||
|
|
@ -186,6 +186,8 @@ export interface SandboxProviderCapabilities {
|
|||
* HTTP/2 is the preferred transport. `queue_v1` is the soft-deprecated fallback.
|
||||
*/
|
||||
duplexCommandStream?: boolean;
|
||||
/** Provider can expose runnerd through a private authenticated WebSocket ingress. */
|
||||
runnerWebSocketIngress?: boolean;
|
||||
}
|
||||
|
||||
export interface PluginEnvironmentDriverDeclaration {
|
||||
|
|
|
|||
|
|
@ -75,6 +75,7 @@ export const instanceExperimentalSettingsSchema = z.object({
|
|||
// off the host keeps the file bridge for every run with no manifest change and
|
||||
// no redeploy. The host reads this per run before it selects the transport.
|
||||
enableSandboxDuplexBridge: z.boolean().default(false),
|
||||
enableRunnerPreviewIngress: z.boolean().default(false),
|
||||
enableWorktreeRunExecution: z.boolean().default(false),
|
||||
worktreeRunExecutionActivatedAt: z.string().datetime().nullable().default(null),
|
||||
worktreeRunExecutionActivationInstanceId: z.string().min(1).nullable().default(null),
|
||||
|
|
|
|||
|
|
@ -168,6 +168,7 @@ export const sandboxProviderCapabilitiesSchema = z.object({
|
|||
incrementalSessionOutput: z.boolean().optional(),
|
||||
concurrentSyncOperations: z.boolean().optional(),
|
||||
duplexCommandStream: z.boolean().optional(),
|
||||
runnerWebSocketIngress: z.boolean().optional(),
|
||||
}).strict();
|
||||
|
||||
export type SandboxProviderCapabilitiesInput = z.infer<typeof sandboxProviderCapabilitiesSchema>;
|
||||
|
|
|
|||
|
|
@ -598,6 +598,65 @@ describe("agent routes adapter validation", () => {
|
|||
expect(mockAgentService.create).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("rejects non-Codex providers on fresh paperclip_runner agents and hires", async () => {
|
||||
mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableNativeRunner: true });
|
||||
const app = await createApp();
|
||||
const createResponse = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl)
|
||||
.post("/api/companies/company-1/agents")
|
||||
.send({
|
||||
name: "Native OpenCode",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "opencode" },
|
||||
}),
|
||||
);
|
||||
const hireResponse = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl)
|
||||
.post("/api/companies/company-1/agent-hires")
|
||||
.send({
|
||||
name: "Native ACPX",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "acpx" },
|
||||
}),
|
||||
);
|
||||
|
||||
expect(createResponse.status, JSON.stringify(createResponse.body)).toBe(422);
|
||||
expect(createResponse.body.details).toMatchObject({
|
||||
code: "paperclip_runner_provider_unavailable",
|
||||
});
|
||||
expect(hireResponse.status, JSON.stringify(hireResponse.body)).toBe(422);
|
||||
expect(hireResponse.body.details).toMatchObject({
|
||||
code: "paperclip_runner_provider_unavailable",
|
||||
});
|
||||
expect(mockAgentService.create).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects provider changes but preserves edits to historical runner agents", async () => {
|
||||
const existing = await mockAgentService.getById();
|
||||
mockAgentService.getById.mockResolvedValue({
|
||||
...existing,
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "opencode", model: "historical" },
|
||||
});
|
||||
const app = await createApp();
|
||||
const ordinaryEdit = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl)
|
||||
.patch("/api/agents/11111111-1111-4111-8111-111111111111")
|
||||
.send({ name: "Historical Runner" }),
|
||||
);
|
||||
const providerChange = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl)
|
||||
.patch("/api/agents/11111111-1111-4111-8111-111111111111")
|
||||
.send({ adapterConfig: { provider: "acpx" } }),
|
||||
);
|
||||
|
||||
expect(ordinaryEdit.status, JSON.stringify(ordinaryEdit.body)).toBe(200);
|
||||
expect(providerChange.status, JSON.stringify(providerChange.body)).toBe(422);
|
||||
expect(providerChange.body.details).toMatchObject({
|
||||
code: "paperclip_runner_provider_unavailable",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps an existing paperclip_runner agent editable after the flag is disabled", async () => {
|
||||
const existing = await mockAgentService.getById();
|
||||
mockAgentService.getById.mockResolvedValue({
|
||||
|
|
|
|||
|
|
@ -42,11 +42,13 @@ const baseAgent = {
|
|||
|
||||
const mockAgentService = vi.hoisted(() => ({
|
||||
getById: vi.fn(),
|
||||
getConfigRevision: vi.fn(),
|
||||
list: vi.fn(),
|
||||
create: vi.fn(),
|
||||
activatePendingApproval: vi.fn(),
|
||||
terminate: vi.fn(),
|
||||
update: vi.fn(),
|
||||
rollbackConfigRevision: vi.fn(),
|
||||
updatePermissions: vi.fn(),
|
||||
getChainOfCommand: vi.fn(),
|
||||
resolveByReference: vi.fn(),
|
||||
|
|
@ -288,11 +290,13 @@ describe.sequential("agent permission routes", () => {
|
|||
beforeEach(() => {
|
||||
vi.resetAllMocks();
|
||||
mockAgentService.getById.mockReset();
|
||||
mockAgentService.getConfigRevision.mockReset();
|
||||
mockAgentService.list.mockReset();
|
||||
mockAgentService.create.mockReset();
|
||||
mockAgentService.activatePendingApproval.mockReset();
|
||||
mockAgentService.terminate.mockReset();
|
||||
mockAgentService.update.mockReset();
|
||||
mockAgentService.rollbackConfigRevision.mockReset();
|
||||
mockAgentService.updatePermissions.mockReset();
|
||||
mockAgentService.getChainOfCommand.mockReset();
|
||||
mockAgentService.resolveByReference.mockReset();
|
||||
|
|
@ -332,6 +336,7 @@ describe.sequential("agent permission routes", () => {
|
|||
mockSyncInstructionsBundleConfigFromFilePath.mockImplementation((_agent, config) => config);
|
||||
mockGetTelemetryClient.mockReturnValue({ track: vi.fn() });
|
||||
mockAgentService.getById.mockResolvedValue(baseAgent);
|
||||
mockAgentService.getConfigRevision.mockResolvedValue(null);
|
||||
mockAgentService.list.mockResolvedValue([baseAgent]);
|
||||
mockAgentService.getChainOfCommand.mockResolvedValue([]);
|
||||
mockAgentService.resolveByReference.mockResolvedValue({ ambiguous: false, agent: baseAgent });
|
||||
|
|
@ -570,6 +575,63 @@ describe.sequential("agent permission routes", () => {
|
|||
expect(mockAgentService.create).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("allows instance administrators to create and hire with raw provider traces", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
userId: "instance-admin-user",
|
||||
source: "session",
|
||||
isInstanceAdmin: true,
|
||||
companyIds: [companyId],
|
||||
});
|
||||
const body = {
|
||||
name: "Trace capture agent",
|
||||
adapterType: "process",
|
||||
runtimeConfig: { debug: { providerTrace: "raw" } },
|
||||
};
|
||||
|
||||
const createResponse = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl)
|
||||
.post(`/api/companies/${companyId}/agents`)
|
||||
.send(body),
|
||||
);
|
||||
const hireResponse = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl)
|
||||
.post(`/api/companies/${companyId}/agent-hires`)
|
||||
.send(body),
|
||||
);
|
||||
|
||||
expect(createResponse.status, JSON.stringify(createResponse.body)).toBe(201);
|
||||
expect(hireResponse.status, JSON.stringify(hireResponse.body)).toBe(201);
|
||||
expect(mockAgentService.create).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("rejects non-admin rollback into raw provider trace capture", async () => {
|
||||
mockAgentService.getConfigRevision.mockResolvedValue({
|
||||
id: "33333333-3333-4333-8333-333333333333",
|
||||
afterConfig: {
|
||||
adapterType: "process",
|
||||
adapterConfig: {},
|
||||
runtimeConfig: { debug: { providerTrace: "raw" } },
|
||||
},
|
||||
});
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
userId: "agent-admin-user",
|
||||
source: "session",
|
||||
isInstanceAdmin: false,
|
||||
companyIds: [companyId],
|
||||
});
|
||||
|
||||
const response = await requestApp(app, (baseUrl) =>
|
||||
request(baseUrl).post(
|
||||
`/api/agents/${agentId}/config-revisions/33333333-3333-4333-8333-333333333333/rollback`,
|
||||
),
|
||||
);
|
||||
|
||||
expect(response.status).toBe(403);
|
||||
expect(mockAgentService.rollbackConfigRevision).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("blocks api key creation for authenticated company members without agent admin permission", async () => {
|
||||
mockAccessService.canUser.mockResolvedValue(false);
|
||||
|
||||
|
|
|
|||
|
|
@ -401,8 +401,13 @@ describe("general runtime capability resolver — four-driver matrix", () => {
|
|||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: true,
|
||||
duplexCommandStream: true,
|
||||
runnerWebSocketIngress: true,
|
||||
};
|
||||
const VERIFY_ALL = [...ALL_PLUGIN_METHODS, "duplexChannelOpen"];
|
||||
const VERIFY_ALL = [
|
||||
...ALL_PLUGIN_METHODS,
|
||||
"duplexChannelOpen",
|
||||
"environmentRunnerIngressEndpoint",
|
||||
];
|
||||
|
||||
it("test_local_and_ssh_drivers_support_no_capability_regardless_of_declaration_or_worker", () => {
|
||||
// The `local` and `ssh` static support definitions name none of the
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ const SNAPSHOT: EffectiveExecutionCapabilities = {
|
|||
// inbound sync, so the opt-in stays off.
|
||||
concurrentSyncOperations: false,
|
||||
duplexCommandStream: false,
|
||||
runnerWebSocketIngress: false,
|
||||
};
|
||||
|
||||
// A snapshot that grants every capability. A test overrides one flag to prove
|
||||
|
|
@ -36,6 +37,7 @@ const FULL_GRANT: EffectiveExecutionCapabilities = {
|
|||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: true,
|
||||
duplexCommandStream: true,
|
||||
runnerWebSocketIngress: true,
|
||||
};
|
||||
|
||||
// Build a sandbox execution target with a fixed snapshot and a fixed
|
||||
|
|
@ -197,6 +199,62 @@ describe("resolveEnvironmentExecutionTarget effective capability snapshot", () =
|
|||
expect(target?.kind).toBe("remote");
|
||||
expect(resolveCapabilities).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("carries the environment-owned warm runner lifecycle and reuse requirement", async () => {
|
||||
const { target } = await buildSandboxTarget({
|
||||
snapshot: FULL_GRANT,
|
||||
supportsSync: false,
|
||||
config: {
|
||||
reuseLease: true,
|
||||
runnerLifecycleMode: "warm",
|
||||
runnerIdleTimeoutMs: 45_000,
|
||||
},
|
||||
});
|
||||
|
||||
expect(target.runnerLifecyclePolicy).toEqual({
|
||||
mode: "warm",
|
||||
idleTimeoutMs: 45_000,
|
||||
});
|
||||
expect(target.reusableLeaseConfigured).toBe(true);
|
||||
});
|
||||
|
||||
it("carries host-owned sandbox acquisition provenance without persisting provider ids in metadata", async () => {
|
||||
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
|
||||
driver: "sandbox",
|
||||
config: { provider: "daytona", reuseLease: true, timeoutMs: 30_000 },
|
||||
});
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
db: {} as never,
|
||||
companyId: "company-1",
|
||||
// This substrate PR does not advertise remote paperclip_runner support
|
||||
// until the Rust WSS transport lands. A supported direct adapter exercises
|
||||
// the same host-owned acquisition contract without widening rollout here.
|
||||
adapterType: "codex_local",
|
||||
environment: { id: "env-1", driver: "sandbox", config: { provider: "daytona" } },
|
||||
leaseId: "lease-row-1",
|
||||
leaseMetadata: {
|
||||
remoteCwd: "/work",
|
||||
sandboxLeaseAcquisition: { outcome: "resumed" },
|
||||
},
|
||||
lease: {
|
||||
id: "lease-row-1",
|
||||
providerLeaseId: "daytona-sandbox-1",
|
||||
leasePolicy: "reuse_by_environment",
|
||||
metadata: { sandboxLeaseAcquisition: { outcome: "resumed" } },
|
||||
} as never,
|
||||
environmentRuntime: {
|
||||
supportsSync: () => false,
|
||||
resolveCapabilities: vi.fn(async () => ({ ...FULL_GRANT })),
|
||||
} as unknown as EnvironmentRuntimeService,
|
||||
});
|
||||
if (target?.kind !== "remote" || target.transport !== "sandbox") {
|
||||
throw new Error("expected a sandbox target");
|
||||
}
|
||||
expect(target.sandboxLeaseAcquisition).toEqual({
|
||||
outcome: "resumed",
|
||||
providerLeaseId: "daytona-sandbox-1",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("effective snapshot gates the sync decision", () => {
|
||||
|
|
|
|||
|
|
@ -48,6 +48,7 @@ const DUPLEX_GRANT: EffectiveExecutionCapabilities = {
|
|||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: true,
|
||||
duplexCommandStream: true,
|
||||
runnerWebSocketIngress: true,
|
||||
};
|
||||
|
||||
const DUPLEX_ABSENT: EffectiveExecutionCapabilities = {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { mkdtemp, readdir, rm } from "node:fs/promises";
|
||||
import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
|
@ -49,6 +49,8 @@ import {
|
|||
} from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
|
||||
import { traceparentFromContextToken } from "../instrumentation.ts";
|
||||
import { ROOT_CONTEXT, trace } from "@opentelemetry/api";
|
||||
import { buildNativeHarnessBackupManifest } from "../services/native-runtime/native-session-executor.ts";
|
||||
import { createNativeHarnessBackupStamp } from "../services/native-runtime/native-harness-backup-stamp.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
|
@ -314,7 +316,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
};
|
||||
}
|
||||
|
||||
async function seedReusablePluginSandboxLease() {
|
||||
async function seedReusablePluginSandboxLease(adapterType: string | null = null) {
|
||||
const pluginId = randomUUID();
|
||||
const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment();
|
||||
const providerConfig = {
|
||||
|
|
@ -418,11 +420,11 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
environmentId: environment.id,
|
||||
executionWorkspaceId,
|
||||
agentId,
|
||||
adapterType: null,
|
||||
adapterType,
|
||||
provider: "fake-plugin",
|
||||
runtimeFingerprint: reusableRuntimeFingerprint({
|
||||
provider: "fake-plugin",
|
||||
adapterType: null,
|
||||
adapterType,
|
||||
config: providerConfig,
|
||||
}),
|
||||
},
|
||||
|
|
@ -432,6 +434,244 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
return { pluginId, companyId, agentId, environment, runId, executionWorkspaceId, reusableLease };
|
||||
}
|
||||
|
||||
it("retains a successful reusable sandbox lease without stopping the provider resource", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
throw new Error(`Unexpected plugin method while retaining warm lease: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const released = await runtimeWithPlugin.releaseRunLeases(
|
||||
runId,
|
||||
"released",
|
||||
undefined,
|
||||
"keep_running",
|
||||
);
|
||||
|
||||
expect(released).toHaveLength(1);
|
||||
expect(released[0]?.lease).toMatchObject({
|
||||
id: reusableLease.id,
|
||||
status: "retained",
|
||||
cleanupStatus: "success",
|
||||
});
|
||||
await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({
|
||||
status: "retained",
|
||||
cleanupStatus: "success",
|
||||
});
|
||||
expect(workerManager.call).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("stops a reusable sandbox and keeps its lease resumable", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentReleaseLease") return undefined;
|
||||
throw new Error(`Unexpected plugin method while stopping lease: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const released = await runtimeWithPlugin.releaseRunLeases(
|
||||
runId,
|
||||
"failed",
|
||||
undefined,
|
||||
"stop_and_retain",
|
||||
);
|
||||
|
||||
expect(released[0]?.lease).toMatchObject({
|
||||
id: reusableLease.id,
|
||||
status: "released",
|
||||
});
|
||||
expect(workerManager.call).toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentReleaseLease",
|
||||
expect.objectContaining({ providerLeaseId: reusableLease.providerLeaseId }),
|
||||
expect.any(Number),
|
||||
);
|
||||
});
|
||||
|
||||
it("resumes the same provider lease on the next per-turn run", async () => {
|
||||
const seeded = await seedReusablePluginSandboxLease();
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === seeded.pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentAcquireLease") {
|
||||
return {
|
||||
providerLeaseId: "sandbox-exact-resume",
|
||||
metadata: {
|
||||
provider: "fake-plugin",
|
||||
image: "fake:test",
|
||||
timeoutMs: 1234,
|
||||
reuseLease: true,
|
||||
remoteCwd: "/workspace",
|
||||
},
|
||||
};
|
||||
}
|
||||
if (method === "environmentReleaseLease") return undefined;
|
||||
if (method === "environmentResumeLease") {
|
||||
return {
|
||||
providerLeaseId: "sandbox-exact-resume",
|
||||
metadata: {
|
||||
provider: "fake-plugin",
|
||||
image: "fake:test",
|
||||
timeoutMs: 1234,
|
||||
reuseLease: true,
|
||||
remoteCwd: "/workspace",
|
||||
},
|
||||
};
|
||||
}
|
||||
throw new Error(`Unexpected plugin method during exact resume: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
await environmentService(db).releaseLease(seeded.reusableLease.id, "expired");
|
||||
const first = await runtimeWithPlugin.acquireRunLease({
|
||||
companyId: seeded.companyId,
|
||||
environment: seeded.environment,
|
||||
issueId: null,
|
||||
agentId: seeded.agentId,
|
||||
heartbeatRunId: seeded.runId,
|
||||
persistedExecutionWorkspace: {
|
||||
id: seeded.executionWorkspaceId,
|
||||
mode: "shared_workspace",
|
||||
},
|
||||
});
|
||||
await runtimeWithPlugin.releaseRunLeases(
|
||||
seeded.runId,
|
||||
"released",
|
||||
undefined,
|
||||
"stop_and_retain",
|
||||
);
|
||||
|
||||
const nextRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: nextRunId,
|
||||
companyId: seeded.companyId,
|
||||
agentId: seeded.agentId,
|
||||
invocationSource: "manual",
|
||||
status: "running",
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
});
|
||||
const acquired = await runtimeWithPlugin.acquireRunLease({
|
||||
companyId: seeded.companyId,
|
||||
environment: seeded.environment,
|
||||
issueId: null,
|
||||
agentId: seeded.agentId,
|
||||
heartbeatRunId: nextRunId,
|
||||
persistedExecutionWorkspace: {
|
||||
id: seeded.executionWorkspaceId,
|
||||
mode: "shared_workspace",
|
||||
},
|
||||
});
|
||||
|
||||
expect(first.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "created" });
|
||||
expect(acquired.lease.providerLeaseId).toBe("sandbox-exact-resume");
|
||||
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({ outcome: "resumed" });
|
||||
expect(workerManager.call.mock.calls.filter((call) => call[1] === "environmentAcquireLease"))
|
||||
.toHaveLength(1);
|
||||
});
|
||||
|
||||
it("destroys a disposable paperclip_runner sandbox after the turn", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentDestroyLease") return undefined;
|
||||
throw new Error(`Unexpected plugin method while destroying lease: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const released = await runtimeWithPlugin.releaseRunLeases(
|
||||
runId,
|
||||
"released",
|
||||
undefined,
|
||||
"destroy",
|
||||
);
|
||||
|
||||
expect(released[0]?.lease).toMatchObject({
|
||||
id: reusableLease.id,
|
||||
status: "expired",
|
||||
});
|
||||
expect(workerManager.call).toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentDestroyLease",
|
||||
expect.objectContaining({ providerLeaseId: reusableLease.providerLeaseId }),
|
||||
expect.any(Number),
|
||||
);
|
||||
});
|
||||
|
||||
it("does not destroy native sandbox state when its verified backup is missing", async () => {
|
||||
const { pluginId, runId, reusableLease } = await seedReusablePluginSandboxLease();
|
||||
await environmentService(db).updateLeaseMetadata(reusableLease.id, {
|
||||
...(reusableLease.metadata ?? {}),
|
||||
sandboxLeaseAcquisition: { outcome: "created" },
|
||||
});
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
throw new Error(`Provider teardown must not run without a backup: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
const errors: unknown[] = [];
|
||||
|
||||
const released = await runtimeWithPlugin.releaseRunLeases(
|
||||
runId,
|
||||
"released",
|
||||
(_leaseId, error) => errors.push(error),
|
||||
"destroy",
|
||||
);
|
||||
|
||||
expect(released).toEqual([]);
|
||||
expect(errors).toHaveLength(1);
|
||||
expect(String(errors[0])).toContain("runner_harness_backup_unavailable");
|
||||
expect(workerManager.call).not.toHaveBeenCalled();
|
||||
await expect(environmentService(db).getLeaseById(reusableLease.id)).resolves.toMatchObject({
|
||||
status: "active",
|
||||
});
|
||||
});
|
||||
|
||||
it("acquires and releases a local run lease through the runtime seam", async () => {
|
||||
const { companyId, environment, runId } = await seedEnvironment();
|
||||
|
||||
|
|
@ -4166,7 +4406,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
);
|
||||
});
|
||||
|
||||
it("falls back to acquire when plugin-backed sandbox lease resume throws", async () => {
|
||||
it("preserves the exact lease when plugin-backed sandbox resume throws", async () => {
|
||||
const pluginId = randomUUID();
|
||||
const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment();
|
||||
const providerConfig = {
|
||||
|
|
@ -4308,7 +4548,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const acquired = await runtimeWithPlugin.acquireRunLease({
|
||||
await expect(runtimeWithPlugin.acquireRunLease({
|
||||
companyId,
|
||||
environment,
|
||||
issueId: null,
|
||||
|
|
@ -4318,34 +4558,196 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
id: executionWorkspaceId,
|
||||
mode: "shared_workspace",
|
||||
},
|
||||
});
|
||||
})).rejects.toThrow("the lease was preserved and no replacement was created");
|
||||
|
||||
expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease");
|
||||
expect(workerManager.call).toHaveBeenNthCalledWith(1, pluginId, "environmentResumeLease", expect.objectContaining({
|
||||
driverKey: "fake-plugin",
|
||||
providerLeaseId: "stale-plugin-lease",
|
||||
}), 31234);
|
||||
expect(workerManager.call).toHaveBeenNthCalledWith(2, pluginId, "environmentDestroyLease", expect.objectContaining({
|
||||
driverKey: "fake-plugin",
|
||||
providerLeaseId: "stale-plugin-lease",
|
||||
}), 31234);
|
||||
expect(workerManager.call).toHaveBeenNthCalledWith(3, pluginId, "environmentAcquireLease", expect.objectContaining({
|
||||
driverKey: "fake-plugin",
|
||||
config: {
|
||||
image: "fake:test",
|
||||
timeoutMs: 1234,
|
||||
reuseLease: true,
|
||||
},
|
||||
agentId,
|
||||
executionWorkspaceId,
|
||||
runId,
|
||||
}), 31234);
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentDestroyLease",
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentAcquireLease",
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
await expect(environmentService(db).getLeaseById(staleLease.id)).resolves.toMatchObject({
|
||||
status: "expired",
|
||||
cleanupStatus: "success",
|
||||
status: "active",
|
||||
});
|
||||
});
|
||||
|
||||
it("does not allocate a native-runner replacement without a verified backup", async () => {
|
||||
const seeded = await seedReusablePluginSandboxLease("paperclip_runner");
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === seeded.pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentResumeLease") {
|
||||
return { providerLeaseId: null, metadata: { expired: true } };
|
||||
}
|
||||
if (method === "environmentDestroyLease") return undefined;
|
||||
if (method === "environmentAcquireLease") {
|
||||
throw new Error("replacement must not be allocated");
|
||||
}
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
await expect(runtimeWithPlugin.acquireRunLease({
|
||||
companyId: seeded.companyId,
|
||||
environment: seeded.environment,
|
||||
issueId: null,
|
||||
agentId: seeded.agentId,
|
||||
adapterType: "paperclip_runner",
|
||||
heartbeatRunId: seeded.runId,
|
||||
persistedExecutionWorkspace: {
|
||||
id: seeded.executionWorkspaceId,
|
||||
mode: "shared_workspace",
|
||||
},
|
||||
})).rejects.toThrow("runner_harness_backup_unavailable");
|
||||
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
seeded.pluginId,
|
||||
"environmentAcquireLease",
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
seeded.pluginId,
|
||||
"environmentDestroyLease",
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
});
|
||||
|
||||
it("permits native-runner replacement only after verifying the stamped backup", async () => {
|
||||
const seeded = await seedReusablePluginSandboxLease("paperclip_runner");
|
||||
const backupBase = await mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-replacement-"));
|
||||
const previousStateDirectory = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = backupBase;
|
||||
try {
|
||||
const normalizedSessionId = "native-replacement-session";
|
||||
const runnerInstanceId = "native-replacement-runner";
|
||||
const sessionRoot = path.join(
|
||||
backupBase,
|
||||
createHash("sha256").update(normalizedSessionId).digest("hex"),
|
||||
);
|
||||
const current = path.join(sessionRoot, "failover-backups", "current");
|
||||
await mkdir(path.join(current, "runner"), { recursive: true });
|
||||
await mkdir(path.join(current, "codex-home", "sessions"), { recursive: true });
|
||||
await writeFile(path.join(current, "runner", "runner-state.json"), "runner-state");
|
||||
await writeFile(path.join(current, "codex-home", "sessions", "thread.jsonl"), "thread-state");
|
||||
const execution = {
|
||||
provider: { kind: "codex", model: null, approvalPolicy: "never" },
|
||||
binding: {
|
||||
companyId: seeded.companyId,
|
||||
runId: seeded.runId,
|
||||
issueId: "issue",
|
||||
agentId: seeded.agentId,
|
||||
executionWorkspaceId: seeded.executionWorkspaceId,
|
||||
},
|
||||
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: {
|
||||
normalizedSessionId,
|
||||
driverKind: "codex_app_server",
|
||||
protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
},
|
||||
} as never;
|
||||
const manifest = buildNativeHarnessBackupManifest({
|
||||
backupRoot: current,
|
||||
execution,
|
||||
runnerInstanceId,
|
||||
providerSessionIdentity: {
|
||||
providerSessionId: "thread-1",
|
||||
providerBackendSessionId: "session-1",
|
||||
providerSessionIdentity: null,
|
||||
},
|
||||
sourceProviderLeaseId: seeded.reusableLease.providerLeaseId!,
|
||||
});
|
||||
const manifestPath = path.join(current, "manifest.json");
|
||||
await writeFile(manifestPath, JSON.stringify(manifest));
|
||||
const stamp = createNativeHarnessBackupStamp({
|
||||
manifestPath,
|
||||
normalizedSessionId,
|
||||
runnerInstanceId,
|
||||
completedAt: manifest.completedAt,
|
||||
});
|
||||
await environmentService(db).updateLeaseMetadata(seeded.reusableLease.id, {
|
||||
...(seeded.reusableLease.metadata ?? {}),
|
||||
nativeHarnessBackup: stamp,
|
||||
});
|
||||
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === seeded.pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string) => {
|
||||
if (method === "environmentResumeLease") {
|
||||
return { providerLeaseId: null, metadata: { expired: true } };
|
||||
}
|
||||
if (method === "environmentDestroyLease") return undefined;
|
||||
if (method === "environmentAcquireLease") {
|
||||
return {
|
||||
providerLeaseId: "replacement-plugin-lease",
|
||||
metadata: {
|
||||
provider: "fake-plugin",
|
||||
image: "fake:test",
|
||||
timeoutMs: 1234,
|
||||
reuseLease: true,
|
||||
remoteCwd: "/workspace",
|
||||
},
|
||||
};
|
||||
}
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
}),
|
||||
getWorker: vi.fn(() => ({
|
||||
supportedMethods: [
|
||||
"environmentResumeLease",
|
||||
"environmentReleaseLease",
|
||||
"environmentDestroyLease",
|
||||
],
|
||||
})),
|
||||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
const acquired = await runtimeWithPlugin.acquireRunLease({
|
||||
companyId: seeded.companyId,
|
||||
environment: seeded.environment,
|
||||
issueId: null,
|
||||
agentId: seeded.agentId,
|
||||
adapterType: "paperclip_runner",
|
||||
heartbeatRunId: seeded.runId,
|
||||
persistedExecutionWorkspace: {
|
||||
id: seeded.executionWorkspaceId,
|
||||
mode: "shared_workspace",
|
||||
},
|
||||
});
|
||||
|
||||
expect(acquired.lease.providerLeaseId).toBe("replacement-plugin-lease");
|
||||
expect(acquired.lease.metadata?.sandboxLeaseAcquisition).toEqual({
|
||||
outcome: "replacement",
|
||||
reason: "not_found",
|
||||
});
|
||||
} finally {
|
||||
if (previousStateDirectory === undefined) {
|
||||
delete process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
} else {
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = previousStateDirectory;
|
||||
}
|
||||
await rm(backupBase, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("fails closed and does not resume when a worker restart drops the resume method after the capability snapshot", async () => {
|
||||
const pluginId = randomUUID();
|
||||
const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment();
|
||||
|
|
@ -4505,7 +4907,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
} as unknown as PluginWorkerManager;
|
||||
const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager });
|
||||
|
||||
const acquired = await runtimeWithPlugin.acquireRunLease({
|
||||
await expect(runtimeWithPlugin.acquireRunLease({
|
||||
companyId,
|
||||
environment,
|
||||
issueId: null,
|
||||
|
|
@ -4515,7 +4917,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
id: executionWorkspaceId,
|
||||
mode: "shared_workspace",
|
||||
},
|
||||
});
|
||||
})).rejects.toThrow("the lease was preserved and no replacement was created");
|
||||
|
||||
// The runtime re-checks the live worker before the resume dispatch. The
|
||||
// restarted worker no longer advertises `environmentResumeLease`, so the
|
||||
|
|
@ -4526,23 +4928,21 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
|||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
// It destroys the stale reusable lease and acquires a fresh one.
|
||||
expect(workerManager.call).toHaveBeenCalledWith(
|
||||
// It preserves the stale reusable lease and does not allocate a replacement.
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentDestroyLease",
|
||||
expect.objectContaining({ driverKey: "fake-plugin", providerLeaseId: "stale-plugin-lease" }),
|
||||
31234,
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(workerManager.call).toHaveBeenCalledWith(
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentAcquireLease",
|
||||
expect.objectContaining({ driverKey: "fake-plugin", agentId, executionWorkspaceId, runId }),
|
||||
31234,
|
||||
expect.anything(),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(acquired.lease.providerLeaseId).toBe("fresh-plugin-lease");
|
||||
await expect(environmentService(db).getLeaseById(staleLease.id)).resolves.toMatchObject({
|
||||
status: "expired",
|
||||
cleanupStatus: "success",
|
||||
status: "active",
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ const ALL_PROVIDER_METHODS = [
|
|||
"environmentSyncIn",
|
||||
"environmentSyncOut",
|
||||
"duplexChannelOpen",
|
||||
"environmentRunnerIngressEndpoint",
|
||||
];
|
||||
|
||||
describe("general capability classifier", () => {
|
||||
|
|
@ -34,6 +35,7 @@ describe("general capability classifier", () => {
|
|||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: true,
|
||||
duplexCommandStream: true,
|
||||
runnerWebSocketIngress: true,
|
||||
},
|
||||
supportedCapabilities: ENVIRONMENT_DRIVER_CAPABILITY_SUPPORT.sandbox.supportedCapabilities,
|
||||
});
|
||||
|
|
@ -96,6 +98,7 @@ describe("general capability classifier", () => {
|
|||
incrementalSessionOutput: true,
|
||||
concurrentSyncOperations: true,
|
||||
duplexCommandStream: true,
|
||||
runnerWebSocketIngress: true,
|
||||
},
|
||||
supportedCapabilities: ENVIRONMENT_DRIVER_CAPABILITY_SUPPORT[driver].supportedCapabilities,
|
||||
});
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { describe, expect, it, vi } from "vitest";
|
|||
import type { Db } from "@paperclipai/db";
|
||||
|
||||
import { cancelHeartbeatNativeRun } from "../services/heartbeat.js";
|
||||
import { resolveNativeCancellationStatus } from "../services/native-runtime/native-session-executor.js";
|
||||
|
||||
describe("native heartbeat cancellation authority", () => {
|
||||
it("does not enter native cancellation for a direct-adapter run", async () => {
|
||||
|
|
@ -54,4 +55,31 @@ describe("native heartbeat cancellation authority", () => {
|
|||
cancel,
|
||||
})).rejects.toThrow("native_cancellation_outcome_not_audited");
|
||||
});
|
||||
|
||||
it("keeps generalized executor cancellation scoped to the requested authority", () => {
|
||||
expect(resolveNativeCancellationStatus({
|
||||
scope: "run",
|
||||
priorIssueStatus: "in_progress",
|
||||
agentId: "remote-runner",
|
||||
})).toMatchObject({
|
||||
statusAction: "preserve",
|
||||
toStatus: "in_progress",
|
||||
reasonCode: "cancellation_run_only",
|
||||
effects: [{ kind: "release_run_resources" }],
|
||||
});
|
||||
|
||||
expect(resolveNativeCancellationStatus({
|
||||
scope: "issue",
|
||||
priorIssueStatus: "in_progress",
|
||||
agentId: "remote-runner",
|
||||
})).toMatchObject({
|
||||
statusAction: "cancelled",
|
||||
toStatus: "cancelled",
|
||||
reasonCode: "cancellation_issue_authorized",
|
||||
effects: [
|
||||
{ kind: "release_checkout" },
|
||||
{ kind: "cancel_continuations" },
|
||||
],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@ describe("instance settings service", () => {
|
|||
enableWorkspaceDirtyQuarantineRepair: false,
|
||||
enableOwnerInstanceAdmin: false,
|
||||
enableSandboxDuplexBridge: false,
|
||||
enableRunnerPreviewIngress: false,
|
||||
enableWorktreeRunExecution: false,
|
||||
worktreeRunExecutionActivatedAt: null,
|
||||
worktreeRunExecutionActivationInstanceId: null,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { resolveNativeSandboxLifecycle } from "../services/heartbeat.js";
|
||||
|
||||
const reusableSandbox = {
|
||||
kind: "remote" as const,
|
||||
transport: "sandbox",
|
||||
reusableLeaseConfigured: true,
|
||||
effectiveCapabilities: { reusableLeases: true },
|
||||
};
|
||||
|
||||
describe("paperclip_runner sandbox lifecycle", () => {
|
||||
it("keeps a warm reusable sandbox running", () => {
|
||||
expect(resolveNativeSandboxLifecycle({
|
||||
adapterType: "paperclip_runner",
|
||||
lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 },
|
||||
target: reusableSandbox,
|
||||
})).toEqual({
|
||||
runnerProcess: "warm",
|
||||
sandboxResource: "keep_running",
|
||||
failoverBackup: "verified",
|
||||
});
|
||||
});
|
||||
|
||||
it("stops and reuses a per-turn reusable sandbox", () => {
|
||||
expect(resolveNativeSandboxLifecycle({
|
||||
adapterType: "paperclip_runner",
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
target: reusableSandbox,
|
||||
})).toEqual({
|
||||
runnerProcess: "per_turn",
|
||||
sandboxResource: "stop_and_reuse",
|
||||
failoverBackup: "verified",
|
||||
});
|
||||
});
|
||||
|
||||
it("destroys a per-turn disposable sandbox", () => {
|
||||
expect(resolveNativeSandboxLifecycle({
|
||||
adapterType: "paperclip_runner",
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
target: {
|
||||
...reusableSandbox,
|
||||
reusableLeaseConfigured: false,
|
||||
},
|
||||
})).toEqual({
|
||||
runnerProcess: "per_turn",
|
||||
sandboxResource: "destroy_after_turn",
|
||||
failoverBackup: "verified",
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects warm mode without an effective reusable-lease capability", () => {
|
||||
expect(() => resolveNativeSandboxLifecycle({
|
||||
adapterType: "paperclip_runner",
|
||||
lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 },
|
||||
target: {
|
||||
...reusableSandbox,
|
||||
effectiveCapabilities: { reusableLeases: false },
|
||||
},
|
||||
})).toThrow("runner_warm_lifecycle_requires_reusable_provider_lease");
|
||||
});
|
||||
|
||||
it("leaves legacy and non-sandbox targets untouched", () => {
|
||||
expect(resolveNativeSandboxLifecycle({
|
||||
adapterType: "codex_local",
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
target: reusableSandbox,
|
||||
})).toBeNull();
|
||||
expect(resolveNativeSandboxLifecycle({
|
||||
adapterType: "paperclip_runner",
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
|
||||
target: { kind: "local" },
|
||||
})).toBeNull();
|
||||
});
|
||||
});
|
||||
|
|
@ -232,7 +232,7 @@ describe("P6-25 pre-result native session recovery", () => {
|
|||
nextAttemptAt: new Date(0),
|
||||
failureCode: "native_session_interrupted",
|
||||
failureDetail: {
|
||||
message: "provider_initialize_timeout: provider=codex stage=health",
|
||||
message: "provider_initialize_timeout: provider=opencode stage=health",
|
||||
originalFailureCode: "provider_initialize_timeout",
|
||||
recoveryMode: "bootstrap_retry",
|
||||
providerSessionEstablished: false,
|
||||
|
|
@ -288,7 +288,7 @@ describe("P6-25 pre-result native session recovery", () => {
|
|||
await expect(db.select().from(nativeRunFinalizations)).resolves.toHaveLength(8);
|
||||
});
|
||||
|
||||
it("uses checkpoint-free bootstrap retry only when durable evidence proves no provider session existed", async () => {
|
||||
it("uses provider-neutral checkpoint-free bootstrap retry only when durable evidence proves no session existed", async () => {
|
||||
await expect(claimNativeSessionResumptions({
|
||||
db,
|
||||
runnerInstanceId: "reaper",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,171 @@
|
|||
import { EventEmitter } from "node:events";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import WebSocket from "ws";
|
||||
import type { DurablePrpControlPlane } from "../vendor/paperclip-runner/index.js";
|
||||
import {
|
||||
__runnerPrpOutboundTesting,
|
||||
connectRunnerPrpIngress,
|
||||
WsJsonWireConnection,
|
||||
} from "./runner-prp-outbound.js";
|
||||
|
||||
class FakeSocket extends EventEmitter {
|
||||
readyState: number = WebSocket.OPEN;
|
||||
send = vi.fn();
|
||||
close = vi.fn((code?: number) => {
|
||||
this.readyState = WebSocket.CLOSED;
|
||||
this.emit("close", code ?? 1000, Buffer.alloc(0));
|
||||
});
|
||||
}
|
||||
|
||||
describe("runner provider-ingress WebSocket wire", () => {
|
||||
it("bounds credential refresh by the fixed deadline and cancellation", async () => {
|
||||
const signal = new AbortController().signal;
|
||||
await expect(
|
||||
__runnerPrpOutboundTesting.awaitWithinDeadline({
|
||||
operation: async () => await new Promise<never>(() => undefined),
|
||||
deadline: Date.now() - 1,
|
||||
signal,
|
||||
}),
|
||||
).rejects.toThrow("deadline elapsed");
|
||||
|
||||
const abort = new AbortController();
|
||||
const pending = __runnerPrpOutboundTesting.awaitWithinDeadline({
|
||||
operation: async () => await new Promise<never>(() => undefined),
|
||||
deadline: Date.now() + 60_000,
|
||||
signal: abort.signal,
|
||||
});
|
||||
abort.abort();
|
||||
await expect(pending).rejects.toThrow("cancelled");
|
||||
});
|
||||
|
||||
it("delivers one terminal close to both PRP authority and reconnect ownership", () => {
|
||||
const socket = new FakeSocket();
|
||||
const wire = new WsJsonWireConnection(
|
||||
socket as unknown as WebSocket,
|
||||
);
|
||||
const authorityClose = vi.fn();
|
||||
const reconnectClose = vi.fn();
|
||||
wire.onClose(authorityClose);
|
||||
wire.onClose(reconnectClose);
|
||||
|
||||
const error = new Error("preview disconnected");
|
||||
socket.emit("error", error);
|
||||
socket.emit("close", 1006, Buffer.from("duplicate close"));
|
||||
|
||||
expect(authorityClose).toHaveBeenCalledOnce();
|
||||
expect(reconnectClose).toHaveBeenCalledOnce();
|
||||
expect(authorityClose).toHaveBeenCalledWith({
|
||||
message: "websocket_error",
|
||||
error,
|
||||
});
|
||||
});
|
||||
|
||||
it("replays a close to a listener registered after the socket ended", () => {
|
||||
const socket = new FakeSocket();
|
||||
const wire = new WsJsonWireConnection(
|
||||
socket as unknown as WebSocket,
|
||||
);
|
||||
socket.emit("close", 1001, Buffer.from("sandbox restart"));
|
||||
const listener = vi.fn();
|
||||
wire.onClose(listener);
|
||||
expect(listener).toHaveBeenCalledWith({
|
||||
code: 1001,
|
||||
message: "sandbox restart",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports a terminal ingress failure to startup and active-run ownership", async () => {
|
||||
const endpoint = {
|
||||
kind: "authenticated_websocket" as const,
|
||||
websocketUrl: "ws://preview.invalid/api/runner/v1/connect/run-1",
|
||||
secretHeaders: [],
|
||||
generation: "generation-1",
|
||||
refresh: async () => endpoint,
|
||||
close: async () => undefined,
|
||||
};
|
||||
const handle = connectRunnerPrpIngress({
|
||||
authority: {
|
||||
attachWireConnection: vi.fn(),
|
||||
activeRunnerConnectionCount: () => 0,
|
||||
} as unknown as DurablePrpControlPlane,
|
||||
endpoint,
|
||||
startupDeadlineMs: 0,
|
||||
recoveryGraceMs: 0,
|
||||
});
|
||||
|
||||
const [ready, failure] = await Promise.allSettled([
|
||||
handle.ready,
|
||||
handle.failure,
|
||||
]);
|
||||
expect(ready).toMatchObject({
|
||||
status: "rejected",
|
||||
reason: expect.objectContaining({ code: "runner_ingress_unavailable" }),
|
||||
});
|
||||
expect(failure).toMatchObject({
|
||||
status: "rejected",
|
||||
reason: expect.objectContaining({ code: "runner_ingress_unavailable" }),
|
||||
});
|
||||
await handle.close();
|
||||
});
|
||||
|
||||
it("keeps an unobserved active-run failure from becoming process-global", async () => {
|
||||
const endpoint = {
|
||||
kind: "authenticated_websocket" as const,
|
||||
websocketUrl: "ws://preview.invalid/api/runner/v1/connect/run-unobserved",
|
||||
secretHeaders: [],
|
||||
generation: "generation-1",
|
||||
refresh: async () => endpoint,
|
||||
close: async () => undefined,
|
||||
};
|
||||
const unhandled = vi.fn();
|
||||
process.on("unhandledRejection", unhandled);
|
||||
try {
|
||||
const handle = connectRunnerPrpIngress({
|
||||
authority: {
|
||||
attachWireConnection: vi.fn(),
|
||||
activeRunnerConnectionCount: () => 0,
|
||||
} as unknown as DurablePrpControlPlane,
|
||||
endpoint,
|
||||
startupDeadlineMs: 0,
|
||||
recoveryGraceMs: 0,
|
||||
});
|
||||
await expect(handle.ready).rejects.toMatchObject({
|
||||
code: "runner_ingress_unavailable",
|
||||
});
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
expect(unhandled).not.toHaveBeenCalled();
|
||||
await handle.close();
|
||||
} finally {
|
||||
process.off("unhandledRejection", unhandled);
|
||||
}
|
||||
});
|
||||
|
||||
it("does not report intentional close as an active-run ingress failure", async () => {
|
||||
const endpoint = {
|
||||
kind: "authenticated_websocket" as const,
|
||||
websocketUrl: "wss://127.0.0.1:9/api/runner/v1/connect/run-1",
|
||||
secretHeaders: [],
|
||||
generation: "generation-1",
|
||||
refresh: async () => endpoint,
|
||||
close: vi.fn(async () => undefined),
|
||||
};
|
||||
const handle = connectRunnerPrpIngress({
|
||||
authority: {
|
||||
attachWireConnection: vi.fn(),
|
||||
activeRunnerConnectionCount: () => 0,
|
||||
} as unknown as DurablePrpControlPlane,
|
||||
endpoint,
|
||||
startupDeadlineMs: 60_000,
|
||||
recoveryGraceMs: 60_000,
|
||||
});
|
||||
const failureObserver = vi.fn();
|
||||
void handle.failure.catch(failureObserver);
|
||||
void handle.ready.catch(() => undefined);
|
||||
|
||||
await handle.close();
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
expect(failureObserver).not.toHaveBeenCalled();
|
||||
expect(endpoint.close).toHaveBeenCalledOnce();
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,400 @@
|
|||
import WebSocket from "ws";
|
||||
import type { RunnerIngressEndpoint } from "@paperclipai/adapter-utils/runner-connectivity";
|
||||
import type {
|
||||
DurablePrpControlPlane,
|
||||
PrpWireConnection,
|
||||
TransportCloseReason,
|
||||
} from "../vendor/paperclip-runner/index.js";
|
||||
|
||||
const DEFAULT_STARTUP_DEADLINE_MS = 60_000;
|
||||
const DEFAULT_RECOVERY_GRACE_MS = 120_000;
|
||||
const MAX_FRAME_BYTES = 4 * 1024 * 1024;
|
||||
const RECONNECT_DELAYS_MS = [250, 500, 1_000, 2_000, 5_000] as const;
|
||||
|
||||
export type RunnerIngressFailureCode =
|
||||
| "runner_ingress_unavailable"
|
||||
| "runner_ingress_auth_failed";
|
||||
|
||||
export class RunnerIngressConnectionError extends Error {
|
||||
constructor(
|
||||
readonly code: RunnerIngressFailureCode,
|
||||
message: string,
|
||||
options?: ErrorOptions,
|
||||
) {
|
||||
super(`${code}: ${message}`, options);
|
||||
this.name = "RunnerIngressConnectionError";
|
||||
}
|
||||
}
|
||||
|
||||
export class WsJsonWireConnection implements PrpWireConnection {
|
||||
#onJson: ((value: unknown) => void) | null = null;
|
||||
#onClose = new Set<(reason: TransportCloseReason) => void>();
|
||||
#pendingJson: unknown[] = [];
|
||||
#closeReason: TransportCloseReason | null = null;
|
||||
|
||||
constructor(readonly socket: WebSocket) {
|
||||
socket.on("message", (data, isBinary) => {
|
||||
if (isBinary) {
|
||||
this.close(1003);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const value = JSON.parse(data.toString()) as unknown;
|
||||
if (this.#onJson === null) {
|
||||
this.#pendingJson.push(value);
|
||||
} else {
|
||||
this.#onJson(value);
|
||||
}
|
||||
} catch {
|
||||
this.close(1007);
|
||||
}
|
||||
});
|
||||
socket.on("close", (code, reason) => {
|
||||
this.#notifyClose({ code, message: reason.toString() });
|
||||
});
|
||||
socket.on("error", (error) => {
|
||||
this.#notifyClose({ message: "websocket_error", error });
|
||||
});
|
||||
}
|
||||
|
||||
sendJson(value: unknown): void {
|
||||
if (this.socket.readyState === WebSocket.OPEN) {
|
||||
this.socket.send(JSON.stringify(value));
|
||||
}
|
||||
}
|
||||
|
||||
close(code = 1000): void {
|
||||
if (
|
||||
this.socket.readyState === WebSocket.OPEN ||
|
||||
this.socket.readyState === WebSocket.CONNECTING
|
||||
) {
|
||||
this.socket.close(code);
|
||||
}
|
||||
}
|
||||
|
||||
onJson(listener: (value: unknown) => void): void {
|
||||
this.#onJson = listener;
|
||||
for (const value of this.#pendingJson.splice(0)) listener(value);
|
||||
}
|
||||
|
||||
onClose(listener: (reason: TransportCloseReason) => void): void {
|
||||
if (this.#closeReason) listener(this.#closeReason);
|
||||
else this.#onClose.add(listener);
|
||||
}
|
||||
|
||||
#notifyClose(reason: TransportCloseReason): void {
|
||||
if (this.#closeReason) return;
|
||||
this.#closeReason = reason;
|
||||
for (const listener of this.#onClose) listener(reason);
|
||||
this.#onClose.clear();
|
||||
}
|
||||
}
|
||||
|
||||
function validateEndpoint(endpoint: RunnerIngressEndpoint): void {
|
||||
const url = new URL(endpoint.websocketUrl);
|
||||
if (
|
||||
endpoint.kind !== "authenticated_websocket" ||
|
||||
url.protocol !== "wss:" ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
url.search ||
|
||||
url.hash
|
||||
) {
|
||||
throw new RunnerIngressConnectionError(
|
||||
"runner_ingress_unavailable",
|
||||
"The provider returned an invalid runner ingress endpoint.",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function endpointHeaders(endpoint: RunnerIngressEndpoint): Record<string, string> {
|
||||
const headers: Record<string, string> = {};
|
||||
for (const header of endpoint.secretHeaders) {
|
||||
if (!/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/.test(header.name) || !header.value) {
|
||||
throw new RunnerIngressConnectionError(
|
||||
"runner_ingress_unavailable",
|
||||
"The provider returned an invalid runner ingress credential.",
|
||||
);
|
||||
}
|
||||
headers[header.name] = header.value;
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
async function delay(ms: number, signal: AbortSignal): Promise<void> {
|
||||
if (signal.aborted) return;
|
||||
await new Promise<void>((resolve) => {
|
||||
const timer = setTimeout(resolve, ms);
|
||||
signal.addEventListener(
|
||||
"abort",
|
||||
() => {
|
||||
clearTimeout(timer);
|
||||
resolve();
|
||||
},
|
||||
{ once: true },
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
async function awaitWithinDeadline<T>(input: {
|
||||
operation: () => Promise<T>;
|
||||
deadline: number;
|
||||
signal: AbortSignal;
|
||||
}): Promise<T> {
|
||||
if (input.signal.aborted) {
|
||||
throw new Error("runner ingress operation was cancelled");
|
||||
}
|
||||
const remaining = input.deadline - Date.now();
|
||||
if (remaining <= 0) {
|
||||
throw new Error("runner ingress operation deadline elapsed");
|
||||
}
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
let onAbort: (() => void) | undefined;
|
||||
const gate = new Promise<never>((_resolve, reject) => {
|
||||
timer = setTimeout(
|
||||
() => reject(new Error("runner ingress operation deadline elapsed")),
|
||||
remaining,
|
||||
);
|
||||
onAbort = () => reject(new Error("runner ingress operation was cancelled"));
|
||||
input.signal.addEventListener("abort", onAbort, { once: true });
|
||||
});
|
||||
const operation = Promise.resolve().then(input.operation);
|
||||
try {
|
||||
return await Promise.race([operation, gate]);
|
||||
} finally {
|
||||
if (timer !== undefined) clearTimeout(timer);
|
||||
if (onAbort) input.signal.removeEventListener("abort", onAbort);
|
||||
}
|
||||
}
|
||||
|
||||
async function openEndpoint(input: {
|
||||
endpoint: RunnerIngressEndpoint;
|
||||
signal: AbortSignal;
|
||||
}): Promise<{ wire: WsJsonWireConnection; statusCode: number | null }> {
|
||||
validateEndpoint(input.endpoint);
|
||||
return await new Promise((resolve, reject) => {
|
||||
let statusCode: number | null = null;
|
||||
const socket = new WebSocket(input.endpoint.websocketUrl, {
|
||||
headers: endpointHeaders(input.endpoint),
|
||||
followRedirects: false,
|
||||
perMessageDeflate: false,
|
||||
maxPayload: MAX_FRAME_BYTES,
|
||||
handshakeTimeout: 15_000,
|
||||
});
|
||||
const wire = new WsJsonWireConnection(socket);
|
||||
const onAbort = () => {
|
||||
wire.close(1001);
|
||||
reject(
|
||||
new RunnerIngressConnectionError(
|
||||
"runner_ingress_unavailable",
|
||||
"Runner ingress connection was cancelled.",
|
||||
),
|
||||
);
|
||||
};
|
||||
input.signal.addEventListener("abort", onAbort, { once: true });
|
||||
socket.once("unexpected-response", (_request, response) => {
|
||||
statusCode = response.statusCode ?? null;
|
||||
response.resume();
|
||||
input.signal.removeEventListener("abort", onAbort);
|
||||
socket.terminate();
|
||||
reject(
|
||||
Object.assign(new Error("runner_ingress_upgrade_rejected"), {
|
||||
statusCode,
|
||||
}),
|
||||
);
|
||||
});
|
||||
socket.once("open", () => {
|
||||
input.signal.removeEventListener("abort", onAbort);
|
||||
resolve({ wire, statusCode });
|
||||
});
|
||||
socket.once("error", (error) => {
|
||||
input.signal.removeEventListener("abort", onAbort);
|
||||
reject(Object.assign(error, { statusCode }));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function socketClosed(wire: WsJsonWireConnection): Promise<void> {
|
||||
return new Promise((resolve) => wire.onClose(() => resolve()));
|
||||
}
|
||||
|
||||
async function waitForPrpReady(input: {
|
||||
isAuthenticated: () => boolean;
|
||||
wire: WsJsonWireConnection;
|
||||
deadline: number;
|
||||
signal: AbortSignal;
|
||||
}): Promise<boolean> {
|
||||
while (
|
||||
!input.signal.aborted &&
|
||||
input.wire.socket.readyState === WebSocket.OPEN &&
|
||||
Date.now() < input.deadline
|
||||
) {
|
||||
if (input.isAuthenticated()) return true;
|
||||
await delay(25, input.signal);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
export interface RunnerPrpOutboundHandle {
|
||||
readonly ready: Promise<void>;
|
||||
/** Rejects only when active-run recovery has exhausted its fixed budget. */
|
||||
readonly failure: Promise<never>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Keep a Paperclip-originated provider-ingress WebSocket attached to one PRP
|
||||
* authority. Authentication is the readiness signal; HTTP success alone is not.
|
||||
*/
|
||||
export function connectRunnerPrpIngress(input: {
|
||||
authority: DurablePrpControlPlane;
|
||||
endpoint: RunnerIngressEndpoint;
|
||||
startupDeadlineMs?: number;
|
||||
recoveryGraceMs?: number;
|
||||
random?: () => number;
|
||||
onStateChange?: (
|
||||
state: "connecting" | "authenticated" | "reconnecting" | "failed" | "closed",
|
||||
failureCode?: RunnerIngressFailureCode,
|
||||
) => void;
|
||||
}): RunnerPrpOutboundHandle {
|
||||
const abort = new AbortController();
|
||||
let activeWire: WsJsonWireConnection | null = null;
|
||||
let endpoint = input.endpoint;
|
||||
const startupDeadline =
|
||||
Date.now() + (input.startupDeadlineMs ?? DEFAULT_STARTUP_DEADLINE_MS);
|
||||
const recoveryGraceMs = input.recoveryGraceMs ?? DEFAULT_RECOVERY_GRACE_MS;
|
||||
const random = input.random ?? Math.random;
|
||||
let readyResolve!: () => void;
|
||||
let readyReject!: (error: unknown) => void;
|
||||
let readySettled = false;
|
||||
const ready = new Promise<void>((resolve, reject) => {
|
||||
readyResolve = resolve;
|
||||
readyReject = reject;
|
||||
});
|
||||
let failureReject!: (error: unknown) => void;
|
||||
const failure = new Promise<never>((_resolve, reject) => {
|
||||
failureReject = reject;
|
||||
});
|
||||
// The owning native session normally races this promise against the provider
|
||||
// turn. A turn can finish first, though, and a later preview outage must not
|
||||
// become an unhandled process-level rejection. Keep the original promise
|
||||
// rejected for explicit observers while registering an internal handler.
|
||||
void failure.catch(() => undefined);
|
||||
void ready.catch(() => undefined);
|
||||
|
||||
const loop = (async () => {
|
||||
let attempt = 0;
|
||||
let recoveryDeadline = startupDeadline;
|
||||
while (!abort.signal.aborted) {
|
||||
try {
|
||||
input.onStateChange?.(attempt === 0 ? "connecting" : "reconnecting");
|
||||
const opened = await openEndpoint({ endpoint, signal: abort.signal });
|
||||
activeWire = opened.wire;
|
||||
const attachment = input.authority.attachWireConnection(opened.wire);
|
||||
const authenticated = await waitForPrpReady({
|
||||
isAuthenticated: attachment.isAuthenticated,
|
||||
wire: opened.wire,
|
||||
deadline: recoveryDeadline,
|
||||
signal: abort.signal,
|
||||
});
|
||||
if (!authenticated) {
|
||||
opened.wire.close(1008);
|
||||
throw new RunnerIngressConnectionError(
|
||||
"runner_ingress_auth_failed",
|
||||
"Runner ingress did not complete PRP authentication before the deadline.",
|
||||
);
|
||||
}
|
||||
if (!readySettled) {
|
||||
readySettled = true;
|
||||
readyResolve();
|
||||
}
|
||||
input.onStateChange?.("authenticated");
|
||||
attempt = 0;
|
||||
await socketClosed(opened.wire);
|
||||
activeWire = null;
|
||||
recoveryDeadline = Date.now() + recoveryGraceMs;
|
||||
} catch (error) {
|
||||
activeWire = null;
|
||||
// close() owns intentional shutdown. openEndpoint rejects its pending
|
||||
// handshake when the abort signal fires, but that cancellation is not
|
||||
// an active-run transport failure and must not reject the long-lived
|
||||
// failure promise (an unobserved rejection here can terminate Node).
|
||||
if (abort.signal.aborted) return;
|
||||
let failure: unknown = error;
|
||||
const statusCode =
|
||||
typeof error === "object" && error !== null && "statusCode" in error
|
||||
? Number((error as { statusCode?: unknown }).statusCode)
|
||||
: null;
|
||||
if (
|
||||
(statusCode === 401 || statusCode === 403) &&
|
||||
Date.now() < recoveryDeadline
|
||||
) {
|
||||
try {
|
||||
const refreshedEndpoint = await awaitWithinDeadline({
|
||||
operation: () => endpoint.refresh(),
|
||||
deadline: recoveryDeadline,
|
||||
signal: abort.signal,
|
||||
});
|
||||
// Preview credentials can rotate without changing the sandbox
|
||||
// generation. Refresh every rejected credential, but never let a
|
||||
// successful refresh reset or step past the fixed recovery budget.
|
||||
if (Date.now() < recoveryDeadline) {
|
||||
endpoint = refreshedEndpoint;
|
||||
continue;
|
||||
}
|
||||
} catch (refreshError) {
|
||||
if (abort.signal.aborted) return;
|
||||
failure = refreshError;
|
||||
}
|
||||
}
|
||||
if (Date.now() >= recoveryDeadline) {
|
||||
const terminal =
|
||||
failure instanceof RunnerIngressConnectionError
|
||||
? failure
|
||||
: new RunnerIngressConnectionError(
|
||||
statusCode === 401 || statusCode === 403
|
||||
? "runner_ingress_auth_failed"
|
||||
: "runner_ingress_unavailable",
|
||||
"Runner ingress connection failed.",
|
||||
{ cause: failure },
|
||||
);
|
||||
if (!readySettled) {
|
||||
readySettled = true;
|
||||
readyReject(terminal);
|
||||
}
|
||||
input.onStateChange?.("failed", terminal.code);
|
||||
failureReject(terminal);
|
||||
return;
|
||||
}
|
||||
const base = RECONNECT_DELAYS_MS[
|
||||
Math.min(attempt, RECONNECT_DELAYS_MS.length - 1)
|
||||
]!;
|
||||
attempt += 1;
|
||||
await delay(Math.max(1, Math.round(base * (0.75 + random() * 0.5))), abort.signal);
|
||||
}
|
||||
}
|
||||
})();
|
||||
|
||||
return {
|
||||
ready,
|
||||
failure,
|
||||
async close() {
|
||||
abort.abort();
|
||||
activeWire?.close(1001);
|
||||
await loop;
|
||||
await endpoint.close();
|
||||
input.onStateChange?.("closed");
|
||||
if (!readySettled) {
|
||||
readySettled = true;
|
||||
readyReject(
|
||||
new RunnerIngressConnectionError(
|
||||
"runner_ingress_unavailable",
|
||||
"Runner ingress was closed before it became ready.",
|
||||
),
|
||||
);
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const __runnerPrpOutboundTesting = { awaitWithinDeadline };
|
||||
|
|
@ -1690,6 +1690,18 @@ export function agentRoutes(
|
|||
);
|
||||
}
|
||||
|
||||
function assertProviderTraceSettingTransition(
|
||||
req: Request,
|
||||
nextRuntimeConfig: unknown,
|
||||
previousRuntimeConfig?: unknown,
|
||||
): void {
|
||||
const previousRaw =
|
||||
asRecord(asRecord(previousRuntimeConfig)?.debug)?.providerTrace === "raw";
|
||||
const nextRaw =
|
||||
asRecord(asRecord(nextRuntimeConfig)?.debug)?.providerTrace === "raw";
|
||||
if (previousRaw !== nextRaw) assertInstanceAdmin(req);
|
||||
}
|
||||
|
||||
async function assertAgentDefaultEnvironmentSelection(
|
||||
companyId: string,
|
||||
environmentId: string | null | undefined,
|
||||
|
|
@ -1785,19 +1797,6 @@ export function agentRoutes(
|
|||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function assertCanPersistRawProviderTrace(
|
||||
req: Request,
|
||||
runtimeConfig: unknown,
|
||||
): void {
|
||||
const debug = asRecord(asRecord(runtimeConfig)?.debug);
|
||||
if (debug?.providerTrace === "raw") {
|
||||
// Raw provider payloads can contain prompts, tool inputs, and provider
|
||||
// metadata. Apply the same instance-admin boundary on every persistence
|
||||
// path so create/hire cannot bypass the PATCH guard.
|
||||
assertInstanceAdmin(req);
|
||||
}
|
||||
}
|
||||
|
||||
function asNonEmptyString(value: unknown): string | null {
|
||||
if (typeof value !== "string") return null;
|
||||
const trimmed = value.trim();
|
||||
|
|
@ -3503,6 +3502,11 @@ export function agentRoutes(
|
|||
if (!rollbackConfig) {
|
||||
throw unprocessable("Invalid revision snapshot");
|
||||
}
|
||||
assertProviderTraceSettingTransition(
|
||||
req,
|
||||
rollbackConfig.runtimeConfig,
|
||||
existing.runtimeConfig,
|
||||
);
|
||||
const rollbackAdapterType = assertKnownAdapterType(
|
||||
typeof rollbackConfig.adapterType === "string"
|
||||
? rollbackConfig.adapterType
|
||||
|
|
@ -3623,6 +3627,7 @@ export function agentRoutes(
|
|||
} = req.body;
|
||||
hireInput.adapterType = await assertSelectableAdapterType(hireInput.adapterType);
|
||||
const rawHireAdapterConfig = (hireInput.adapterConfig ?? {}) as Record<string, unknown>;
|
||||
assertProviderTraceSettingTransition(req, hireInput.runtimeConfig);
|
||||
assertFreshPaperclipRunnerProvider(
|
||||
hireInput.adapterType,
|
||||
rawHireAdapterConfig,
|
||||
|
|
@ -3633,7 +3638,6 @@ export function agentRoutes(
|
|||
);
|
||||
assertNoAgentAdapterConfigMutation(req, rawHireAdapterConfig);
|
||||
assertNoAgentRuntimeConfigAdapterConfigMutation(req, hireInput.runtimeConfig);
|
||||
assertCanPersistRawProviderTrace(req, hireInput.runtimeConfig);
|
||||
const hiredAgentId = randomUUID();
|
||||
const requestedAdapterConfig = applyCodexLocalKeyIsolation(
|
||||
companyId,
|
||||
|
|
@ -3847,6 +3851,7 @@ export function agentRoutes(
|
|||
} = req.body;
|
||||
createInput.adapterType = await assertSelectableAdapterType(createInput.adapterType);
|
||||
const rawCreateAdapterConfig = (createInput.adapterConfig ?? {}) as Record<string, unknown>;
|
||||
assertProviderTraceSettingTransition(req, createInput.runtimeConfig);
|
||||
assertFreshPaperclipRunnerProvider(
|
||||
createInput.adapterType,
|
||||
rawCreateAdapterConfig,
|
||||
|
|
@ -3857,7 +3862,6 @@ export function agentRoutes(
|
|||
);
|
||||
assertNoAgentAdapterConfigMutation(req, rawCreateAdapterConfig);
|
||||
assertNoAgentRuntimeConfigAdapterConfigMutation(req, createInput.runtimeConfig);
|
||||
assertCanPersistRawProviderTrace(req, createInput.runtimeConfig);
|
||||
const agentId = randomUUID();
|
||||
const requestedAdapterConfig = applyCodexLocalKeyIsolation(
|
||||
companyId,
|
||||
|
|
@ -4301,7 +4305,11 @@ export function agentRoutes(
|
|||
return;
|
||||
}
|
||||
assertNoAgentRuntimeConfigAdapterConfigMutation(req, runtimeConfig);
|
||||
assertCanPersistRawProviderTrace(req, runtimeConfig);
|
||||
assertProviderTraceSettingTransition(
|
||||
req,
|
||||
runtimeConfig,
|
||||
existing.runtimeConfig,
|
||||
);
|
||||
requestedRuntimeConfig = runtimeConfig;
|
||||
}
|
||||
const touchesAdapterConfiguration =
|
||||
|
|
|
|||
|
|
@ -77,6 +77,13 @@ const fakeSandboxEnvironmentConfigSchema = z.object({
|
|||
.default("ubuntu:24.04"),
|
||||
reuseLease: z.boolean().optional().default(false),
|
||||
streamRunLogs: z.boolean().optional(),
|
||||
runnerLifecycleMode: z.enum(["inherit", "per_turn", "warm"]).optional(),
|
||||
runnerIdleTimeoutMs: z.coerce
|
||||
.number()
|
||||
.int()
|
||||
.min(1_000)
|
||||
.max(86_400_000)
|
||||
.optional(),
|
||||
archiveOnRelease: z.boolean().optional(),
|
||||
}).strict();
|
||||
|
||||
|
|
@ -93,6 +100,13 @@ const pluginSandboxEnvironmentConfigSchema = z.object({
|
|||
timeoutMs: z.coerce.number().int().min(1).max(86_400_000).optional(),
|
||||
reuseLease: z.boolean().optional().default(false),
|
||||
streamRunLogs: z.boolean().optional(),
|
||||
runnerLifecycleMode: z.enum(["inherit", "per_turn", "warm"]).optional(),
|
||||
runnerIdleTimeoutMs: z.coerce
|
||||
.number()
|
||||
.int()
|
||||
.min(1_000)
|
||||
.max(86_400_000)
|
||||
.optional(),
|
||||
archiveOnRelease: z.boolean().optional(),
|
||||
}).catchall(z.unknown());
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ export interface EnvironmentDriverCapabilitySupport {
|
|||
|
||||
const NO_CAPABILITY_SUPPORT: ReadonlySet<SandboxCapabilityKey> = new Set<SandboxCapabilityKey>();
|
||||
|
||||
// The eight capability keys, written out here as literal strings. This module
|
||||
// The nine capability keys, written out here as literal strings. This module
|
||||
// does not import `environment-runtime.ts` as a value (see the module comment
|
||||
// above), so it cannot read `SANDBOX_CAPABILITY_KEYS` from there. Keep this
|
||||
// list equal to that list.
|
||||
|
|
@ -50,6 +50,7 @@ const ALL_CAPABILITY_SUPPORT: ReadonlySet<SandboxCapabilityKey> = new Set<Sandbo
|
|||
"incrementalSessionOutput",
|
||||
"concurrentSyncOperations",
|
||||
"duplexCommandStream",
|
||||
"runnerWebSocketIngress",
|
||||
]);
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { adapterSupportsRemoteManagedEnvironments } from "@paperclipai/shared";
|
|||
import {
|
||||
adapterExecutionTargetToRemoteSpec,
|
||||
type AdapterExecutionTarget,
|
||||
type SandboxLeaseAcquisition,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability";
|
||||
import {
|
||||
|
|
@ -71,6 +72,40 @@ function toBoolean(value: unknown): boolean | undefined {
|
|||
return typeof value === "boolean" ? value : undefined;
|
||||
}
|
||||
|
||||
function sandboxLeaseAcquisitionFromMetadata(
|
||||
value: unknown,
|
||||
providerLeaseId: string | null | undefined,
|
||||
): SandboxLeaseAcquisition | null {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
||||
const candidate = value as Record<string, unknown>;
|
||||
if (
|
||||
candidate.outcome !== "created" &&
|
||||
candidate.outcome !== "resumed" &&
|
||||
candidate.outcome !== "replacement"
|
||||
) return null;
|
||||
const resolvedProviderLeaseId =
|
||||
typeof candidate.providerLeaseId === "string" && candidate.providerLeaseId
|
||||
? candidate.providerLeaseId
|
||||
: providerLeaseId;
|
||||
if (!resolvedProviderLeaseId) return null;
|
||||
const reason = candidate.reason;
|
||||
if (
|
||||
reason !== undefined &&
|
||||
reason !== "not_found" &&
|
||||
reason !== "expired" &&
|
||||
reason !== "identity_mismatch" &&
|
||||
reason !== "resume_failed"
|
||||
) return null;
|
||||
return {
|
||||
outcome: candidate.outcome,
|
||||
providerLeaseId: resolvedProviderLeaseId,
|
||||
...(typeof candidate.previousProviderLeaseId === "string"
|
||||
? { previousProviderLeaseId: candidate.previousProviderLeaseId }
|
||||
: {}),
|
||||
...(reason ? { reason } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the tail of `final` that the provider did NOT already stream.
|
||||
*
|
||||
|
|
@ -331,11 +366,39 @@ export async function resolveEnvironmentExecutionTarget(input: {
|
|||
shellCommand,
|
||||
remoteCwd,
|
||||
enableSandboxDuplexBridge,
|
||||
runnerLifecyclePolicy:
|
||||
parsed.config.runnerLifecycleMode === "warm"
|
||||
? {
|
||||
mode: "warm",
|
||||
idleTimeoutMs:
|
||||
typeof parsed.config.runnerIdleTimeoutMs === "number"
|
||||
? parsed.config.runnerIdleTimeoutMs
|
||||
: 300_000,
|
||||
}
|
||||
: parsed.config.runnerLifecycleMode === "per_turn"
|
||||
? { mode: "per_turn", idleTimeoutMs: null }
|
||||
: null,
|
||||
reusableLeaseConfigured: parsed.config.reuseLease === true,
|
||||
sandboxLeaseAcquisition: sandboxLeaseAcquisitionFromMetadata(
|
||||
input.lease?.metadata?.sandboxLeaseAcquisition,
|
||||
input.lease?.providerLeaseId,
|
||||
),
|
||||
// Attach the host duplex observability recorder next to the runner. The bridge
|
||||
// binds it to the fixed observability surface. Absent keeps the no-op
|
||||
// default, so the surface stays inert on a run with no injected recorder.
|
||||
duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null,
|
||||
...(effectiveCapabilities ? { effectiveCapabilities: Object.freeze({ ...effectiveCapabilities }) } : {}),
|
||||
...(input.environmentRuntime?.getRunnerIngressEndpoint && input.lease
|
||||
? {
|
||||
getRunnerIngressEndpoint: ({ port, path }) =>
|
||||
input.environmentRuntime!.getRunnerIngressEndpoint({
|
||||
environment: input.environment as Environment,
|
||||
lease: input.lease!,
|
||||
port,
|
||||
path,
|
||||
}),
|
||||
}
|
||||
: {}),
|
||||
environmentId: input.environment.id ?? null,
|
||||
leaseId: input.leaseId ?? null,
|
||||
timeoutMs,
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ import {
|
|||
buildEnvironmentLeaseContext,
|
||||
type EnvironmentRuntimeLeaseRecord,
|
||||
type EnvironmentRuntimeService,
|
||||
type ProviderResourceDisposition,
|
||||
} from "./environment-runtime.js";
|
||||
import { ENVIRONMENT_DRIVER_TRAITS } from "./environment-driver-traits.js";
|
||||
import {
|
||||
|
|
@ -583,6 +584,17 @@ export function environmentRunOrchestrator(
|
|||
agentId: string;
|
||||
status?: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed">;
|
||||
failureReason?: string;
|
||||
/** Explicit paperclip_runner resource lifecycle. Omitted for legacy adapters. */
|
||||
providerResourceDisposition?: ProviderResourceDisposition;
|
||||
nativeLifecycleTelemetry?: {
|
||||
provider: string;
|
||||
harness: string;
|
||||
lifecycleMode: "per_turn" | "warm";
|
||||
sandboxResource:
|
||||
| "keep_running"
|
||||
| "stop_and_reuse"
|
||||
| "destroy_after_turn";
|
||||
};
|
||||
}): Promise<EnvironmentReleaseResult> {
|
||||
const status = input.status ?? "released";
|
||||
const result: EnvironmentReleaseResult = { released: [], errors: [] };
|
||||
|
|
@ -593,6 +605,7 @@ export function environmentRunOrchestrator(
|
|||
input.heartbeatRunId,
|
||||
status,
|
||||
(leaseId, error) => result.errors.push({ leaseId, error }),
|
||||
input.providerResourceDisposition,
|
||||
);
|
||||
} catch (err) {
|
||||
result.errors.push({ leaseId: "*", error: err });
|
||||
|
|
@ -621,6 +634,8 @@ export function environmentRunOrchestrator(
|
|||
status: released.lease.status,
|
||||
cleanupStatus: released.lease.cleanupStatus,
|
||||
failureReason: input.failureReason ?? released.lease.failureReason,
|
||||
providerResourceDisposition:
|
||||
input.providerResourceDisposition ?? "legacy_default",
|
||||
},
|
||||
});
|
||||
} catch {
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ import type {
|
|||
} from "@paperclipai/shared";
|
||||
import { resolveDeclaredSandboxCapabilities } from "@paperclipai/shared";
|
||||
import type { EffectiveExecutionCapabilities } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { RunnerIngressEndpoint } from "@paperclipai/adapter-utils/runner-connectivity";
|
||||
import type {
|
||||
CommandManagedDuplexChannel,
|
||||
} from "@paperclipai/adapter-utils/command-managed-runtime";
|
||||
|
|
@ -34,6 +35,7 @@ import {
|
|||
} from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
|
||||
import { environmentService } from "./environments.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
import { verifyNativeHarnessBackupStamp } from "./native-runtime/native-harness-backup-stamp.js";
|
||||
import {
|
||||
collectEnvironmentSecretRefs,
|
||||
parseEnvironmentDriverConfig,
|
||||
|
|
@ -107,6 +109,7 @@ export const SANDBOX_CAPABILITY_KEYS = [
|
|||
"incrementalSessionOutput",
|
||||
"concurrentSyncOperations",
|
||||
"duplexCommandStream",
|
||||
"runnerWebSocketIngress",
|
||||
] as const;
|
||||
|
||||
export type SandboxCapabilityKey = (typeof SANDBOX_CAPABILITY_KEYS)[number];
|
||||
|
|
@ -124,6 +127,7 @@ const SANDBOX_CAPABILITY_OPT_IN_KEYS: ReadonlySet<SandboxCapabilityKey> = new Se
|
|||
"incrementalSessionOutput",
|
||||
"concurrentSyncOperations",
|
||||
"duplexCommandStream",
|
||||
"runnerWebSocketIngress",
|
||||
]);
|
||||
|
||||
/**
|
||||
|
|
@ -180,6 +184,7 @@ const SANDBOX_CAPABILITY_PREREQUISITE_METHODS: Record<SandboxCapabilityKey, read
|
|||
incrementalSessionOutput: [["environmentExecute"]],
|
||||
concurrentSyncOperations: [["environmentSyncIn"], ["environmentSyncOut"]],
|
||||
duplexCommandStream: [["duplexChannelOpen"]],
|
||||
runnerWebSocketIngress: [["environmentRunnerIngressEndpoint"]],
|
||||
};
|
||||
|
||||
function capabilityIsVerified(
|
||||
|
|
@ -290,6 +295,7 @@ export function classifyEnvironmentCapabilities(input: {
|
|||
incrementalSessionOutput: resolve("incrementalSessionOutput"),
|
||||
concurrentSyncOperations: resolve("concurrentSyncOperations"),
|
||||
duplexCommandStream: resolve("duplexCommandStream"),
|
||||
runnerWebSocketIngress: resolve("runnerWebSocketIngress"),
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -556,6 +562,12 @@ export interface EnvironmentDriverOpenDuplexChannelInput extends EnvironmentDriv
|
|||
command: readonly string[];
|
||||
}
|
||||
|
||||
export interface EnvironmentDriverRunnerIngressInput
|
||||
extends EnvironmentDriverLeaseInput {
|
||||
port: number;
|
||||
path: string;
|
||||
}
|
||||
|
||||
export interface EnvironmentRuntimeDriver {
|
||||
readonly driver: string;
|
||||
acquireRunLease(input: EnvironmentDriverAcquireInput): Promise<EnvironmentLease>;
|
||||
|
|
@ -583,6 +595,9 @@ export interface EnvironmentRuntimeDriver {
|
|||
openDuplexChannel?(
|
||||
input: EnvironmentDriverOpenDuplexChannelInput,
|
||||
): Promise<CommandManagedDuplexChannel>;
|
||||
getRunnerIngressEndpoint?(
|
||||
input: EnvironmentDriverRunnerIngressInput,
|
||||
): Promise<RunnerIngressEndpoint>;
|
||||
/** True when the lease's plugin worker advertises both sync verbs. */
|
||||
supportsSync?(input: EnvironmentDriverLeaseInput): boolean;
|
||||
/**
|
||||
|
|
@ -678,12 +693,57 @@ export class SandboxOrphanCleanupWriteError extends Error {
|
|||
}
|
||||
}
|
||||
|
||||
/** A reusable sandbox could not be resumed, but has not been proven lost. */
|
||||
export class ReusableSandboxResumeError extends Error {
|
||||
readonly provider: string;
|
||||
readonly providerLeaseId: string;
|
||||
|
||||
constructor(input: {
|
||||
provider: string;
|
||||
providerLeaseId: string;
|
||||
cause?: unknown;
|
||||
}) {
|
||||
super(
|
||||
`Reusable sandbox lease "${input.providerLeaseId}" could not be resumed; ` +
|
||||
"the lease was preserved and no replacement was created.",
|
||||
input.cause === undefined ? undefined : { cause: input.cause },
|
||||
);
|
||||
this.name = "ReusableSandboxResumeError";
|
||||
this.provider = input.provider;
|
||||
this.providerLeaseId = input.providerLeaseId;
|
||||
}
|
||||
}
|
||||
|
||||
export class RunnerHarnessBackupUnavailableError extends Error {
|
||||
readonly providerLeaseId: string;
|
||||
|
||||
constructor(providerLeaseId: string) {
|
||||
super(
|
||||
`runner_harness_backup_unavailable: reusable sandbox "${providerLeaseId}" ` +
|
||||
"was confirmed lost, but no complete verified failover backup is available",
|
||||
);
|
||||
this.name = "RunnerHarnessBackupUnavailableError";
|
||||
this.providerLeaseId = providerLeaseId;
|
||||
}
|
||||
}
|
||||
|
||||
export interface EnvironmentRuntimeLeaseRecord {
|
||||
environment: Environment;
|
||||
lease: EnvironmentLease;
|
||||
leaseContext: ReturnType<typeof buildEnvironmentLeaseContext>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Host-side decision for the provider resource after a run. This is kept
|
||||
* separate from heartbeat status: a failed turn can still leave a reusable
|
||||
* sandbox resumable, while a disposable successful turn must destroy it.
|
||||
* An omitted disposition preserves the legacy adapter behavior.
|
||||
*/
|
||||
export type ProviderResourceDisposition =
|
||||
| "keep_running"
|
||||
| "stop_and_retain"
|
||||
| "destroy";
|
||||
|
||||
const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_TIMEOUT_MS = 5_000;
|
||||
const DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS = 100;
|
||||
|
||||
|
|
@ -704,6 +764,24 @@ function delay(ms: number): Promise<void> {
|
|||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
function transientSandboxResumeFailure(error: unknown): boolean {
|
||||
const candidate = error && typeof error === "object"
|
||||
? error as Record<string, unknown>
|
||||
: {};
|
||||
const status = typeof candidate.status === "number"
|
||||
? candidate.status
|
||||
: typeof candidate.statusCode === "number"
|
||||
? candidate.statusCode
|
||||
: null;
|
||||
if (status === 429 || (status !== null && status >= 500)) return true;
|
||||
const code = typeof candidate.code === "string" ? candidate.code.toUpperCase() : "";
|
||||
if (["ETIMEDOUT", "ECONNRESET", "ECONNREFUSED", "EAI_AGAIN", "ENETUNREACH"].includes(code)) {
|
||||
return true;
|
||||
}
|
||||
const message = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase();
|
||||
return /\b(timeout|timed out|rate limit|temporar|network|connection reset|service unavailable)\b/.test(message);
|
||||
}
|
||||
|
||||
function getLeaseDriverKey(
|
||||
lease: Pick<EnvironmentLease, "metadata">,
|
||||
environment: Pick<Environment, "driver"> | null,
|
||||
|
|
@ -1794,23 +1872,40 @@ function createSandboxEnvironmentDriver(
|
|||
: null;
|
||||
|
||||
let providerLease: PluginEnvironmentLease | null = null;
|
||||
let replacementReason:
|
||||
| "not_found"
|
||||
| "expired"
|
||||
| "identity_mismatch"
|
||||
| "resume_failed"
|
||||
| undefined;
|
||||
if (reusableLease?.providerLeaseId) {
|
||||
// The `supportsReusableLeases` check above reads a snapshot of the
|
||||
// worker methods. The runtime then does asynchronous database work
|
||||
// (list, fingerprint, obsolete-lease cleanup) before this dispatch. A
|
||||
// worker restart in that window can drop `environmentResumeLease`
|
||||
// while the snapshot still marks the method verified. Re-check the
|
||||
// live worker here and fail closed when the method is absent: skip the
|
||||
// resume, destroy the stale reusable lease, and acquire a fresh lease
|
||||
// below. The runtime never dispatches a resume the live worker cannot
|
||||
// serve.
|
||||
// live worker here and fail closed when the method is absent. The
|
||||
// runtime preserves the recorded lease and never dispatches a resume
|
||||
// the live worker cannot serve.
|
||||
const workerVerifiesResume = pluginWorkerVerifiesLifecycleMethod(
|
||||
pluginProvider.resolved.plugin.id,
|
||||
"environmentResumeLease",
|
||||
);
|
||||
if (workerVerifiesResume) {
|
||||
try {
|
||||
const resumed = await pluginWorkerManager.call(
|
||||
if (!workerVerifiesResume) {
|
||||
throw new ReusableSandboxResumeError({
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: reusableLease.providerLeaseId,
|
||||
});
|
||||
}
|
||||
try {
|
||||
const resumeDeadline = Date.now() + 60_000;
|
||||
const configuredResumeTimeoutMs =
|
||||
resolvePluginSandboxRpcTimeoutMs(workerConfig) ?? 60_000;
|
||||
let retryDelayMs = 250;
|
||||
let resumed: PluginEnvironmentLease;
|
||||
while (true) {
|
||||
try {
|
||||
resumed = await pluginWorkerManager.call(
|
||||
pluginProvider.resolved.plugin.id,
|
||||
"environmentResumeLease",
|
||||
{
|
||||
|
|
@ -1822,21 +1917,65 @@ function createSandboxEnvironmentDriver(
|
|||
providerLeaseId: reusableLease.providerLeaseId,
|
||||
leaseMetadata: reusableLease.metadata ?? undefined,
|
||||
},
|
||||
resolvePluginSandboxRpcTimeoutMs(workerConfig),
|
||||
Math.min(
|
||||
configuredResumeTimeoutMs,
|
||||
Math.max(1, resumeDeadline - Date.now()),
|
||||
),
|
||||
);
|
||||
providerLease =
|
||||
typeof resumed.providerLeaseId === "string" && resumed.providerLeaseId.length > 0
|
||||
? resumed
|
||||
: null;
|
||||
} catch {
|
||||
providerLease = null;
|
||||
break;
|
||||
} catch (error) {
|
||||
if (
|
||||
!transientSandboxResumeFailure(error) ||
|
||||
Date.now() + retryDelayMs * 1.25 >= resumeDeadline
|
||||
) throw error;
|
||||
const jitteredDelayMs = Math.max(
|
||||
1,
|
||||
Math.round(retryDelayMs * (0.75 + Math.random() * 0.5)),
|
||||
);
|
||||
await delay(jitteredDelayMs);
|
||||
retryDelayMs = Math.min(retryDelayMs * 2, 5_000);
|
||||
}
|
||||
}
|
||||
providerLease =
|
||||
typeof resumed.providerLeaseId === "string" && resumed.providerLeaseId.length > 0
|
||||
? resumed
|
||||
: null;
|
||||
if (!providerLease) {
|
||||
const sentinel = isRecord(resumed.metadata?.workspaceSentinel)
|
||||
? resumed.metadata.workspaceSentinel
|
||||
: null;
|
||||
replacementReason = sentinel && sentinel.result !== "matched"
|
||||
? "identity_mismatch"
|
||||
: resumed.metadata?.expired === true
|
||||
? "not_found"
|
||||
: "expired";
|
||||
}
|
||||
} catch (error) {
|
||||
throw new ReusableSandboxResumeError({
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: reusableLease.providerLeaseId,
|
||||
cause: error,
|
||||
});
|
||||
}
|
||||
if (!providerLease) {
|
||||
if (
|
||||
input.adapterType === "paperclip_runner" &&
|
||||
!verifyNativeHarnessBackupStamp(
|
||||
reusableLease.metadata?.nativeHarnessBackup,
|
||||
reusableLease.providerLeaseId,
|
||||
)
|
||||
) {
|
||||
throw new RunnerHarnessBackupUnavailableError(
|
||||
reusableLease.providerLeaseId,
|
||||
);
|
||||
}
|
||||
// The verified, lease-bound backup authorizes destructive
|
||||
// replacement. Keep the existing sandbox intact when validation
|
||||
// fails so the only recoverable provider state is not lost.
|
||||
await destroyReusableSandboxLease({
|
||||
environment: input.environment,
|
||||
lease: reusableLease,
|
||||
failureReason: workerVerifiesResume ? "resume_failed" : "resume_capability_lost",
|
||||
failureReason: replacementReason ?? "resume_failed",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
@ -1906,6 +2045,21 @@ function createSandboxEnvironmentDriver(
|
|||
sandboxProviderPlugin: true,
|
||||
...sandboxConfigForLeaseMetadata(storedConfig),
|
||||
...sanitizedProviderMetadata,
|
||||
sandboxLeaseAcquisition: providerLease
|
||||
? {
|
||||
outcome: "resumed",
|
||||
}
|
||||
: reusableLease?.providerLeaseId
|
||||
? {
|
||||
outcome: "replacement",
|
||||
reason: replacementReason ?? "resume_failed",
|
||||
}
|
||||
: {
|
||||
outcome: "created",
|
||||
},
|
||||
...(reusableLease?.metadata?.nativeHarnessBackup
|
||||
? { nativeHarnessBackup: reusableLease.metadata.nativeHarnessBackup }
|
||||
: {}),
|
||||
...(reusableScope ? { reusableSandboxLease: reusableScope } : {}),
|
||||
};
|
||||
try {
|
||||
|
|
@ -2072,10 +2226,10 @@ function createSandboxEnvironmentDriver(
|
|||
});
|
||||
} catch (error) {
|
||||
if (reusableLease) {
|
||||
await destroyReusableSandboxLease({
|
||||
environment: input.environment,
|
||||
lease: reusableLease,
|
||||
failureReason: "resume_failed",
|
||||
throw new ReusableSandboxResumeError({
|
||||
provider: parsed.config.provider,
|
||||
providerLeaseId: reusableLease.providerLeaseId!,
|
||||
cause: error,
|
||||
});
|
||||
}
|
||||
throw error;
|
||||
|
|
@ -2112,6 +2266,22 @@ function createSandboxEnvironmentDriver(
|
|||
driver: input.environment.driver,
|
||||
executionWorkspaceMode: input.executionWorkspaceMode,
|
||||
...providerLease.metadata,
|
||||
sandboxLeaseAcquisition:
|
||||
reusableLease && providerLease.providerLeaseId === reusableLease.providerLeaseId
|
||||
? {
|
||||
outcome: "resumed",
|
||||
}
|
||||
: reusableLease?.providerLeaseId
|
||||
? {
|
||||
outcome: "replacement",
|
||||
reason: "resume_failed",
|
||||
}
|
||||
: {
|
||||
outcome: "created",
|
||||
},
|
||||
...(reusableLease?.metadata?.nativeHarnessBackup
|
||||
? { nativeHarnessBackup: reusableLease.metadata.nativeHarnessBackup }
|
||||
: {}),
|
||||
...(reusableScope ? { reusableSandboxLease: reusableScope } : {}),
|
||||
};
|
||||
try {
|
||||
|
|
@ -2573,6 +2743,91 @@ function createSandboxEnvironmentDriver(
|
|||
return await resolveSandboxCapabilitiesForLease(input);
|
||||
},
|
||||
|
||||
async getRunnerIngressEndpoint(input) {
|
||||
if (!input.lease.metadata?.sandboxProviderPlugin || !pluginWorkerManager) {
|
||||
throw new Error("Sandbox driver does not support runner ingress for this lease.");
|
||||
}
|
||||
const pluginId = readString(input.lease.metadata.pluginId);
|
||||
const providerKey = readString(input.lease.metadata.provider);
|
||||
if (!pluginId || !providerKey || !input.lease.providerLeaseId) {
|
||||
throw new Error("Sandbox runner ingress is missing its provider identity.");
|
||||
}
|
||||
const config = await resolvePluginSandboxRuntimeConfig({
|
||||
environment: input.environment,
|
||||
lease: input.lease,
|
||||
provider: providerKey,
|
||||
});
|
||||
const sanitizedConfig = stripSandboxProviderEnvelope(
|
||||
config as SandboxEnvironmentConfig,
|
||||
);
|
||||
const acquire = async (): Promise<RunnerIngressEndpoint> => {
|
||||
const result = await pluginWorkerManager.call(
|
||||
pluginId,
|
||||
"environmentRunnerIngressEndpoint",
|
||||
{
|
||||
driverKey: providerKey,
|
||||
companyId: input.lease.companyId,
|
||||
environmentId: input.environment.id,
|
||||
issueId: input.lease.issueId,
|
||||
config: sanitizedConfig,
|
||||
lease: {
|
||||
providerLeaseId: input.lease.providerLeaseId,
|
||||
metadata: input.lease.metadata ?? undefined,
|
||||
expiresAt: input.lease.expiresAt?.toISOString() ?? null,
|
||||
},
|
||||
port: input.port,
|
||||
path: input.path,
|
||||
},
|
||||
resolvePluginSandboxRpcTimeoutMs(sanitizedConfig),
|
||||
);
|
||||
const endpointUrl = new URL(result.websocketUrl);
|
||||
if (
|
||||
result.kind !== "authenticated_websocket" ||
|
||||
endpointUrl.protocol !== "wss:" ||
|
||||
endpointUrl.username ||
|
||||
endpointUrl.password ||
|
||||
endpointUrl.search ||
|
||||
endpointUrl.hash ||
|
||||
endpointUrl.pathname !== input.path ||
|
||||
!result.generation
|
||||
) {
|
||||
throw new Error("Sandbox provider returned an invalid runner ingress endpoint.");
|
||||
}
|
||||
const secretHeaders = result.secretHeaders.map((header) => {
|
||||
if (!/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/.test(header.name) || !header.value) {
|
||||
throw new Error("Sandbox provider returned an invalid runner ingress secret header.");
|
||||
}
|
||||
const secretHeader = { name: header.name } as {
|
||||
name: string;
|
||||
readonly value: string;
|
||||
toJSON(): { name: string; value: "[REDACTED]" };
|
||||
};
|
||||
Object.defineProperty(secretHeader, "value", {
|
||||
enumerable: false,
|
||||
configurable: false,
|
||||
writable: false,
|
||||
value: header.value,
|
||||
});
|
||||
Object.defineProperty(secretHeader, "toJSON", {
|
||||
enumerable: false,
|
||||
configurable: false,
|
||||
writable: false,
|
||||
value: () => ({ name: header.name, value: "[REDACTED]" as const }),
|
||||
});
|
||||
return Object.freeze(secretHeader);
|
||||
});
|
||||
return {
|
||||
kind: "authenticated_websocket",
|
||||
websocketUrl: endpointUrl.toString(),
|
||||
secretHeaders: Object.freeze(secretHeaders),
|
||||
generation: result.generation,
|
||||
refresh: acquire,
|
||||
close: async () => undefined,
|
||||
};
|
||||
};
|
||||
return await acquire();
|
||||
},
|
||||
|
||||
async destroyRunLease(input) {
|
||||
return await destroyReusableSandboxLease({
|
||||
environment: input.environment,
|
||||
|
|
@ -2897,6 +3152,8 @@ const INTERNAL_PLUGIN_SANDBOX_CONFIG_KEYS = new Set([
|
|||
"remoteCwd",
|
||||
"shellCommand",
|
||||
"sandboxProviderPlugin",
|
||||
"sandboxLeaseAcquisition",
|
||||
"nativeHarnessBackup",
|
||||
]);
|
||||
|
||||
// Drop the host-internal and per-lease runtime keys from a sandbox config
|
||||
|
|
@ -3386,6 +3643,7 @@ export function environmentRuntimeService(
|
|||
heartbeatRunId: string,
|
||||
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed"> = "released",
|
||||
onLeaseReleaseError?: (leaseId: string, error: unknown) => void,
|
||||
providerResourceDisposition?: ProviderResourceDisposition,
|
||||
): Promise<EnvironmentRuntimeLeaseRecord[]> {
|
||||
const leaseRows = await db
|
||||
.select()
|
||||
|
|
@ -3413,14 +3671,72 @@ export function environmentRuntimeService(
|
|||
if (!environment) continue;
|
||||
|
||||
const leaseSnapshot = toEnvironmentLeaseSnapshot(leaseRow);
|
||||
if (
|
||||
providerResourceDisposition === "keep_running" &&
|
||||
leaseSnapshot.leasePolicy === "reuse_by_environment"
|
||||
) {
|
||||
const lease = await environmentsSvc.releaseLease(
|
||||
leaseRow.id,
|
||||
"retained",
|
||||
{ cleanupStatus: "success" },
|
||||
);
|
||||
if (lease) {
|
||||
released.push({
|
||||
environment,
|
||||
lease,
|
||||
leaseContext: {
|
||||
executionWorkspaceId: lease.executionWorkspaceId,
|
||||
executionWorkspaceMode:
|
||||
(lease.metadata?.executionWorkspaceMode as ExecutionWorkspace["mode"] | null | undefined) ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const driver = getDriver(getLeaseDriverKey(leaseSnapshot, environment));
|
||||
const lease = driver
|
||||
? await driver.releaseRunLease({
|
||||
if (
|
||||
providerResourceDisposition === "keep_running" &&
|
||||
leaseSnapshot.leasePolicy !== "reuse_by_environment"
|
||||
) {
|
||||
throw new Error(
|
||||
`Cannot keep non-reusable environment lease "${leaseSnapshot.id}" running.`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
providerResourceDisposition === "destroy" &&
|
||||
leaseSnapshot.metadata?.sandboxLeaseAcquisition &&
|
||||
(!leaseSnapshot.providerLeaseId ||
|
||||
!verifyNativeHarnessBackupStamp(
|
||||
leaseSnapshot.metadata.nativeHarnessBackup,
|
||||
leaseSnapshot.providerLeaseId,
|
||||
))
|
||||
) {
|
||||
throw new RunnerHarnessBackupUnavailableError(
|
||||
leaseSnapshot.providerLeaseId ?? leaseSnapshot.id,
|
||||
);
|
||||
}
|
||||
const lease = providerResourceDisposition === "destroy" && driver?.destroyRunLease
|
||||
? await driver.destroyRunLease({
|
||||
environment,
|
||||
lease: leaseSnapshot,
|
||||
status,
|
||||
failureReason: "paperclip_runner_destroy_after_turn",
|
||||
})
|
||||
: await environmentsSvc.releaseLease(leaseRow.id, status);
|
||||
: driver
|
||||
? await driver.releaseRunLease({
|
||||
environment,
|
||||
lease: leaseSnapshot,
|
||||
// A stopped reusable provider resource must remain eligible
|
||||
// for exact-lease resume independently of turn outcome.
|
||||
status:
|
||||
providerResourceDisposition === "stop_and_retain" &&
|
||||
leaseSnapshot.leasePolicy === "reuse_by_environment"
|
||||
? "released"
|
||||
: status,
|
||||
})
|
||||
: await environmentsSvc.releaseLease(
|
||||
leaseRow.id,
|
||||
providerResourceDisposition === "destroy" ? "expired" : status,
|
||||
);
|
||||
if (!lease) continue;
|
||||
|
||||
released.push({
|
||||
|
|
@ -3763,6 +4079,26 @@ export function environmentRuntimeService(
|
|||
}
|
||||
return await driver.openDuplexChannel(input);
|
||||
},
|
||||
|
||||
async getRunnerIngressEndpoint(
|
||||
input: EnvironmentDriverRunnerIngressInput,
|
||||
): Promise<RunnerIngressEndpoint> {
|
||||
const driver = requireDriverKey(
|
||||
getLeaseDriverKey(input.lease, input.environment),
|
||||
);
|
||||
const effective = await driver.resolveCapabilities(input);
|
||||
if (effective.runnerWebSocketIngress !== true) {
|
||||
throw new Error(
|
||||
"Sandbox lease does not grant runner WebSocket ingress.",
|
||||
);
|
||||
}
|
||||
if (!driver.getRunnerIngressEndpoint) {
|
||||
throw new Error(
|
||||
`Environment driver "${driver.driver}" does not support runner ingress.`,
|
||||
);
|
||||
}
|
||||
return await driver.getRunnerIngressEndpoint(input);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -392,7 +392,10 @@ import { extractSkillMentionIds, isUuidLike } from "@paperclipai/shared";
|
|||
import { evaluateCodexCredentialReadiness } from "@paperclipai/adapter-codex-local/server";
|
||||
import { environmentService } from "./environments.js";
|
||||
import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js";
|
||||
import { environmentRuntimeService } from "./environment-runtime.js";
|
||||
import {
|
||||
environmentRuntimeService,
|
||||
type ProviderResourceDisposition,
|
||||
} from "./environment-runtime.js";
|
||||
import { skillVersionSelectionMap } from "./runtime-skill-selections.js";
|
||||
import { environmentRunOrchestrator } from "./environment-run-orchestrator.js";
|
||||
import { isUnsafeSessionWorkspaceCwd } from "./session-workspace-cwd.js";
|
||||
|
|
@ -1674,6 +1677,48 @@ export function leaseReleaseStatusForRunStatus(
|
|||
return status === "failed" || status === "timed_out" ? "failed" : "released";
|
||||
}
|
||||
|
||||
export interface NativeSandboxLifecycle {
|
||||
runnerProcess: "per_turn" | "warm";
|
||||
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
||||
failoverBackup: "verified";
|
||||
}
|
||||
|
||||
export function resolveNativeSandboxLifecycle(input: {
|
||||
adapterType: string;
|
||||
lifecyclePolicy:
|
||||
| { mode: "per_turn"; idleTimeoutMs: null }
|
||||
| { mode: "warm"; idleTimeoutMs: number };
|
||||
target: {
|
||||
kind: "local" | "remote";
|
||||
transport?: string;
|
||||
reusableLeaseConfigured?: boolean;
|
||||
effectiveCapabilities?: { reusableLeases: boolean } | null;
|
||||
} | null;
|
||||
}): NativeSandboxLifecycle | null {
|
||||
if (
|
||||
input.adapterType !== "paperclip_runner" ||
|
||||
input.target?.kind !== "remote" ||
|
||||
input.target.transport !== "sandbox"
|
||||
)
|
||||
return null;
|
||||
const reusableLease =
|
||||
input.target.reusableLeaseConfigured === true &&
|
||||
input.target.effectiveCapabilities?.reusableLeases === true;
|
||||
if (input.lifecyclePolicy.mode === "warm" && !reusableLease) {
|
||||
throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease");
|
||||
}
|
||||
return {
|
||||
runnerProcess: input.lifecyclePolicy.mode,
|
||||
sandboxResource:
|
||||
input.lifecyclePolicy.mode === "warm"
|
||||
? "keep_running"
|
||||
: reusableLease
|
||||
? "stop_and_reuse"
|
||||
: "destroy_after_turn",
|
||||
failoverBackup: "verified",
|
||||
};
|
||||
}
|
||||
|
||||
export function applyPersistedExecutionWorkspaceConfig(input: {
|
||||
config: Record<string, unknown>;
|
||||
workspaceConfig: ExecutionWorkspaceConfig | null;
|
||||
|
|
@ -8611,6 +8656,13 @@ export function heartbeatService(
|
|||
agentId: string;
|
||||
status: string | null | undefined;
|
||||
failureReason?: string | null;
|
||||
providerResourceDisposition?: ProviderResourceDisposition;
|
||||
nativeLifecycleTelemetry?: {
|
||||
provider: string;
|
||||
harness: string;
|
||||
lifecycleMode: "per_turn" | "warm";
|
||||
sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
||||
};
|
||||
}) {
|
||||
const releaseResult = await envOrchestrator
|
||||
.releaseForRun({
|
||||
|
|
@ -8619,6 +8671,8 @@ export function heartbeatService(
|
|||
agentId: input.agentId,
|
||||
status: leaseReleaseStatusForRunStatus(input.status),
|
||||
failureReason: input.failureReason ?? undefined,
|
||||
providerResourceDisposition: input.providerResourceDisposition,
|
||||
nativeLifecycleTelemetry: input.nativeLifecycleTelemetry,
|
||||
})
|
||||
.catch((err) => {
|
||||
logger.warn(
|
||||
|
|
@ -17408,6 +17462,17 @@ export function heartbeatService(
|
|||
activeRunExecutions.add(run.id);
|
||||
let runScratch: HeartbeatRunScratch | null = null;
|
||||
let nativeSessionResumeScheduled = false;
|
||||
let providerResourceDispositionForRun:
|
||||
ProviderResourceDisposition | undefined;
|
||||
let nativeLifecycleTelemetryForRun:
|
||||
| {
|
||||
provider: string;
|
||||
harness: string;
|
||||
lifecycleMode: "per_turn" | "warm";
|
||||
sandboxResource:
|
||||
"keep_running" | "stop_and_reuse" | "destroy_after_turn";
|
||||
}
|
||||
| undefined;
|
||||
let providerTraceCapture: Awaited<
|
||||
ReturnType<typeof traceStore.prepare>
|
||||
> | null = null;
|
||||
|
|
@ -19879,7 +19944,26 @@ export function heartbeatService(
|
|||
: 300_000,
|
||||
}
|
||||
: { mode: "per_turn" as const, idleTimeoutMs: null };
|
||||
const effectiveLifecyclePolicy = agentLifecyclePolicy;
|
||||
const environmentLifecyclePolicy =
|
||||
executionTarget?.kind === "remote" &&
|
||||
executionTarget.transport === "sandbox"
|
||||
? executionTarget.runnerLifecyclePolicy ?? null
|
||||
: null;
|
||||
const effectiveLifecyclePolicy =
|
||||
environmentLifecyclePolicy ?? agentLifecyclePolicy;
|
||||
if (
|
||||
effectiveLifecyclePolicy.mode === "warm" &&
|
||||
executionTarget?.kind === "remote" &&
|
||||
executionTarget.transport === "sandbox" &&
|
||||
(
|
||||
executionTarget.reusableLeaseConfigured !== true ||
|
||||
executionTarget.effectiveCapabilities?.reusableLeases !== true
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
"runner_warm_environment_requires_reusable_lease",
|
||||
);
|
||||
}
|
||||
const persistedProfile = persistedRunnerProfile;
|
||||
if (persistedNativeExecutionInput) {
|
||||
nativeExecution = persistedNativeExecutionInput;
|
||||
|
|
@ -19977,10 +20061,30 @@ export function heartbeatService(
|
|||
: {},
|
||||
}
|
||||
: null,
|
||||
provider:
|
||||
nativeRuntimeResolution.profile.backend === "opencode_server"
|
||||
? "opencode"
|
||||
: nativeRuntimeResolution.profile.backend === "acpx_runtime"
|
||||
? "acpx"
|
||||
: "codex",
|
||||
...(nativeRuntimeResolution.profile.backend === "acpx_runtime"
|
||||
? {
|
||||
acpxAgent: parseObject(runtimeConfig).acpxAgent as
|
||||
"pi" | "claude" | "codex",
|
||||
}
|
||||
: {}),
|
||||
codexApprovalPolicy: resolvePaperclipRunnerPermissionMode(
|
||||
"codex",
|
||||
parseObject(agent.adapterConfig).codexPermissionMode,
|
||||
) as "never" | "on-request" | "untrusted",
|
||||
opencodePermissionMode: resolvePaperclipRunnerPermissionMode(
|
||||
"opencode",
|
||||
parseObject(runtimeConfig).opencodePermissionMode,
|
||||
) as "allow" | "ask" | "deny",
|
||||
acpxPermissionMode: resolvePaperclipRunnerPermissionMode(
|
||||
"acpx",
|
||||
parseObject(runtimeConfig).acpxPermissionMode,
|
||||
) as "approve-all" | "approve-reads" | "deny-all",
|
||||
model:
|
||||
typeof parseObject(agent.adapterConfig).model === "string"
|
||||
? String(parseObject(agent.adapterConfig).model)
|
||||
|
|
@ -20015,6 +20119,44 @@ export function heartbeatService(
|
|||
}
|
||||
}
|
||||
}
|
||||
const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({
|
||||
adapterType: agent.adapterType,
|
||||
lifecyclePolicy: nativeExecution.session.lifecyclePolicy,
|
||||
target: executionTarget,
|
||||
});
|
||||
if (nativeSandboxLifecycle) {
|
||||
nativeLifecycleTelemetryForRun = {
|
||||
provider: nativeExecution.provider.kind,
|
||||
harness: nativeExecution.session.driverKind,
|
||||
lifecycleMode: nativeExecution.session.lifecyclePolicy.mode,
|
||||
sandboxResource: nativeSandboxLifecycle.sandboxResource,
|
||||
};
|
||||
const selectedLifecycleSpan = getStartupTracer(
|
||||
"paperclip.environment-lifecycle",
|
||||
).startSpan("sandbox.lifecycle.selected", {
|
||||
attributes: {
|
||||
"paperclip.native.span.provider": nativeExecution.provider.kind,
|
||||
"paperclip.native.span.harness":
|
||||
nativeExecution.session.driverKind,
|
||||
"paperclip.native.span.lifecycle_mode":
|
||||
nativeExecution.session.lifecyclePolicy.mode,
|
||||
"paperclip.native.span.sandbox_resource":
|
||||
nativeSandboxLifecycle.sandboxResource,
|
||||
"paperclip.native.span.outcome": "selected",
|
||||
"paperclip.native.span.bytes_transferred": 0,
|
||||
},
|
||||
});
|
||||
selectedLifecycleSpan.end();
|
||||
}
|
||||
providerResourceDispositionForRun =
|
||||
nativeSandboxLifecycle?.sandboxResource === "keep_running"
|
||||
? "keep_running"
|
||||
: nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse"
|
||||
? "stop_and_retain"
|
||||
: nativeSandboxLifecycle?.sandboxResource ===
|
||||
"destroy_after_turn"
|
||||
? "destroy"
|
||||
: undefined;
|
||||
await db.transaction(async (tx) => {
|
||||
const lockedRun = await tx
|
||||
.select()
|
||||
|
|
@ -20473,6 +20615,27 @@ export function heartbeatService(
|
|||
}
|
||||
: {}),
|
||||
},
|
||||
runnerExecutionTarget: executionTarget,
|
||||
enableRunnerPreviewIngress:
|
||||
resolvedInstanceSettings.experimental
|
||||
.enableRunnerPreviewIngress === true,
|
||||
runnerPublicUrl:
|
||||
runtimeEnv.PAPERCLIP_RUNNER_PUBLIC_URL?.trim() || null,
|
||||
runnerCaBundlePath:
|
||||
runtimeEnv.PAPERCLIP_RUNNER_CA_BUNDLE_PATH?.trim() ||
|
||||
null,
|
||||
runnerRemoteBinaryPath:
|
||||
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH?.trim() ||
|
||||
null,
|
||||
runnerRemoteCodexPath:
|
||||
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH?.trim() ||
|
||||
null,
|
||||
runnerRemoteCodexNpmSpec:
|
||||
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC?.trim() ||
|
||||
null,
|
||||
runnerRemoteProviderPackPath:
|
||||
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH?.trim() ||
|
||||
null,
|
||||
enqueueWakeup,
|
||||
onSpawn: async (meta) => {
|
||||
markDispatchStarted();
|
||||
|
|
@ -21786,6 +21949,8 @@ export function heartbeatService(
|
|||
agentId: run.agentId,
|
||||
status: latestRun?.status,
|
||||
failureReason: latestRun?.error ?? undefined,
|
||||
providerResourceDisposition: providerResourceDispositionForRun,
|
||||
nativeLifecycleTelemetry: nativeLifecycleTelemetryForRun,
|
||||
});
|
||||
await releaseRuntimeServicesForRun(run.id).catch(() => undefined);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -249,6 +249,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
|||
enableWorkspaceDirtyQuarantineRepair: parsed.data.enableWorkspaceDirtyQuarantineRepair ?? true,
|
||||
enableOwnerInstanceAdmin: parsed.data.enableOwnerInstanceAdmin ?? false,
|
||||
enableSandboxDuplexBridge: parsed.data.enableSandboxDuplexBridge ?? false,
|
||||
enableRunnerPreviewIngress: parsed.data.enableRunnerPreviewIngress ?? false,
|
||||
enableWorktreeRunExecution: parsed.data.enableWorktreeRunExecution ?? false,
|
||||
worktreeRunExecutionActivatedAt: parsed.data.worktreeRunExecutionActivatedAt ?? null,
|
||||
worktreeRunExecutionActivationInstanceId:
|
||||
|
|
@ -288,6 +289,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
|||
enableWorkspaceDirtyQuarantineRepair: true,
|
||||
enableOwnerInstanceAdmin: false,
|
||||
enableSandboxDuplexBridge: false,
|
||||
enableRunnerPreviewIngress: false,
|
||||
enableWorktreeRunExecution: false,
|
||||
worktreeRunExecutionActivatedAt: null,
|
||||
worktreeRunExecutionActivationInstanceId: null,
|
||||
|
|
|
|||
|
|
@ -1,12 +1,18 @@
|
|||
import type {
|
||||
NativeAcpxAgent,
|
||||
NativeAcpxPermissionMode,
|
||||
NativeCodexApprovalPolicy,
|
||||
NativeExecutionInputV4,
|
||||
NativeInteractionResponseEnvelope,
|
||||
NativeOpenCodePermissionMode,
|
||||
NativePlanningContext,
|
||||
NativeRuntimeContextSnapshot,
|
||||
StrictCompletionContractInput,
|
||||
} from "../../vendor/paperclip-runner/index.js";
|
||||
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import {
|
||||
parseNativeExecutionInput,
|
||||
resolveQualifiedAcpxProfile,
|
||||
} from "../../vendor/paperclip-runner/index.js";
|
||||
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
|
||||
|
||||
/** Closed constructor: callers cannot spread legacy context or environment data. */
|
||||
|
|
@ -38,7 +44,11 @@ export function buildNativeExecutionInput(input: {
|
|||
branchName: string | null;
|
||||
};
|
||||
normalizedSessionId: string | null;
|
||||
provider?: "codex" | "opencode" | "acpx";
|
||||
acpxAgent?: NativeAcpxAgent;
|
||||
codexApprovalPolicy?: NativeCodexApprovalPolicy;
|
||||
opencodePermissionMode?: NativeOpenCodePermissionMode;
|
||||
acpxPermissionMode?: NativeAcpxPermissionMode;
|
||||
model?: string | null;
|
||||
lifecyclePolicy?: NativeExecutionInputV4["session"]["lifecyclePolicy"];
|
||||
executionMode?: "default" | "plan";
|
||||
|
|
@ -57,6 +67,12 @@ export function buildNativeExecutionInput(input: {
|
|||
}
|
||||
const executionMode = input.executionMode
|
||||
?? (input.issue.workMode === "planning" ? "plan" : "default");
|
||||
const acpxProfile = input.provider === "acpx"
|
||||
? resolveQualifiedAcpxProfile(
|
||||
input.acpxAgent ?? "pi",
|
||||
input.model ?? "",
|
||||
)
|
||||
: null;
|
||||
const wakePrompt = renderPaperclipWakePrompt(input.wakePayload, {
|
||||
resumedSession: input.resumedSession === true,
|
||||
suppressIssueDescription: input.taskPrompt.trim().length > 0,
|
||||
|
|
@ -90,15 +106,44 @@ export function buildNativeExecutionInput(input: {
|
|||
},
|
||||
session: {
|
||||
normalizedSessionId: input.normalizedSessionId,
|
||||
driverKind: "codex_app_server",
|
||||
driverKind: input.provider === "opencode"
|
||||
? "opencode_server"
|
||||
: input.provider === "acpx"
|
||||
? "acpx_runtime"
|
||||
: "codex_app_server",
|
||||
protocolVersion: 1,
|
||||
lifecyclePolicy: input.lifecyclePolicy ?? { mode: "per_turn", idleTimeoutMs: null },
|
||||
},
|
||||
provider: {
|
||||
kind: "codex",
|
||||
model: input.model ?? null,
|
||||
approvalPolicy: input.codexApprovalPolicy ?? "never",
|
||||
},
|
||||
provider: input.provider === "acpx"
|
||||
? {
|
||||
kind: "acpx",
|
||||
agent: acpxProfile!.agent,
|
||||
model: input.model,
|
||||
permissionMode: input.acpxPermissionMode ?? "approve-all",
|
||||
profile: {
|
||||
driverKind: acpxProfile!.driverKind,
|
||||
protocolVersion: acpxProfile!.protocolVersion,
|
||||
acpxVersion: acpxProfile!.acpxVersion,
|
||||
agent: acpxProfile!.agent,
|
||||
agentProfileVersion: acpxProfile!.agentProfileVersion,
|
||||
agentServerPackage: acpxProfile!.agentServerPackage,
|
||||
agentServerVersion: acpxProfile!.agentServerVersion,
|
||||
agentRuntimePackage: acpxProfile!.agentRuntimePackage,
|
||||
agentRuntimeVersion: acpxProfile!.agentRuntimeVersion,
|
||||
commandDigest: acpxProfile!.commandDigest,
|
||||
},
|
||||
}
|
||||
: input.provider === "opencode"
|
||||
? {
|
||||
kind: "opencode",
|
||||
model: input.model,
|
||||
permissionMode: input.opencodePermissionMode ?? "allow",
|
||||
}
|
||||
: {
|
||||
kind: "codex",
|
||||
model: input.model ?? null,
|
||||
approvalPolicy: input.codexApprovalPolicy ?? "never",
|
||||
},
|
||||
completionContract: input.completionContract,
|
||||
interactionResponses: input.interactionResponses ?? [],
|
||||
credentialBindings: [],
|
||||
|
|
|
|||
|
|
@ -0,0 +1,137 @@
|
|||
import { createHash } from "node:crypto";
|
||||
import {
|
||||
existsSync,
|
||||
lstatSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
readlinkSync,
|
||||
} from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
||||
|
||||
export interface NativeHarnessBackupStamp {
|
||||
schema: "paperclip.native-harness-backup-stamp.v1";
|
||||
normalizedSessionId: string;
|
||||
runnerInstanceId: string;
|
||||
manifestSha256: string;
|
||||
completedAt: string;
|
||||
}
|
||||
|
||||
function stateRoot(normalizedSessionId: string): string {
|
||||
return resolve(
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR
|
||||
?? resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", "durable-sessions"),
|
||||
createHash("sha256").update(normalizedSessionId).digest("hex"),
|
||||
);
|
||||
}
|
||||
|
||||
function digestDirectory(directory: string): { sha256: string; bytes: number } {
|
||||
const hash = createHash("sha256");
|
||||
let bytes = 0;
|
||||
const visit = (current: string, relative: string) => {
|
||||
const entries = readdirSync(current, { withFileTypes: true })
|
||||
.sort((left, right) => left.name.localeCompare(right.name));
|
||||
if (entries.length === 0) hash.update(`directory:${relative}\0`);
|
||||
for (const entry of entries) {
|
||||
const path = resolve(current, entry.name);
|
||||
const relativePath = relative ? `${relative}/${entry.name}` : entry.name;
|
||||
const stats = lstatSync(path);
|
||||
if (entry.isDirectory()) {
|
||||
hash.update(`directory:${relativePath}:${stats.mode & 0o777}\0`);
|
||||
visit(path, relativePath);
|
||||
} else if (entry.isSymbolicLink()) {
|
||||
hash.update(`symlink:${relativePath}:${readlinkSync(path)}\0`);
|
||||
} else if (entry.isFile()) {
|
||||
const contents = readFileSync(path);
|
||||
bytes += contents.byteLength;
|
||||
hash.update(`file:${relativePath}:${stats.mode & 0o777}:${contents.byteLength}\0`);
|
||||
hash.update(contents);
|
||||
} else {
|
||||
throw new Error(`runner_harness_backup_unsupported_entry:${relativePath}`);
|
||||
}
|
||||
}
|
||||
};
|
||||
visit(directory, "");
|
||||
return { sha256: `sha256:${hash.digest("hex")}`, bytes };
|
||||
}
|
||||
|
||||
export function createNativeHarnessBackupStamp(input: {
|
||||
manifestPath: string;
|
||||
normalizedSessionId: string;
|
||||
runnerInstanceId: string;
|
||||
completedAt: string;
|
||||
}): NativeHarnessBackupStamp {
|
||||
const manifestBytes = readFileSync(input.manifestPath);
|
||||
return {
|
||||
schema: "paperclip.native-harness-backup-stamp.v1",
|
||||
normalizedSessionId: input.normalizedSessionId,
|
||||
runnerInstanceId: input.runnerInstanceId,
|
||||
manifestSha256: `sha256:${createHash("sha256").update(manifestBytes).digest("hex")}`,
|
||||
completedAt: input.completedAt,
|
||||
};
|
||||
}
|
||||
|
||||
export function verifyNativeHarnessBackupStamp(
|
||||
value: unknown,
|
||||
expectedProviderLeaseId: string,
|
||||
): boolean {
|
||||
if (!expectedProviderLeaseId) return false;
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
|
||||
const stamp = value as Record<string, unknown>;
|
||||
if (
|
||||
stamp.schema !== "paperclip.native-harness-backup-stamp.v1" ||
|
||||
typeof stamp.normalizedSessionId !== "string" || !stamp.normalizedSessionId ||
|
||||
typeof stamp.runnerInstanceId !== "string" || !stamp.runnerInstanceId ||
|
||||
typeof stamp.manifestSha256 !== "string" || !stamp.manifestSha256.startsWith("sha256:")
|
||||
) return false;
|
||||
const backupRoot = resolve(stateRoot(stamp.normalizedSessionId), "failover-backups");
|
||||
for (const candidate of [resolve(backupRoot, "current"), resolve(backupRoot, "previous")]) {
|
||||
const manifestPath = resolve(candidate, "manifest.json");
|
||||
if (!existsSync(manifestPath)) continue;
|
||||
try {
|
||||
const bytes = readFileSync(manifestPath);
|
||||
const manifestSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
|
||||
if (manifestSha256 !== stamp.manifestSha256) continue;
|
||||
const manifest = JSON.parse(bytes.toString("utf8")) as Record<string, unknown>;
|
||||
if (
|
||||
manifest.schema !== "paperclip.native-harness-backup.v1" ||
|
||||
manifest.normalizedSessionId !== stamp.normalizedSessionId ||
|
||||
manifest.runnerInstanceId !== stamp.runnerInstanceId ||
|
||||
manifest.sourceProviderLeaseId !== expectedProviderLeaseId ||
|
||||
!Array.isArray(manifest.directories) ||
|
||||
manifest.directories.length === 0
|
||||
) continue;
|
||||
let valid = true;
|
||||
for (const entry of manifest.directories) {
|
||||
if (!entry || typeof entry !== "object" || Array.isArray(entry)) {
|
||||
valid = false;
|
||||
break;
|
||||
}
|
||||
const declared = entry as Record<string, unknown>;
|
||||
if (
|
||||
typeof declared.name !== "string" ||
|
||||
!/^[A-Za-z0-9._-]+$/.test(declared.name) ||
|
||||
typeof declared.sha256 !== "string" ||
|
||||
typeof declared.bytes !== "number"
|
||||
) {
|
||||
valid = false;
|
||||
break;
|
||||
}
|
||||
const directory = resolve(candidate, declared.name);
|
||||
if (!existsSync(directory)) {
|
||||
valid = false;
|
||||
break;
|
||||
}
|
||||
const actual = digestDirectory(directory);
|
||||
if (actual.sha256 !== declared.sha256 || actual.bytes !== declared.bytes) {
|
||||
valid = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (valid) return true;
|
||||
} catch {
|
||||
// Try the previous atomically-published backup.
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -20,6 +20,7 @@ function execution(runId: string, cwd = "/workspace") {
|
|||
agentId,
|
||||
workspace: { id: runId, cwd, repoUrl: null, repoRef: null, branchName: null },
|
||||
normalizedSessionId,
|
||||
provider: "codex",
|
||||
completionContract: {
|
||||
id: "70000000-0000-4000-8000-000000000007",
|
||||
sha256: `sha256:${"a".repeat(64)}`,
|
||||
|
|
@ -43,6 +44,7 @@ function planningExecution(runId: string, revisionId: string) {
|
|||
agentId,
|
||||
workspace: { id: runId, cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
||||
normalizedSessionId,
|
||||
provider: "codex",
|
||||
executionMode: "plan",
|
||||
planningContext: {
|
||||
documentId: "80000000-0000-4000-8000-000000000008",
|
||||
|
|
@ -189,7 +191,7 @@ describe("rebindNativeSessionCheckpoint", () => {
|
|||
});
|
||||
|
||||
describe("buildNativeExecutionInput wake projection", () => {
|
||||
it("writes native v4 and pins the complete Codex configuration", () => {
|
||||
it("writes native v4 and pins every provider's complete effective configuration", () => {
|
||||
const common = {
|
||||
companyId,
|
||||
runId: currentRunId,
|
||||
|
|
@ -203,14 +205,36 @@ describe("buildNativeExecutionInput wake projection", () => {
|
|||
} as const;
|
||||
const codex = buildNativeExecutionInput({
|
||||
...common,
|
||||
provider: "codex",
|
||||
codexApprovalPolicy: "on-request",
|
||||
});
|
||||
const opencode = buildNativeExecutionInput({
|
||||
...common,
|
||||
provider: "opencode",
|
||||
model: "openrouter/z-ai/glm-5.2",
|
||||
opencodePermissionMode: "ask",
|
||||
});
|
||||
const acpx = buildNativeExecutionInput({
|
||||
...common,
|
||||
provider: "acpx",
|
||||
acpxAgent: "claude",
|
||||
model: "claude-sonnet-5",
|
||||
acpxPermissionMode: "deny-all",
|
||||
});
|
||||
|
||||
expect(codex).toMatchObject({
|
||||
schema: "paperclip.native-execution-input.v4",
|
||||
provider: { kind: "codex", approvalPolicy: "on-request" },
|
||||
});
|
||||
expect(JSON.stringify(codex))
|
||||
expect(opencode).toMatchObject({
|
||||
schema: "paperclip.native-execution-input.v4",
|
||||
provider: { kind: "opencode", permissionMode: "ask" },
|
||||
});
|
||||
expect(acpx).toMatchObject({
|
||||
schema: "paperclip.native-execution-input.v4",
|
||||
provider: { kind: "acpx", permissionMode: "deny-all" },
|
||||
});
|
||||
expect(JSON.stringify([codex, opencode, acpx]))
|
||||
.not.toMatch(/OPENAI_API_KEY|ANTHROPIC_API_KEY|AWS_SECRET_ACCESS_KEY|PAPERCLIP_API_KEY/);
|
||||
});
|
||||
|
||||
|
|
@ -257,6 +281,8 @@ describe("buildNativeExecutionInput wake projection", () => {
|
|||
branchName: null,
|
||||
},
|
||||
normalizedSessionId,
|
||||
provider: "opencode",
|
||||
model: "openrouter/z-ai/glm-5.2",
|
||||
completionContract: {
|
||||
id: "70000000-0000-4000-8000-000000000007",
|
||||
sha256: `sha256:${"a".repeat(64)}`,
|
||||
|
|
|
|||
|
|
@ -172,6 +172,21 @@ describe("resolveNativeRuntimeMode", () => {
|
|||
}));
|
||||
});
|
||||
|
||||
it("keeps a persisted OpenCode recovery on its immutable driver", () => {
|
||||
expect(resolveHeartbeatNativeRuntimeMode({
|
||||
...eligible,
|
||||
enabled: false,
|
||||
persisted: {
|
||||
runtimeMode: "native",
|
||||
runtimeModeReason: "eligible_opt_in",
|
||||
runtimeModeResolvedAt: new Date(),
|
||||
driverKind: "opencode_server",
|
||||
},
|
||||
})).toEqual(expect.objectContaining({
|
||||
profile: { mode: "native", backend: "opencode_server", protocolVersion: 1 },
|
||||
}));
|
||||
});
|
||||
|
||||
it("rejects an explicit native profile outside the approved boundary", () => {
|
||||
expect(resolveNativeRuntimeMode({ ...eligible, agent: { ...eligible.agent, adapterType: "claude_local" } }))
|
||||
.toEqual(expect.objectContaining({ kind: "legacy", reason: "direct_adapter" }));
|
||||
|
|
@ -179,16 +194,19 @@ describe("resolveNativeRuntimeMode", () => {
|
|||
.toThrow(NativeRuntimeEligibilityError);
|
||||
});
|
||||
|
||||
it("rejects remote targets for fresh paperclip_runner starts", () => {
|
||||
expect(() => resolveNativeRuntimeMode({
|
||||
it("admits remote targets only through paperclip_runner", () => {
|
||||
expect(resolveNativeRuntimeMode({
|
||||
...eligible,
|
||||
target: { kind: "remote" },
|
||||
runtimeConfig: {},
|
||||
adapterConfig: { provider: "codex" },
|
||||
agent: { ...eligible.agent, adapterType: "paperclip_runner" },
|
||||
})).toThrow(expect.objectContaining({
|
||||
code: "paperclip_runner_environment_unsupported",
|
||||
}));
|
||||
})).toMatchObject({ kind: "native" });
|
||||
expect(resolveNativeRuntimeMode({
|
||||
...eligible,
|
||||
target: { kind: "remote" },
|
||||
agent: { ...eligible.agent, adapterType: "codex_local" },
|
||||
})).toMatchObject({ kind: "legacy", reason: "direct_adapter" });
|
||||
});
|
||||
|
||||
it("allows paperclip_runner to use a transient local workspace for projectless issues", () => {
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ export type NativeRuntimeResolution =
|
|||
reason: "eligible_opt_in";
|
||||
profile: {
|
||||
mode: "native";
|
||||
backend: "codex_app_server";
|
||||
backend: "codex_app_server" | "opencode_server" | "acpx_runtime";
|
||||
protocolVersion: 1;
|
||||
};
|
||||
authorityDecision: NativeStatusDecision;
|
||||
|
|
@ -129,12 +129,6 @@ export function resolveNativeRuntimeMode(input: {
|
|||
"Paperclip Runner requires a standard, planning, or ask task.",
|
||||
);
|
||||
}
|
||||
if (!input.target || input.target.kind !== "local") {
|
||||
throw ineligible(
|
||||
"paperclip_runner_environment_unsupported",
|
||||
"Paperclip Runner currently requires a local execution environment.",
|
||||
);
|
||||
}
|
||||
const rollout = resolveNativeMigrationStatus({
|
||||
facts: { applicationEnabled: true },
|
||||
priorIssueStatus: "in_progress",
|
||||
|
|
@ -267,13 +261,17 @@ export function resolveHeartbeatNativeRuntimeMode(input: {
|
|||
);
|
||||
}
|
||||
const driverKind = input.persisted.driverKind;
|
||||
const backend = driverKind === null
|
||||
|| driverKind === undefined
|
||||
|| driverKind === "codex"
|
||||
|| driverKind === "codex_app_server"
|
||||
? "codex_app_server"
|
||||
: null;
|
||||
if (!backend) {
|
||||
const backend = driverKind === "opencode_server"
|
||||
? "opencode_server"
|
||||
: driverKind === "acpx_runtime"
|
||||
? "acpx_runtime"
|
||||
: driverKind === null
|
||||
|| driverKind === undefined
|
||||
|| driverKind === "codex"
|
||||
|| driverKind === "codex_app_server"
|
||||
? "codex_app_server"
|
||||
: null;
|
||||
if (backend === null) {
|
||||
throw ineligible(
|
||||
"paperclip_runner_driver_unsupported",
|
||||
`Persisted Paperclip Runner driver is unsupported: ${driverKind}`,
|
||||
|
|
|
|||
|
|
@ -26,10 +26,13 @@ export type {
|
|||
ControlPlanePort,
|
||||
HarnessRuntimeRequestKind,
|
||||
HarnessRuntimeRequestResolution,
|
||||
NativeAcpxAgent,
|
||||
NativeAcpxPermissionMode,
|
||||
NativeCodexApprovalPolicy,
|
||||
NativeExecutionInput,
|
||||
NativeExecutionInputV4,
|
||||
NativeInteractionResponseEnvelope,
|
||||
NativeOpenCodePermissionMode,
|
||||
NativePlanningContext,
|
||||
NativeRunEvent,
|
||||
NativeRunResult,
|
||||
|
|
@ -45,8 +48,12 @@ export type {
|
|||
PrpStructuredRunResult,
|
||||
PrpTerminalState,
|
||||
PrpVerificationReasonCode,
|
||||
PrpWireConnection,
|
||||
ReplayControlPlaneEventsInput,
|
||||
RunnerProcessHandle,
|
||||
RunnerProcessLaunchSpec,
|
||||
StrictCompletionContractInput,
|
||||
TransportCloseReason,
|
||||
} from "@paperclipai/paperclip-runner";
|
||||
export type DurablePrpControlPlane =
|
||||
import("@paperclipai/paperclip-runner").DurablePrpControlPlane;
|
||||
|
|
@ -90,6 +97,8 @@ export const parseNativeExecutionInput = runner.parseNativeExecutionInput;
|
|||
export const parseNativeRuntimeContext = runner.parseNativeRuntimeContext;
|
||||
export const parsePaperclipQuestionSet = runner.parsePaperclipQuestionSet;
|
||||
export const parsePaperclipQuestionResponse = runner.parsePaperclipQuestionResponse;
|
||||
export const resolveQualifiedAcpxProfile = runner.resolveQualifiedAcpxProfile;
|
||||
export const resolveSourceCodexHome = runner.resolveSourceCodexHome;
|
||||
export const validatePrpEvent = runner.validatePrpEvent;
|
||||
export const validatePrpStructuredRunResult =
|
||||
runner.validatePrpStructuredRunResult;
|
||||
|
|
|
|||
|
|
@ -73,6 +73,7 @@ function defaultExperimentalSettings(): InstanceExperimentalSettingsPayload {
|
|||
return {
|
||||
enableEnvironments: false,
|
||||
enableNativeRunner: false,
|
||||
enableRunnerPreviewIngress: false,
|
||||
enableManagedSandboxOnly: false,
|
||||
enableIsolatedWorkspaces: false,
|
||||
enableStreamlinedLeftNavigation: true,
|
||||
|
|
|
|||
Loading…
Reference in New Issue