From b446ff59bfd4c22ce8042f0a8a5daad5c7adc02c Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Mon, 17 Aug 2026 22:02:16 -0700 Subject: [PATCH] refactor(acpx-engine): coordinator-owned ACP run lifecycle with a typed resource ledger (#11576) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent adapters run agent sessions through the ACPX engine > - The ACPX engine handled one run attempt as a long implicit procedure > - That shape made resource ownership, cleanup order, and failure behavior hard to verify > - This pull request gives the attempt a coordinator, a typed resource ledger, separate run sites, and explicit turn and settlement sequences > - The benefit is clear ownership, one cleanup path, safer session reuse, and testable failure behavior ## Linked Issues or Issue Description **What existing behavior does this improve?** The ACPX engine manages startup, turn execution, session reuse, and cleanup inside one large run procedure. **Current behavior** The run procedure owns several resources through implicit control flow. Cleanup and session reuse behavior depend on lane-specific branches and error paths. **Proposed behavior** The coordinator owns the run attempt. A typed ledger records six resources and their states. Host and sandbox run sites own lane-specific acquisition. Turn and settlement sequences expose typed outcomes. The engine emits allowlisted phase telemetry. **Reason and benefit** Explicit ownership makes cleanup and failure behavior easier to inspect. The fault matrix and characterization tests protect the external result while the refactor reduces hidden control flow. **Breaking changes** None to the public adapter contract. The host warm-save path now closes and relaunches the runtime because a transferred runtime could retain a run-scoped credential. A cold session-handshake failure now closes the created runtime. **Additional context** This pull request contains the ACPX engine lifecycle refactor, its tests, and the lifecycle document. ## What Changed - Add a run coordinator for startup, turn execution, settlement, and result reproduction. - Add a typed resource ledger with open, sealed, and consumed states. - Add host and sandbox run sites for lane-specific resource acquisition. - Replace separate runtime maps with a generic session reuse store. - Split session fingerprint identity from the outer session key. - Add typed turn and settlement sequences with one cleanup owner. - Add a closed allowlist for phase telemetry. - Add characterization tests and a 17-case fault matrix. - Add `doc/acp-run-lifecycle.md`. ## Verification - `npx vitest run packages/adapter-utils/src/acpx-engine/` passes 18 files and 286 tests at the submitted commit. - `pnpm --filter @paperclipai/adapter-utils typecheck` reports 0 errors at the submitted commit. - Run the full pull request checks after GitHub starts CI. - Run Greptile review after the pull request opens. ## Risks - The refactor changes internal control flow across the ACPX engine. - Host warm-save behavior now closes and relaunches the runtime. - Settlement changes the handling of a cold session-handshake failure from a leak to a close. - The characterization baselines and fault matrix reduce the risk of an external behavior change. > Paperclip is the open source app people use to manage AI agents for work > The adapter layer runs agent sessions through the ACPX engine > The engine needs explicit lifecycle ownership for reliable cleanup > This pull request adds coordinator-owned phases and a typed resource ledger > The result makes lifecycle behavior easier to test and review ## Model Used OpenAI GPT-5 Codex. Exact model ID: GPT-5. The model used tool execution, repository inspection, and code review support. The implementation author supplied the submitted code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/acp-run-lifecycle.md | 101 ++ .../composed-run-characterization.test.ts | 25 +- .../src/acpx-engine/execute-identity.test.ts | 95 + .../src/acpx-engine/execute.test.ts | 53 +- .../adapter-utils/src/acpx-engine/execute.ts | 1573 ++++++++++------- .../src/acpx-engine/run-contracts.test.ts | 85 + .../src/acpx-engine/run-contracts.ts | 509 ++++++ .../src/acpx-engine/run-coordinator.test.ts | 184 ++ .../src/acpx-engine/run-coordinator.ts | 125 ++ .../src/acpx-engine/run-fault-matrix.test.ts | 883 +++++++++ .../acpx-engine/run-resource-ledger.test.ts | 185 ++ .../src/acpx-engine/run-resource-ledger.ts | 201 +++ .../src/acpx-engine/run-site-host.test.ts | 113 ++ .../src/acpx-engine/run-site-host.ts | 146 ++ .../src/acpx-engine/run-site-sandbox.test.ts | 257 +++ .../src/acpx-engine/run-site-sandbox.ts | 459 +++++ .../acpx-engine/session-reuse-store.test.ts | 160 ++ .../src/acpx-engine/session-reuse-store.ts | 189 ++ .../settlement-characterization.test.ts | 43 +- .../acpx-engine/settlement-sequence.test.ts | 323 ++++ .../src/acpx-engine/settlement-sequence.ts | 204 +++ .../startup-characterization.test.ts | 38 +- .../src/acpx-engine/startup-timing.test.ts | 59 + .../src/acpx-engine/startup-timing.ts | 69 + .../src/acpx-engine/turn-sequence.test.ts | 190 ++ .../src/acpx-engine/turn-sequence.ts | 100 ++ packages/shared/src/telemetry/README.md | 39 + 27 files changed, 5659 insertions(+), 749 deletions(-) create mode 100644 doc/acp-run-lifecycle.md create mode 100644 packages/adapter-utils/src/acpx-engine/execute-identity.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-contracts.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-contracts.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-coordinator.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-coordinator.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-fault-matrix.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-resource-ledger.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-resource-ledger.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-site-host.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-site-host.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-site-sandbox.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/run-site-sandbox.ts create mode 100644 packages/adapter-utils/src/acpx-engine/session-reuse-store.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/session-reuse-store.ts create mode 100644 packages/adapter-utils/src/acpx-engine/settlement-sequence.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/settlement-sequence.ts create mode 100644 packages/adapter-utils/src/acpx-engine/turn-sequence.test.ts create mode 100644 packages/adapter-utils/src/acpx-engine/turn-sequence.ts diff --git a/doc/acp-run-lifecycle.md b/doc/acp-run-lifecycle.md new file mode 100644 index 0000000000..ea68b76d1c --- /dev/null +++ b/doc/acp-run-lifecycle.md @@ -0,0 +1,101 @@ +# ACP run lifecycle + +This document describes how one ACP agent run starts, runs its turn, and releases +its resources. It covers the run coordinator, the six run resources, the +settlement order, the server-owned staging lease, and the known limitations. + +## The run coordinator + +One `run()` coordinator owns one attempt. It sequences four steps in a fixed +order: + +1. **Startup.** Bring up the runtime and establish the session. +2. **Turn.** Run the agent turn against the ready runtime. +3. **Settlement.** Release every resource the run acquired. +4. **Result reproduction.** Return the external result to the caller. + +The coordinator holds the routing table as data. It reads the startup outcome and +routes on it: + +- A build failure or a partial-bridge failure throws from startup. The coordinator + replays the startup-rollback entries and rethrows the original error. +- A runtime-create, handshake, missing-handle, or configuration failure produces a + settle result. The coordinator settles the resources, then reproduces the error + result. +- Every other startup outcome produces a ready result. The coordinator runs the + turn, settles the resources, then reproduces the turn result. + +The coordinator has no catch around the turn: the turn never rejects and returns a +typed completion. The coordinator reproduces the external result AFTER settlement, +so a caller never observes a result before the run releases its resources. + +## The six run resources + +A run resource ledger holds the resources between acquisition and settlement. The +ledger is the single owner. The closed set of run resources is: + +- `acp_runtime` — the composite of the runtime, the session handle, and the child + process. `runtime.close()` is the one release boundary for all three. +- `staged_runtime` — the workspace and the runtime staged into the sandbox, plus + the one-time host-side cleanup of the staged temporary files. +- `control_bridge` — the control-plane callback bridge. +- `agent_bridge` — the agent process-session bridge. +- `managed_home` — the per-run managed-home copy-back hook. +- `staging_lease` — the per-session staging lease. + +The host lane acquires only `acp_runtime`. The sandbox lane also acquires the +staging and transport resources. Each resource enters the ledger the moment it +exists, so the settlement always closes it, even on an early failure. + +## The settlement order + +The settlement sequence is the one live cleanup owner for every settled path. It +claims the ledger once, makes the pure reuse decision, then runs the ordered +steps: + +1. `end_session` — close every runtime the reuse decision did not transfer, and + drop the warm entry when it closes. +2. `settle_reuse` — perform the decision. A save transfers the reuse candidate to + the site store; every other case discards it. +3. `stop_transport` — stop both bridges in one settled batch. +4. `sync_back` — run the site sync-back (the managed-home copy-back). +5. `release_staging_lease` — release the staging lease last (see below). + +An error policy governs every step: a step records its error, and the later steps +still run. Every step is a no-op on an empty resource slot. + +The reuse decision is pure and runs before any close. A save is eligible only when +a candidate exists, the settlement cause permits a save, and no run-scoped +credential that the candidate would carry into the store remains valid. + +## The server-owned staging lease outer context + +The staging lease is a per-session lease. Only one run of a session may stage into +the same remote workspace at a time. A second run of the same session waits on the +lease until the first run releases it. + +The lease releases as the run's final act, after the coordinator settles every +other resource AND reproduces the result. This ordering keeps a same-session +second run blocked on the lease until the first run fully returns, so the second +run never re-stages into a workspace the first run still uses. The release runs in +a `finally`, so an earlier teardown fault never strands the lease. + +## Per-phase telemetry + +The run emits one telemetry event per named lifecycle phase. Each event carries +only the phase name, the wall-time duration, and the outcome (`ok` or `failed`). +The phase name is one member of a closed allowlist. An event never carries a +command, an argument, a path, an environment value, or a raw identifier. A +telemetry failure never fails the run. + +## Known limitations and deferred work + +- **Host-lane runtime reuse is disabled.** A run-minted API key is a stateless + token that the control plane never revokes on run end. A warm host runtime would + carry a still-valid credential into the next run. The host lane therefore closes + and relaunches on every run, rather than keeping the runtime warm, until a + run-scoped credential rebind protocol exists. +- **The sandbox staged-files reuse stays enabled.** Its reuse payload carries no + credential, so a compatible resume reuses the already-staged runtime. +- **The per-phase telemetry is observability-only.** It records the duration and + the outcome of each phase; it does not change run control flow. diff --git a/packages/adapter-utils/src/acpx-engine/composed-run-characterization.test.ts b/packages/adapter-utils/src/acpx-engine/composed-run-characterization.test.ts index b9723bebc5..67e4e4df06 100644 --- a/packages/adapter-utils/src/acpx-engine/composed-run-characterization.test.ts +++ b/packages/adapter-utils/src/acpx-engine/composed-run-characterization.test.ts @@ -715,7 +715,7 @@ describe("composed ACPX run: host-lane warm handle set", () => { vi.clearAllMocks(); }); - it("warm-saves a persistent local runtime and reuses it for the next compatible run", async () => { + it("closes and relaunches a persistent local runtime, so the next compatible run re-creates it", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); let createCount = 0; @@ -746,8 +746,9 @@ describe("composed ACPX run: host-lane warm handle set", () => { onEvent: async () => {}, } as never); expect(first.exitCode).toBe(0); - // A persistent local completed turn warm-saves the runtime handle. - expect(warmHandles.size).toBe(1); + // Amendment B: a persistent local completed turn closes and relaunches the + // runtime instead of warm-saving it (the run-minted API key is never revoked). + expect(warmHandles.size).toBe(0); expect(createCount).toBe(1); const second = await execute({ @@ -761,9 +762,8 @@ describe("composed ACPX run: host-lane warm handle set", () => { onEvent: async () => {}, } as never); expect(second.exitCode).toBe(0); - // The compatible second run reuses the warm runtime, so createRuntime is not - // called again. - expect(createCount).toBe(1); + // No warm runtime survived, so the compatible second run re-creates it. + expect(createCount).toBe(2); }); it("closes (does not warm-save) a completed non-persistent runtime, so the next run re-creates", async () => { @@ -819,7 +819,7 @@ describe("composed ACPX run: host-lane warm handle set", () => { expect(createCount).toBe(2); }); - it("emits a skipped acp.handshake event on a warm-handle hit and does not re-create the runtime", async () => { + it("runs a fresh acp.handshake on the second run and re-creates the runtime (no warm reuse)", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); let createCount = 0; @@ -863,15 +863,14 @@ describe("composed ACPX run: host-lane warm handle set", () => { }, } as never); - // The warm-hit skips the handshake work: it emits exactly one acp.handshake - // event with outcome = skipped and a zero wall time, and never re-creates the - // runtime. + // Amendment B: the first run closed and relaunched, so no warm handle survives. + // The second run runs a fresh acp.handshake (outcome ok, not skipped) and + // re-creates the runtime. const handshakeEvents = secondEvents.filter( (event) => event.eventType === "run.startup.step" && event.payload?.step === "acp.handshake", ); expect(handshakeEvents).toHaveLength(1); - expect(handshakeEvents[0]!.payload?.outcome).toBe("skipped"); - expect(handshakeEvents[0]!.payload?.durationMs).toBe(0); - expect(createCount).toBe(1); + expect(handshakeEvents[0]!.payload?.outcome).not.toBe("skipped"); + expect(createCount).toBe(2); }); }); diff --git a/packages/adapter-utils/src/acpx-engine/execute-identity.test.ts b/packages/adapter-utils/src/acpx-engine/execute-identity.test.ts new file mode 100644 index 0000000000..b182d61242 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/execute-identity.test.ts @@ -0,0 +1,95 @@ +// Identity split and launch-environment finalization tests (Phase 17). +// +// Two tests are compile-time: the real assertion is that this file typechecks. +// A `@ts-expect-error` marks an assignment that MUST fail; if it starts to +// compile, tsc reports the unused directive and the typecheck fails. Each +// compile-time assertion lives in a declared, unexecuted function, so vitest +// never runs a synthetic value. + +import { describe, expect, it } from "vitest"; +import { buildSessionFingerprint, finalizeLaunchEnvironment } from "./execute.js"; +import type { + LaunchEnvironment, + RunScopedContribution, + RunSiteReuseCandidate, + SessionFingerprintIdentity, +} from "./run-contracts.js"; + +// A complete fingerprint identity with representative values for the 17 fields. +const SAMPLE_FINGERPRINT_IDENTITY: SessionFingerprintIdentity = { + acpxAgent: "claude", + agentCommand: "claude", + cwd: "/work", + mode: "persistent", + permissionMode: "approve-all", + nonInteractivePermissions: "deny", + requestedModel: "", + requestedThinkingEffort: "", + fastMode: false, + remoteExecutionIdentity: null, + additionalSourcesIdentity: {}, + skillsIdentity: {}, + skillPromptInstructions: "", + paperclipClaudeSettings: null, + mcpServers: [], + secretManifestHash: "0000", + adapterEnvHash: "0000", +}; + +describe("acpx identity split and launch environment", () => { + it("test_fingerprint_builder_accepts_only_session_fingerprint_identity", () => { + // The builder accepts a fingerprint identity and returns a stable hash. + const first = buildSessionFingerprint(SAMPLE_FINGERPRINT_IDENTITY); + const second = buildSessionFingerprint(SAMPLE_FINGERPRINT_IDENTITY); + expect(first).toBe(second); + expect(first).toMatch(/^[0-9a-f]{16}$/); + + // The compile-time assertions live in an unexecuted function. + function _assertOnlyIdentity(): void { + // An outer-key field is not a fingerprint field, so the builder rejects it. + // @ts-expect-error companyId is not a SessionFingerprintIdentity field. + buildSessionFingerprint({ ...SAMPLE_FINGERPRINT_IDENTITY, companyId: "c" }); + // @ts-expect-error taskKey is not a SessionFingerprintIdentity field. + buildSessionFingerprint({ ...SAMPLE_FINGERPRINT_IDENTITY, taskKey: "t" }); + } + void _assertOnlyIdentity; + }); + + it("test_finalize_launch_environment_is_sole_constructor_of_branded_environment", () => { + // The finalizer merges every contribution and freezes the launch env. + const baseEnv: Record = { LAUNCH_ENV_TEST_BASE: "base" }; + const launchEnvironment = finalizeLaunchEnvironment(baseEnv, [ + { scope: "session", env: { LAUNCH_ENV_TEST_CONTRIB: "contrib" } }, + ]); + expect(launchEnvironment.env.LAUNCH_ENV_TEST_BASE).toBe("base"); + expect(launchEnvironment.env.LAUNCH_ENV_TEST_CONTRIB).toBe("contrib"); + expect(Object.isFrozen(launchEnvironment.env)).toBe(true); + + // No plain object literal can stand in for the branded environment; it lacks + // the private brand that only `finalizeLaunchEnvironment` sets. + function _assertBrand(): void { + // @ts-expect-error missing the private launch-environment brand. + const _fake: LaunchEnvironment = { env: {} }; + void _fake; + } + void _assertBrand; + }); + + it("test_run_scoped_contribution_is_not_assignable_to_a_reuse_payload", () => { + // A run-scoped contribution and the branded launch environment must never + // enter a reuse payload (Amendment B). The reuse candidate types reject both. + function _assertNotReusable( + contribution: RunScopedContribution, + launchEnvironment: LaunchEnvironment, + ): void { + // @ts-expect-error a run-scoped contribution is not a reuse candidate. + const _a: RunSiteReuseCandidate = contribution; + // @ts-expect-error the branded launch environment is not a reuse candidate. + const _b: RunSiteReuseCandidate = launchEnvironment; + void _a; + void _b; + } + void _assertNotReusable; + expect(true).toBe(true); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts index 35d89fccda..a61510c70b 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.test.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts @@ -336,7 +336,7 @@ const ALLOWED_TURN_SPAN_ATTRIBUTE_KEYS = new Set([ ]); describe("shared ACPX engine runtime behavior", () => { - it("persists ACP agent process identity before prompting and reuses it for the next warm heartbeat", async () => { + it("persists ACP agent process identity before prompting on each run (host lane re-creates, no warm reuse)", async () => { const root = await makeTempRoot(); const startedAt = "2026-07-30T07:00:00.000Z"; const processPid = 43_210; @@ -419,7 +419,9 @@ describe("shared ACPX engine runtime behavior", () => { } as never); expect(second.exitCode).toBe(0); - expect(runtimeCreateCount).toBe(1); + // Amendment B: the host lane closes and relaunches (no warm reuse), so the + // second run re-creates the runtime and re-persists the process identity. + expect(runtimeCreateCount).toBe(2); expect(secondOnSpawn).toHaveBeenCalledOnce(); expect(turnStartedBeforeProcessIdentity).toBe(false); }); @@ -4382,7 +4384,7 @@ describe("ACPX engine per-step startup timing (run.startup.step events)", () => } }); - it("emits a skipped acp.handshake event when a warm-handle hit skips the handshake", async () => { + it("runs a fresh acp.handshake on a compatible second run (host lane re-creates, no warm hit)", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); const warmHandles = new Map(); @@ -4408,9 +4410,9 @@ describe("ACPX engine per-step startup timing (run.startup.step events)", () => onMeta: async () => {}, onEvent: async () => {}, } as never); - // The second run reuses the warm handle, so the handshake does no work. It - // must emit exactly one acp.handshake event with outcome = skipped and a - // zero wall time, not a misleading zero-work `ok` event. + // Amendment B: the first run closed and relaunched (no warm save), so the + // compatible second run re-creates the runtime and runs a fresh handshake — one + // acp.handshake event whose outcome is not `skipped`. await execute({ runId: "run-warm-2", agent: { id: "agent-1", companyId: "company-1" }, @@ -4428,8 +4430,7 @@ describe("ACPX engine per-step startup timing (run.startup.step events)", () => (event) => event.eventType === "run.startup.step" && event.payload?.step === "acp.handshake", ); expect(handshakeEvents).toHaveLength(1); - expect(handshakeEvents[0]!.payload?.outcome).toBe("skipped"); - expect(handshakeEvents[0]!.payload?.durationMs).toBe(0); + expect(handshakeEvents[0]!.payload?.outcome).not.toBe("skipped"); }); it("does not emit startup-step events on a local (non-sandbox) run except workspace.resolve", async () => { @@ -4663,7 +4664,7 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every runtimeSessionName: "runtime-session", }; - it("test_handshake_failure_closes_runtime_and_removes_warm_entry", async () => { + it("test_persistent_host_failure_closes_recreated_runtime_no_warm_reuse", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); const closeSpy = vi.fn(async () => {}); @@ -4709,10 +4710,13 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every onSpawn: async () => {}, } as never); expect(first.exitCode).toBe(0); - expect(warmHandles.size).toBe(1); + // Amendment B: the first clean persistent turn closes and relaunches instead of + // warm-saving, so no warm handle survives. + expect(warmHandles.size).toBe(0); expect(created).toBe(1); - // The warm-hit reuses runtime #1 and fails while persisting process identity. + // The second run finds no warm handle, so it re-creates the runtime and fails + // while persisting process identity. const second = await execute({ runId: "warm-handshake-2", agent: { id: "agent-1", companyId: "company-1" }, @@ -4726,11 +4730,11 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every }, } as never); - expect(created).toBe(1); + expect(created).toBe(2); expect(second.exitCode).toBe(1); expect(second.resultJson?.phase).toBe("ensure_session"); - // The reused runtime is closed and the warm entry removed. - expect(closeSpy).toHaveBeenCalledTimes(1); + // Each run closed its own runtime (one per run); no warm entry survived. + expect(closeSpy).toHaveBeenCalledTimes(2); expect(warmHandles.size).toBe(0); }); @@ -4768,7 +4772,7 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every expect(closeSpy).toHaveBeenCalledTimes(1); }); - it("test_warm_hit_failure_closes_runtime_and_does_not_rearm_idle_timer", async () => { + it("test_host_lane_closes_and_relaunches_every_run_and_leaves_no_warm_entry", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); let created = 0; @@ -4816,9 +4820,13 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every onSpawn: async () => {}, } as never); expect(first.exitCode).toBe(0); - expect(warmHandles.size).toBe(1); + // Amendment B: the first clean persistent turn closes runtime #1 and leaves no + // warm entry, so the host lane never keeps a live runtime warm. + expect(warmHandles.size).toBe(0); + expect(closes).toEqual([1]); - // A warm-hit whose identity-persist step fails reuses runtime #1 (no new create). + // The second run finds no warm entry, so it re-creates runtime #2 and fails + // while persisting process identity; runtime #2 closes. const second = await execute({ runId: "warm-timer-2", agent: { id: "agent-1", companyId: "company-1" }, @@ -4831,15 +4839,12 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every throw new Error("onSpawn boom"); }, } as never); - expect(created).toBe(1); + expect(created).toBe(2); expect(second.exitCode).toBe(1); - // Runtime #1 is closed and its warm entry removed. The warm-hit already cleared - // the idle timer, and the failure does not re-arm it, so the map is empty. - expect(closes).toEqual([1]); + expect(closes).toEqual([1, 2]); expect(warmHandles.size).toBe(0); - // A later heartbeat finds no warm entry and constructs a fresh runtime, proving - // the failed warm-hit did not leave a re-armed entry behind. + // A later heartbeat also finds no warm entry and constructs a fresh runtime. const third = await execute({ runId: "warm-timer-3", agent: { id: "agent-1", companyId: "company-1" }, @@ -4851,7 +4856,7 @@ describe("ACPX engine run lifecycle corrections (F2: close the runtime for every onSpawn: async () => {}, } as never); expect(third.exitCode).toBe(0); - expect(created).toBe(2); + expect(created).toBe(3); }); }); diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts index 99f1e8a02e..47118fbf75 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.ts @@ -84,8 +84,43 @@ import { DEFAULT_ACP_ENGINE_TIMEOUT_SEC, DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS, } from "./constants.js"; +import type { + AcpRunContext, + AcquiredRunResources, + LaunchEnvironment, + LaunchEnvironmentContribution, + PreTurnFailedCause, + SessionFingerprintIdentity, + SessionKeyIdentity, + SettlementCause, + StartupReady, + StartupResult, + TurnCompletion, +} from "./run-contracts.js"; +import { createRunResourceLedger } from "./run-resource-ledger.js"; +import { settleAcpRun, type SettlementSteps } from "./settlement-sequence.js"; +import { + runAttempt, + type RunPlan, + type SettlementReason, + type SettlementDispositionReport, +} from "./run-coordinator.js"; +import { + runTurn as runTurnSequence, + type StartedTurn, + type TurnFinalizeInput, +} from "./turn-sequence.js"; +import { + createHostRunSite, + type AcpxAgentProcessIdentity, + type AcpxProcessIdentitySink, + type ChildStderrState, + type RuntimeCacheEntry, +} from "./run-site-host.js"; +import { createSandboxRunSite, type SandboxRunSite } from "./run-site-sandbox.js"; import { createRuntimeSpanRunner, + emitRunPhaseTiming, emitSkippedStartupStep, getActiveStepContext, measureStartupStep, @@ -105,11 +140,6 @@ const defaultModuleDir = path.dirname(fileURLToPath(import.meta.url)); const PAPERCLIP_MANAGED_CODEX_SKILLS_MANIFEST = ".paperclip-managed-skills.json"; const BENIGN_NES_CLOSE_STDERR = /method: ['"]nes\/close['"].*-32601/; -interface ChildStderrState { - logPath: string | null; - pendingLiveLine: string; -} - function routeChildStderr(state: ChildStderrState, chunk: string) { if (state.logPath) { fsSync.mkdirSync(path.dirname(state.logPath), { recursive: true }); @@ -137,8 +167,6 @@ function flushChildStderr(state: ChildStderrState) { state.pendingLiveLine = ""; } -type AcpxAgentProcessIdentity = { pid: number; startedAt: string }; - type PaperclipAcpRuntimeOptions = AcpRuntimeOptions & { onAgentSpawn?: (meta: AcpxAgentProcessIdentity) => Promise; // Return the current-run parent-context token. It is the `task.run` token @@ -148,23 +176,8 @@ type PaperclipAcpRuntimeOptions = AcpRuntimeOptions & { getRuntimeParentContext?: () => StartupSpanContext | undefined; }; -type AcpxProcessIdentitySink = { - current: AdapterExecutionContext["onSpawn"]; - latest: AcpxAgentProcessIdentity | null; -}; - type AcpxRuntimeFactory = (options: PaperclipAcpRuntimeOptions) => AcpRuntime; -export interface RuntimeCacheEntry { - runtime: AcpRuntime; - handle: AcpRuntimeHandle; - childStderrState: ChildStderrState; - processIdentitySink: AcpxProcessIdentitySink; - fingerprint: string; - lastUsedAt: number; - cleanupTimer?: NodeJS.Timeout; -} - /** * A remote runner-backed session's staged runtime, kept warm across runs so a * compatible resume reuses it instead of re-shipping the workspace / re-seeding @@ -348,6 +361,14 @@ export interface AcpxEngineExecutorOptions { prepareRemoteManagedHome?: ( input: AcpxRemoteManagedHomeContext, ) => Promise; + /** + * Observe the final per-resource disposition report the run records at the end + * of the attempt (`finalized` vs `transferred`). The coordinator records it on + * every exit path: the settlement path reports what the settlement decided, and + * the startup-rollback path reports every rolled-back entry as `finalized`. The + * engine never reads it back; a test injects this hook to assert the report. + */ + onSettlementDisposition?: (report: SettlementDispositionReport) => void; } interface AcpxPreparedRuntime { @@ -388,8 +409,8 @@ interface AcpxPreparedRuntime { stagedRuntime: PreparedAdapterExecutionTargetRuntime | null; // Per-run copy-back hook from the per-adapter remote managed-home seam: runs // the codex auth copy-back (via `restoreWorkspace()`). Invoked once on every - // exit path by `cleanupRemoteBridges`; it never removes staged temp, so it is - // safe on every compatible resume. Null for local runs, the runner-less + // exit path by the settlement `syncBack` step; it never removes staged temp, so + // it is safe on every compatible resume. Null for local runs, the runner-less // fallback, and adapters with no seam. remoteManagedHomeTeardown: (() => Promise) | null; // One-time host-side staged-resource cleanup from the seam (remove staged temp @@ -451,6 +472,46 @@ function shortHash(value: unknown): string { return createHash("sha256").update(stableJson(value)).digest("hex").slice(0, 16); } +/** + * Hash the session fingerprint. The builder accepts only a + * `SessionFingerprintIdentity`, so no company, agent, or task identifier can + * enter the hash. Those identifiers scope the outer session key only (see + * `SessionKeyIdentity`). + */ +export function buildSessionFingerprint(identity: SessionFingerprintIdentity): string { + return shortHash(identity); +} + +/** + * Build the session key from the fingerprint and the outer-key identity. The key + * form is `paperclip:companyId:agentId:taskKey:fingerprint`. + */ +export function buildSessionKey(identity: SessionKeyIdentity, fingerprint: string): string { + return `paperclip:${identity.companyId}:${identity.agentId}:${identity.taskKey}:${fingerprint}`; +} + +/** + * Build the single branded launch environment for a run. This is the sole + * constructor of `LaunchEnvironment`. It applies each contribution into the base + * env in registration order, then resolves and freezes the launch env. + * + * A contribution carries its credential scope. A run-scoped contribution (the + * run API key, a bridge token) and a session-scoped contribution (the Codex auth + * copy-back material) both merge here. Neither scope can escape into a reuse + * payload, because the branded contribution type and the branded environment + * type forbid it. + */ +export function finalizeLaunchEnvironment( + baseEnv: Record, + contributions: readonly LaunchEnvironmentContribution[], +): LaunchEnvironment { + for (const contribution of contributions) { + Object.assign(baseEnv, contribution.env); + } + const env = Object.freeze(resolveRuntimeEnv(baseEnv)); + return { env } as unknown as LaunchEnvironment; +} + // Directory names the staging path never ships for a referenced project (heavy // build/cache output and git history). The content signature skips them so it // reflects only the staged tree and never reads their bytes. Keep this set equal @@ -1405,6 +1466,14 @@ async function buildRuntime(input: { ctx: AdapterExecutionContext; engine: AcpxEngineSettings; deps: AcpxEngineExecutorOptions; + // The run resource ledger. `executeAcpxEngine` creates the one ledger for the + // attempt and passes it here, so the sandbox run site registers what it + // acquires into that single ledger (one ledger until Phase 20 formalizes the + // coordinator). + ledger: AcquiredRunResources; + // The staged-runtime idle bound, in milliseconds. The sandbox run site takes + // it for its reuse store's idle policy. + stagedIdleMs: number; // The injected tracer, the root-span parent-context token, and the // context-builder. Merged into every startup-step option set, so each // boundary span parents to the one root span (`sandbox.startup`) that the @@ -1777,7 +1846,10 @@ async function buildRuntime(input: { useRemoteProcessSession && executionTarget?.kind === "remote" ? executionTarget.remoteCwd : cwd; - const fingerprint = shortHash({ + // The 17 fields the session fingerprint hashes. Company, agent, and task + // identifiers are NOT here; they scope the outer session key only (see + // `keyIdentity`). The fingerprint builder accepts only this identity. + const fingerprintIdentity: SessionFingerprintIdentity = { acpxAgent, agentCommand: agentCommand ?? acpxAgent, cwd: path.resolve(sessionCwd), @@ -1792,7 +1864,7 @@ async function buildRuntime(input: { // added, removed, or re-pinned) invalidates a warm/resumable session so the // next launch stages the current referenced-project trees instead of reusing // a stale staged tree. - additionalSourcesIdentity, + additionalSourcesIdentity: additionalSourcesIdentity as unknown as Record, skillsIdentity, skillPromptInstructions, paperclipClaudeSettings: paperclipClaudeSettings @@ -1812,9 +1884,17 @@ async function buildRuntime(input: { // edits and same-version secret rotations. Per-wake runtime vars never enter // resolvedAdapterEnv, so they don't churn the fingerprint every heartbeat. adapterEnvHash: shortHash(resolvedAdapterEnv), - }); + }; + const fingerprint = buildSessionFingerprint(fingerprintIdentity); const taskKey = asString(input.ctx.runtime.taskKey, "") || wakeTaskId || workspaceId || "default"; - const sessionKey = `paperclip:${agent.companyId}:${agent.id}:${taskKey}:${fingerprint}`; + // Company, agent, and task identity for the outer session key. These parts + // stay out of the fingerprint hash. + const keyIdentity: SessionKeyIdentity = { + companyId: agent.companyId, + agentId: agent.id, + taskKey, + }; + const sessionKey = buildSessionKey(keyIdentity, fingerprint); // Ship the workspace into the sandbox and capture `{ workspaceRemoteDir, // runtimeRootDir, assetDirs, restoreWorkspace }`. Done once here, before the @@ -1866,58 +1946,28 @@ async function buildRuntime(input: { let remoteStagingDispose: (() => Promise) | null = null; let remoteStagingEnvDelta: Record | null = null; let sessionStagingLeaseRelease: (() => void) | null = null; + // The sandbox run site owns the sandbox lane's staging, the per-session + // staging lease, both host-side bridges, the launch-environment contribution, + // and sync-back. `buildRuntime` injects the leaf primitives (the workspace + // stage call, the managed-home seam, the two bridge starts, the step timers, + // and the launch-env finalizer) and reads the site's results back onto the + // prepared-runtime fields, so the existing settlement path stays unchanged. + let sandboxSite: SandboxRunSite | null = null; if (useRemoteProcessSession && executionTarget?.kind === "remote") { const remoteTarget = executionTarget; - const staged = await withSessionStagingLease(stagingLocks, sessionKey, async (): Promise<{ - stagedRuntime: PreparedAdapterExecutionTargetRuntime; - teardown: (() => Promise) | null; - dispose: (() => Promise) | null; - envDelta: Record; - }> => { - const cachedStaged = isCompatibleResume ? stagedRuntimes.get(sessionKey) : undefined; - if (cachedStaged) { - // Reuse the already-staged in-sandbox workspace + managed home. Re-apply - // the env keys the seam repointed onto the in-sandbox home (deterministic, - // identical across the session's runs) and reuse the seam's per-run - // copy-back so the codex auth copy-back still fires on THIS run's teardown - // — the copy-back cadence stays exactly per-run, unchanged from PR 2. The - // copy-back reads the sandbox auth.json live at teardown, so the reused - // closure copies back the current credential, never a stale snapshot, and - // it never removes the staged in-sandbox home (host staged-temp cleanup - // moved to `dispose`, fired only when the entry is dropped), so reusing it - // can't leave this run without its staged home. - // (The workspace restore in that same closure diffs against the ORIGINAL - // staging run's host baseline — an accepted consequence of "reuse, don't - // re-ship": the in-sandbox workspace is the source of truth mid-session - // and the host stays synced from it each run.) - Object.assign(env, cachedStaged.envDelta); - cachedStaged.lastUsedAt = nowMs(); - await input.ctx.onLog( - "stdout", - "[paperclip] Reusing the staged in-sandbox runtime for this resumed session (no workspace re-ship / managed-home re-seed).\n", - ); - return { - stagedRuntime: cachedStaged.stagedRuntime, - teardown: cachedStaged.teardown, - dispose: cachedStaged.dispose, - envDelta: cachedStaged.envDelta, - }; - } - // Not a compatible resume (or no cache entry): stage fresh. If a stale - // entry sits at this key (e.g. an incompatible new session colliding on - // company/agent/task/fingerprint), drop it and release its host staged - // resources first so we neither reuse nor leak it. - const stale = stagedRuntimes.get(sessionKey); - if (stale) { - stagedRuntimes.delete(sessionKey); - if (stale.dispose) await stale.dispose().catch(() => {}); - } - // Record each successful `stage()` result before the seam can fail. A seam - // that throws AFTER a successful stage never returns its `disposeStaged`, so - // the engine disposes the fresh staged runtime here (see the catch below). - let freshlyStagedRuntime: PreparedAdapterExecutionTargetRuntime | null = null; - const stage = async (assets: AdapterManagedRuntimeAsset[]) => { - const staged = await stageAcpRemoteRuntime({ + sandboxSite = createSandboxRunSite({ + ledger: input.ledger, + stagedRuntimes, + stagingLocks, + now: nowMs, + idleMs: input.stagedIdleMs, + sessionCwd, + spawnCwd: cwd, + target: remoteTarget, + env, + isCompatibleResume, + stage: (assets) => + stageAcpRemoteRuntime({ runId, target: remoteTarget, adapterKey: input.engine.adapterType, @@ -1929,33 +1979,10 @@ async function buildRuntime(input: { onLog: input.ctx.onLog, onRuntimeProgress: input.ctx.onRuntimeProgress, runtimeSpan: input.stageRuntimeSpan, - }); - freshlyStagedRuntime = staged; - return staged; - }; - // Snapshot env before the seam so we can capture exactly which keys it - // repointed onto the in-sandbox home (e.g. `CODEX_HOME`) and replay them - // verbatim on a later compatible resume. Add/change only — every seam sets - // (never deletes) its home env var, so a set-based delta is complete. - const envBeforeStage = { ...env }; - // Step 4 — stage.sync: ship the workspace (and, via the seam, the managed - // home) into the sandbox. Only fires on a fresh stage; a compatible resume - // that reuses an already-staged runtime skips this block entirely. The - // measured callback returns the staged result so the timing wrap does not - // disturb definite-assignment of the outer bindings. - const { - stagedRuntime: freshStagedRuntime, - teardown: freshTeardown, - dispose: freshDispose, - } = await measureStartupStep(input.ctx, nowMs, "stage.sync", async (): Promise<{ - stagedRuntime: PreparedAdapterExecutionTargetRuntime; - teardown: (() => Promise) | null; - dispose: (() => Promise) | null; - }> => { - if (input.deps.prepareRemoteManagedHome) { - let seeded: AcpxRemoteManagedHomeResult; - try { - seeded = await input.deps.prepareRemoteManagedHome({ + }), + seedManagedHome: input.deps.prepareRemoteManagedHome + ? async (stage) => { + const seeded = await input.deps.prepareRemoteManagedHome!({ acpxAgent, companyId: agent.companyId, runId, @@ -1968,67 +1995,93 @@ async function buildRuntime(input: { onRuntimeProgress: input.ctx.onRuntimeProgress, stage, }); - } catch (seamErr) { - // The seam failed after a possible successful stage. Dispose the fresh - // staged runtime so the abandoned in-sandbox managed home does not leak, - // then rethrow the seam error. - if (freshlyStagedRuntime) { - await disposeFreshStagedRuntime({ - runId, - target: remoteTarget, - stagedRuntime: freshlyStagedRuntime, - cwd: sessionCwd, - timeoutSec, - onLog: input.ctx.onLog, - }); - } - throw seamErr; + return { + stagedRuntime: seeded.stagedRuntime, + teardown: seeded.teardown ?? null, + dispose: seeded.disposeStaged ?? null, + }; } - return { - stagedRuntime: seeded.stagedRuntime, - teardown: seeded.teardown ?? null, - dispose: seeded.disposeStaged ?? null, - }; + : undefined, + disposeFreshStagedRuntime: (freshStagedRuntime) => + disposeFreshStagedRuntime({ + runId, + target: remoteTarget, + stagedRuntime: freshStagedRuntime, + cwd: sessionCwd, + timeoutSec, + onLog: input.ctx.onLog, + }), + measureStageStep: (run) => measureStartupStep(input.ctx, nowMs, "stage.sync", run, stepMetrics), + publishStagedProjectHints: (stagedProjectDirs) => { + const shapedHints = shapePaperclipWorkspaceEnvForExecution({ + workspaceCwd: effectiveWorkspaceCwd, + workspaceWorktreePath, + workspaceHints, + executionTargetIsRemote, + executionCwd: effectiveExecutionCwd, + stagedProjectDirs, + }).workspaceHints; + if (shapedHints.length > 0) { + env.PAPERCLIP_WORKSPACES_JSON = JSON.stringify(shapedHints); } - return { stagedRuntime: await stage([]), teardown: null, dispose: null }; - }, stepMetrics); - const delta: Record = {}; - for (const [key, value] of Object.entries(env)) { - if (envBeforeStage[key] !== value) delta[key] = value; - } - return { - stagedRuntime: freshStagedRuntime, - teardown: freshTeardown, - dispose: freshDispose, - envDelta: delta, - }; + }, + onReuseLog: () => + input.ctx.onLog( + "stdout", + "[paperclip] Reusing the staged in-sandbox runtime for this resumed session (no workspace re-ship / managed-home re-seed).\n", + ), + startPaperclipBridge: (runtimeRootDir) => + startAdapterExecutionTargetPaperclipBridge({ + runId, + target: { ...remoteTarget, streamRunLogs: false }, + runtimeRootDir, + adapterKey: input.engine.adapterType, + timeoutSec, + hostApiToken: env.PAPERCLIP_API_KEY, + onLog: input.ctx.onLog, + getRuntimeParentContext: input.getRuntimeParentContext, + runtimeSpan: input.runtimeSpan, + }), + startProcessSessionBridge: ({ runtimeRootDir, launchEnv }) => + startAdapterExecutionTargetProcessSessionBridge({ + runId, + target: remoteTarget, + runtimeRootDir, + adapterKey: input.engine.adapterType, + command: "sh", + args: ["-lc", `exec ${agentCommandShell}`], + cwd: sessionCwd, + env: launchEnv, + timeoutSec, + onLog: input.ctx.onLog, + getRuntimeParentContext: input.getRuntimeParentContext, + runtimeSpan: input.runtimeSpan, + streamOutputViaSession: streamAgentSessionOutput, + }), + measureBridgeStep: (step, run) => + measureStartupStep(input.ctx, nowMs, step, run, concurrentBridgeStepMetrics), + finalizeLaunchEnv: (contributions) => finalizeLaunchEnvironment(env, contributions).env, + onPaperclipBridgeLog: () => + input.ctx.onLog("stdout", "[paperclip] Sandbox ACP API callback bridge enabled for this run.\n"), + stopBridges: async ({ controlBridge, agentBridge }) => { + await Promise.allSettled([agentBridge?.stop(), controlBridge?.stop()]); + if (remoteManagedHomeTeardown) { + await remoteManagedHomeTeardown().catch(() => {}); + } + }, }); - sessionStagingLeaseRelease = staged.release; - stagedRuntime = staged.value.stagedRuntime; - remoteManagedHomeTeardown = staged.value.teardown; - remoteStagingDispose = staged.value.dispose; - remoteStagingEnvDelta = staged.value.envDelta; - // Publish the referenced-project workspace hints to the in-sandbox agent. The staged-directory - // map (`project-`) is known only after staging above, so this runs here rather than - // with the initial workspace shaping. Each referenced hint repoints at its staged directory; a - // referenced hint whose project did not stage loses its cwd, so the agent never receives an - // unstaged path. Only the confined sandbox lane stages referenced trees, so only it publishes - // the hints; the local and runner-less lanes keep their env untouched. The set `env` write wins - // over an inherited value in the merged launch env. - const stagedProjectDirs = stagedRuntime?.additionalSourceDirs ?? {}; - if (Object.keys(stagedProjectDirs).length > 0) { - const shapedHints = shapePaperclipWorkspaceEnvForExecution({ - workspaceCwd: effectiveWorkspaceCwd, - workspaceWorktreePath, - workspaceHints, - executionTargetIsRemote, - executionCwd: effectiveExecutionCwd, - stagedProjectDirs, - }).workspaceHints; - if (shapedHints.length > 0) { - env.PAPERCLIP_WORKSPACES_JSON = JSON.stringify(shapedHints); - } - } + // Place the workspace (stage fresh or reuse the already-staged runtime) under + // the per-session staging lease, then read the staged result back onto the + // prepared-runtime fields the settlement path consumes. + const placeWorkspaceStart = nowMs(); + await sandboxSite.placeWorkspace({ sessionKey } as unknown as AcpRunContext); + await emitRunPhaseTiming(input.ctx, "place_workspace", nowMs() - placeWorkspaceStart, "ok"); + const placedStaged = sandboxSite.staged; + stagedRuntime = placedStaged?.stagedRuntime ?? null; + remoteManagedHomeTeardown = placedStaged?.teardown ?? null; + remoteStagingDispose = placedStaged?.dispose ?? null; + remoteStagingEnvDelta = placedStaged?.envDelta ?? null; + sessionStagingLeaseRelease = sandboxSite.stagingLeaseRelease; } // Both bridge starts run under one try so a failure at EITHER — including the // paperclip callback bridge — fires the same abandon-path cleanup. The @@ -2039,88 +2092,34 @@ async function buildRuntime(input: { let paperclipBridge: AdapterExecutionTargetPaperclipBridgeHandle | null = null; let processSessionBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null = null; let runtimeEnv: Record = {}; + const startTransportStart = nowMs(); try { - if (useRemoteProcessSession) { - // Steps 5 + 6 — bring up BOTH host-side sandbox bridges concurrently. Their - // remote subtrees are disjoint (`…/paperclip-bridge/…` vs - // `…/process-sessions/…`), so the env-INDEPENDENT setup of each overlaps, - // trending wall time from serial (~bridge.paperclip + ~bridge.process-session) - // toward ~max(the two). The ONE real dependency — the paperclip bridge's - // returned `env` must reach the process-session LAUNCH — is sequenced by - // `finalizeLaunchEnv`: the process-session bridge runs its env-independent - // dir/script setup first, then awaits that thunk right before its launch, so - // the launch always observes the merged paperclip env. - // - // Both `run.startup.step` events still emit — `measureStartupStep` records - // them in a `finally`, even on a start failure. - const paperclipStart = measureStartupStep(input.ctx, nowMs, "bridge.paperclip", () => - startAdapterExecutionTargetPaperclipBridge({ - runId, - target: { ...executionTarget, streamRunLogs: false }, - runtimeRootDir: stagedRuntime?.runtimeRootDir ?? null, - adapterKey: input.engine.adapterType, - timeoutSec, - hostApiToken: env.PAPERCLIP_API_KEY, - onLog: input.ctx.onLog, - getRuntimeParentContext: input.getRuntimeParentContext, - runtimeSpan: input.runtimeSpan, - }), - concurrentBridgeStepMetrics, - ); - // The single sequencing point (paperclip `env` → process-session launch). - // Memoized so the merge + log + `runtimeEnv` build run EXACTLY once whether - // the process-session bridge consumes it at launch or we finalize it below. - let launchEnvPromise: Promise> | null = null; - const finalizeLaunchEnv = (): Promise> => - (launchEnvPromise ??= (async () => { - const paperclip = await paperclipStart; - if (paperclip) { - Object.assign(env, paperclip.env); - await input.ctx.onLog("stdout", "[paperclip] Sandbox ACP API callback bridge enabled for this run.\n"); - } - return (runtimeEnv = resolveRuntimeEnv(env)); - })()); - const processSessionStart = measureStartupStep(input.ctx, nowMs, "bridge.process-session", () => - startAdapterExecutionTargetProcessSessionBridge({ - runId, - target: executionTarget, - runtimeRootDir: stagedRuntime?.runtimeRootDir ?? null, - adapterKey: input.engine.adapterType, - command: "sh", - args: ["-lc", `exec ${agentCommandShell}`], - cwd: sessionCwd, - // Deferred: the process-session bridge runs its env-independent setup, - // then calls this to get the launch env AFTER the paperclip env merge. - env: finalizeLaunchEnv, - timeoutSec, - onLog: input.ctx.onLog, - getRuntimeParentContext: input.getRuntimeParentContext, - runtimeSpan: input.runtimeSpan, - streamOutputViaSession: streamAgentSessionOutput, - }), - concurrentBridgeStepMetrics, - ); - // Settle BOTH starts (mirrors `cleanupRemoteBridges`' `Promise.allSettled`): - // collect whichever handles started plus the first failure. Both handles - // stay individually declared so the catch below can stop whichever started. - const started = await settleRemoteBridgeStarts(paperclipStart, processSessionStart); - paperclipBridge = started.paperclipBridge; - processSessionBridge = started.processSessionBridge; - if (started.failure) throw started.failure; - // Guarantee the paperclip env merge ran even if the process-session bridge - // returned without consuming the launch env (memoized ⇒ a no-op if it did). - await finalizeLaunchEnv(); + if (useRemoteProcessSession && sandboxSite) { + // The sandbox run site brings up both host-side bridges concurrently, keeps + // the one paperclip-env → process-session-launch dependency at a single + // sequencing point, settles both starts, and returns the started handles + // plus the finalized launch env. On a partial failure it stops nothing and + // rethrows; the catch below stops whichever bridge the site started. + const transport = await sandboxSite.startTransport({ sessionKey } as unknown as AcpRunContext); + paperclipBridge = transport.controlBridge; + processSessionBridge = transport.agentBridge; + runtimeEnv = transport.launchEnv; + await emitRunPhaseTiming(input.ctx, "start_transport", nowMs() - startTransportStart, "ok"); } else { - // Local / runner-less lanes never start a bridge, but the returned prepared - // runtime and the log builder still read `runtimeEnv`. - runtimeEnv = resolveRuntimeEnv(env); + // Local / runner-less lanes never start a bridge, so they add no + // contribution. `finalizeLaunchEnvironment` still produces the one branded + // launch env the prepared runtime and the log builder read. + runtimeEnv = finalizeLaunchEnvironment(env, []).env; } } catch (err) { // On a partial concurrent bring-up failure, ONE bridge may have started while // the other threw; `Promise.allSettled` stops whichever started so no live - // bridge leaks (mirrors `cleanupRemoteBridges`). Both handles are individually - // declared above, so either may be non-null here. - await Promise.allSettled([paperclipBridge?.stop(), processSessionBridge?.stop()]); + // bridge leaks (mirrors the settlement `stopTransport` step). The site sets its + // started bridges before it rethrows, so read them from the site here (the + // local handles stay null when `startTransport` throws before it returns). + const startedControl = sandboxSite?.controlBridge ?? paperclipBridge; + const startedAgent = sandboxSite?.agentBridge ?? processSessionBridge; + await Promise.allSettled([startedControl?.stop(), startedAgent?.stop()]); // The staged home / copy-back teardown must run even if a bridge fails to // start after the workspace + managed home were already staged into the // sandbox, so a refreshed credential is copied back on this error path too. @@ -2128,7 +2127,7 @@ async function buildRuntime(input: { // dispose here (it no longer rides the per-run copy-back) — the run is being // abandoned, so its staged temp must be released — and release the per-session // staging lease so the abandoned run does not strand the next same-session run - // (cleanupRemoteBridges, which normally releases it, is never reached here). + // (the run root `finally`, which normally releases it, is never reached here). // // Route the dispose through the same ownership-guarded removal // `discardStagedRuntime` uses. When this run BORROWED the cached staged runtime @@ -2145,6 +2144,7 @@ async function buildRuntime(input: { dispose: remoteStagingDispose, }); sessionStagingLeaseRelease?.(); + await emitRunPhaseTiming(input.ctx, "start_transport", nowMs() - startTransportStart, "failed"); throw err; } const overrideCommand = processSessionBridge?.agentCommand ?? agentCommand; @@ -2274,60 +2274,50 @@ function resolveRuntimeEnv(env: Record): Record ); } -/** - * Bring up the two host-side sandbox bridges concurrently and settle both. - * - * Mirrors `cleanupRemoteBridges`' `Promise.allSettled` idiom (settle, not - * `Promise.all`): running BOTH starts to completion is what lets the caller STOP - * a bridge that DID start when its sibling threw — so a partial failure never - * leaks a live bridge. Returns whichever handles started plus the first failure - * (paperclip before process-session) for the caller to rethrow through the - * shared abandon path. - */ -async function settleRemoteBridgeStarts( - paperclipStart: Promise, - processSessionStart: Promise, -): Promise<{ - paperclipBridge: AdapterExecutionTargetPaperclipBridgeHandle | null; - processSessionBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null; - failure: unknown; -}> { - const [paperclip, processSession] = await Promise.allSettled([ - paperclipStart, - processSessionStart, +// Stop both host-side bridges in one `allSettled`. This is the settlement +// `stopTransport` effect. The bridge tokens are run-scoped, so they die with the +// bridges here (Amendment B). +async function stopRunTransport(prepared: AcpxPreparedRuntime): Promise { + await Promise.allSettled([ + prepared.processSessionBridge?.stop(), + prepared.paperclipBridge?.stop(), ]); - return { - paperclipBridge: paperclip.status === "fulfilled" ? paperclip.value : null, - processSessionBridge: processSession.status === "fulfilled" ? processSession.value : null, - failure: - paperclip.status === "rejected" - ? paperclip.reason - : processSession.status === "rejected" - ? processSession.reason - : null, - }; } -async function cleanupRemoteBridges(prepared: AcpxPreparedRuntime): Promise { - try { - await Promise.allSettled([ - prepared.processSessionBridge?.stop(), - prepared.paperclipBridge?.stop(), - ]); - // Runs AFTER the bridges stop (mirrors the CLI finally: stop bridge → restore - // workspace). Fires the codex auth copy-back via `restoreWorkspace()` and - // removes staged temp dirs. The seam logs and swallows its own failures — an - // unclean-teardown copy-back miss is the accepted, loud `refresh_token_reused` - // residual on the next host Codex use, never silent HOST-credential corruption - // — so a teardown fault never masks or fails the run result here. - if (prepared.remoteManagedHomeTeardown) { - await prepared.remoteManagedHomeTeardown().catch(() => {}); - } - } finally { - // Release the per-session staging lease in a finally, so an earlier teardown - // fault never strands it and deadlocks the next run of this session. - prepared.sessionStagingLeaseRelease?.(); +// The site sync-back. This is the settlement `syncBack` effect. It runs AFTER the +// bridges stop (mirrors the CLI finally: stop bridge → restore workspace). It +// fires the codex auth copy-back via `restoreWorkspace()` and removes staged temp +// dirs. The seam logs and swallows its own failures — an unclean-teardown +// copy-back miss is the accepted, loud `refresh_token_reused` residual on the +// next host Codex use, never silent HOST-credential corruption — so a teardown +// fault never masks or fails the run result here. +async function syncBackManagedHome(prepared: AcpxPreparedRuntime): Promise { + if (prepared.remoteManagedHomeTeardown) { + await prepared.remoteManagedHomeTeardown().catch(() => {}); } + // The per-session staging lease does NOT release here. The settlement releases + // it last, in its own `finally`, so a same-session second run cannot re-stage + // until this run fully settles and the caller observes the result. +} + +/** How the settlement `endSession` step releases the runtime a run acquired. */ +interface RuntimeSettlementPlan { + // "direct" closes the runtime itself and swallows or records the close error; + // "warm_or_close" prefers a matching warm entry (which also flushes its stderr). + readonly mode: "direct" | "warm_or_close"; + readonly handle: AcpRuntimeHandle; + readonly reason: string; + readonly discardPersistentState: boolean; + // Drop a matching warm entry after a direct close (the pre-turn and turn-error + // paths remove a warm-hit entry that failed). + readonly dropWarmEntry: boolean; + // Record a direct-close error to the run log; a warm_or_close error is + // swallowed. This keeps today's split (the failure paths log, the turn path + // swallows). + readonly recordCloseError: boolean; + // Cancel the running turn with this reason before the close (the turn-error + // path cancels before it closes). Null on every other path. + readonly cancelTurnReason: string | null; } function renderPaperclipEnvNote(env: Record): string { @@ -2799,31 +2789,10 @@ function isResumeFailure(err: unknown): boolean { return /resume|load|not found|no session|unknown session|conversation/i.test(message); } -async function cleanupIdleHandles(input: { - handles: Map; - now: number; - idleMs: number; -}) { - if (input.idleMs <= 0) return; - - const stale: Array<[string, RuntimeCacheEntry]> = []; - for (const entry of input.handles.entries()) { - if (input.now - entry[1].lastUsedAt >= input.idleMs) stale.push(entry); - } - for (const [key, entry] of stale) { - await closeWarmHandle({ - handles: input.handles, - key, - entry, - reason: "paperclip idle cleanup", - }); - } -} - // Drop staged-runtime entries the session has not touched within the warm-idle // window, so the cache does not accumulate abandoned sessions (e.g. every time // a config change shifts the fingerprint to a new key). The per-run copy-back -// already ran on the entry's last run's `cleanupRemoteBridges`; eviction fires +// already ran on the entry's last run's settlement `syncBack`; eviction fires // the entry's one-time `dispose` (host staged-temp cleanup) — the only place // the staged temp is removed now that it no longer rides the per-run teardown. // A later run of the same session simply re-stages fresh (re-shipping into the @@ -2979,37 +2948,6 @@ async function closeWarmHandle(input: { flushChildStderr(input.entry.childStderrState); } -function scheduleIdleHandleCleanup(input: { - handles: Map; - key: string; - entry: RuntimeCacheEntry; - idleMs: number; - now: () => number; -}) { - clearWarmHandleTimer(input.entry); - if (input.idleMs <= 0) return; - - const delayMs = Math.max(1, input.entry.lastUsedAt + input.idleMs - input.now()); - input.entry.cleanupTimer = setTimeout(() => { - void (async () => { - const current = input.handles.get(input.key); - if (current !== input.entry) return; - const idleForMs = input.now() - input.entry.lastUsedAt; - if (idleForMs < input.idleMs) { - scheduleIdleHandleCleanup(input); - return; - } - await closeWarmHandle({ - handles: input.handles, - key: input.key, - entry: input.entry, - reason: "paperclip idle cleanup", - }); - })(); - }, delayMs); - input.entry.cleanupTimer.unref?.(); -} - function warmHandleMatches( entry: RuntimeCacheEntry | undefined, runtime: AcpRuntime, @@ -3249,6 +3187,24 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { billingType: billingIdentity?.billingType ?? ("unknown" as const), }; const warmIdleMs = asNumber(ctx.config.warmHandleIdleMs, DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS); + // The host run site owns the warm-handle store on this run. It operates over + // the engine's persistent `warmHandles` map, so a warm runtime stays live for + // the next compatible resume. The store arms the per-entry idle timer on a + // warm save, sweeps idle entries at run start, and closes a released entry + // through `closeWarmEntry`. The run reads `warmHandleIdleMs` per run, so the + // site takes this run's bound. + const hostSite = createHostRunSite({ + warmHandles, + now, + idleMs: warmIdleMs, + closeWarmEntry: async (entry) => { + await entry.runtime + .close({ handle: entry.handle, reason: "paperclip idle cleanup", discardPersistentState: false }) + .catch(() => {}); + flushChildStderr(entry.childStderrState); + }, + }); + const hostStore = hostSite.reuse(); // The `task.run` and `sandbox.startup` spans must not cover a local or SSH // run: those runs have no sandbox, so they stay out of sandbox telemetry. // Open a real root span only when the target is a remote sandbox and the @@ -3301,6 +3257,12 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { // until the run reaches a clean completed turn, so every failure and every // early exit closes the span with error status. let runFailed = true; + // The run's per-session staging lease release. `buildRuntime` returns it on + // `prepared` after it acquires the lease; the startup step captures it here so + // the run root `finally` can release it as the final settlement act. It stays + // null on the host lane (no staging) and on a build failure (where + // `buildRuntime` already released its own partial lease). + let releaseStagingLease: (() => void) | null = null; try { // Evict idle staged runtimes BEFORE building the runtime, since buildRuntime // consults the staged cache to decide whether a compatible resume may reuse @@ -3366,37 +3328,78 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { let referencedProjectStagingFailuresField: | { referencedProjectStagingFailures: Array<{ projectId: string }> } | Record = {}; - // One teardown policy for the run. `settleRunResources` stops the bridges, - // runs the managed-home copy-back, releases the staging lease (in a finally - // inside cleanupRemoteBridges), and flushes the child stderr. It runs at most - // once, records each step error, and never throws, so a result-emission - // failure or an earlier teardown fault never skips a later step. - let runResourcesSettled = false; - // `turnTeardownDone` marks that the turn path already closed the runtime and - // settled the staged runtime, so a result-mapping throw after the close does - // not run the turn teardown a second time. - let turnTeardownDone = false; const recordTeardownError = async (step: string, teardownErr: unknown) => { const reason = teardownErr instanceof Error ? teardownErr.message : String(teardownErr); await ctx .onLog("stderr", `[paperclip] ACPX teardown step "${step}" failed: ${reason}\n`) .catch(() => {}); }; - const settleRunResources = async () => { - if (runResourcesSettled) return; - runResourcesSettled = true; + // Emit one per-phase timing telemetry event. It is observability-only: it + // carries the phase name (from the closed allowlist), the wall time, and the + // outcome, and never a command, a path, an environment value, or an + // identifier. Telemetry failure never fails the run. + const emitPhase = (phase: string, startMs: number, outcome: "ok" | "failed"): Promise => + emitRunPhaseTiming(ctx, phase, now() - startMs, outcome); + // Time a settlement step and emit its phase timing on every path. A step + // error still emits `failed` before it re-throws to the Phase 3 error policy. + const timedPhase = async (phase: string, run: () => Promise | void): Promise => { + const start = now(); try { - await cleanupRemoteBridges(prepared); - } catch (teardownErr) { - await recordTeardownError("cleanup-remote-bridges", teardownErr); - } - try { - if (childStderrState) flushChildStderr(childStderrState); - } catch (teardownErr) { - await recordTeardownError("flush-child-stderr", teardownErr); + await run(); + } catch (error) { + await emitPhase(phase, start, "failed"); + throw error; } + await emitPhase(phase, start, "ok"); }; - try { + // The turn the run started. It is hoisted to the run scope so the settlement + // `endSession` step can cancel a running turn before it closes the runtime + // (the cancel-before-close order). The turn wrapper assigns it in `turnStart`. + let activeTurn: AcpRuntimeTurn | null = null; + // How the settlement `endSession` step must release the runtime for the path + // this run took. Each exit path that acquired the runtime records it before it + // returns; a build or create-runtime failure never registers the runtime, so + // the step no-ops on the empty slot and this stays null. + let runtimeSettlement: RuntimeSettlementPlan | null = null; + // A synthetic close handle from the prepared identity, for a close that has no + // established session handle (a cold `ensureSession` throw or a missing + // handle). `runtime.close` reads the session key and the runtime session name. + const syntheticCloseHandle = (): AcpRuntimeHandle => ({ + sessionKey: prepared.sessionKey, + backend: prepared.acpxAgent, + runtimeSessionName: prepared.sessionKey, + }); + // The run coordinator owns the one ledger for the attempt. The engine + // creates it here, passes it into `buildRuntime` (the sandbox site registers + // what it acquires into it), and hands it to `runAttempt` as the plan + // ledger. The turn and settlement code still runs inline in the step + // wrappers below; Phases 21-22 move it into its own modules. + const runResourceLedger = createRunResourceLedger(); + // The step wrappers build the external result inline (today's behavior) and + // record it here; the coordinator reproduces it after settlement. An empty + // consumed set satisfies the `settle` result shape without claiming the run + // ledger, because the inline teardown reads `prepared.*`, not the ledger, in + // this phase. + let capturedResult: AdapterExecutionResult | null = null; + const emptyConsumed = createRunResourceLedger().takeForSettlement(); + // Build the `settle` startup result for a pre-turn failure. The step wrapper + // already recorded the external result and ran the inline teardown; this + // carries only the routing kind and the cause for the coordinator. + const settleFor = (phase: PreTurnFailedCause["phase"], error: unknown): StartupResult => ({ + kind: "settle", + cause: { + kind: "pre_turn_failed", + phase, + error: error instanceof Error ? error : new Error(String(error)), + }, + resources: emptyConsumed, + }); + // The startup step: bring up the runtime, establish the session, and apply + // the session config. It returns `ready` on success, `settle` on a pre-turn + // failure after the ledger acquired resources, and throws on a build failure + // or a partial-bridge failure (the coordinator owns that rollback). + const startup = async (): Promise => { + try { // Publish the `sandbox.startup` context to the runtime-parent store for // the whole bring-up. A startup-body exec that runs outside a measured // step reads this token and parents its span to `sandbox.startup`, not to @@ -3405,9 +3408,22 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { // parents to its step span. On a local or SSH target // `spanParent.parentContext` is a no-op token, so the wrap is inert. prepared = await runWithRuntimeParent(spanParent.parentContext, () => - buildRuntime({ ctx, engine, deps, spanParent, getRuntimeParentContext, runtimeSpan: runRuntimeSpan, stageRuntimeSpan: runStageSpan }), + buildRuntime({ + ctx, + engine, + deps, + ledger: runResourceLedger, + stagedIdleMs: warmIdleMs, + spanParent, + getRuntimeParentContext, + runtimeSpan: runRuntimeSpan, + stageRuntimeSpan: runStageSpan, + }), ); buildRuntimeSettled = true; + // Capture the run's staging lease release now that the runtime built. The + // run root `finally` releases it as the final settlement act. + releaseStagingLease = prepared.sessionStagingLeaseRelease; // Per-project staging outcomes for the referenced (mentioned) projects, surfaced back to the // server on the run result. A referenced project that failed to stage into the sandbox is a // first-class, counted failure in the requested-vs-synced observability, not only a warning. The @@ -3427,12 +3443,16 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { "stderr", `[paperclip] ${formatAdapterExecutionTimeoutStartLogLine(prepared.timeoutResolution)}\n`, ); - await cleanupIdleHandles({ handles: warmHandles, now: now(), idleMs: warmIdleMs }); + await hostStore.evictIdle(now()); const previousParams = parseObject(ctx.runtime.sessionParams); const canResume = isCompatibleSession(previousParams, prepared); const resumeSessionId = canResume ? asString(previousParams.acpSessionId, "") || undefined : undefined; - const cached = canResume ? warmHandles.get(prepared.sessionKey) : undefined; + // Borrow the warm entry without removing it, so an overlapping run of the + // same session still sees it. The borrow clears the entry's idle timer, so + // the reused runtime cannot expire under its own timer while this run uses + // it (today's `clearWarmHandleTimer(cached)` after the reuse decision). + const cached = canResume ? hostStore.borrow(prepared.sessionKey) : undefined; childStderrState = cached?.childStderrState ?? { logPath: null, pendingLiveLine: "" }; processIdentitySink = cached?.processIdentitySink ?? { current: ctx.onSpawn, @@ -3493,8 +3513,27 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { const createRuntimeStart = now(); runtime = createRuntime(runtimeOptions); createRuntimeMs = now() - createRuntimeStart; + // The create_runtime phase runs only on a cold start. + await emitRunPhaseTiming(ctx, "create_runtime", createRuntimeMs, "ok"); } - if (cached) clearWarmHandleTimer(cached); + // Register the runtime composite in the ledger now that it exists. The + // settlement `endSession` step closes it on every path the reuse decision + // does not transfer. Registration here (before `ensureSession`) closes the + // cold-handshake leak: a cold `ensureSession` throw before a handle exists + // still leaves the runtime in the ledger for `endSession` to close. A + // create-runtime failure throws before this line, so the runtime never + // enters the ledger and the settlement no-ops on the empty slot. The + // session handle fills in below; the payload carries the best-known handle + // for the composite (`runtime.close` is the one release boundary). + runResourceLedger.register({ + id: "acp_runtime", + payload: { + runtime, + sessionHandle: cached?.handle ?? syntheticCloseHandle(), + childProcessPid: processIdentitySink.latest?.pid ?? null, + }, + scope: "per_run", + }); if (!canResume && asString(previousParams.runtimeSessionName, "")) { await ctx.onLog( "stdout", @@ -3503,6 +3542,9 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { } let handle = cached?.handle ?? null; + // The ensure_session phase covers the handshake (session establishment or + // resume) up to the established handle. + const ensureSessionPhaseStart = now(); resumedSession = Boolean(handle ?? resumeSessionId); try { @@ -3585,85 +3627,81 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { }); } } catch (err) { - // Close the runtime on a pre-turn handshake failure and drop the matching - // warm entry, the same as the configuration failure path. A warm-hit - // reuse already cleared the idle timer before the handshake ran, so the - // failure must close and remove the entry, never re-arm it. The close and - // the staged-runtime drop run before the result emission, so a throwing - // emission never skips them; `settleRunResources` in the finally stops the - // bridges, releases the staging lease, and flushes the child stderr. - if (handle) { - await runtime.close({ - handle, - reason: "paperclip handshake cleanup", - discardPersistentState: false, - }).catch((closeErr) => recordTeardownError("runtime-close", closeErr)); - const existing = warmHandles.get(prepared.sessionKey); - if (warmHandleMatches(existing, runtime, handle) && existing) { - clearWarmHandleTimer(existing); - warmHandles.delete(prepared.sessionKey); - } - } - await discardStagedRuntime({ handles: stagedRuntimes, prepared }); - try { - const { classified, message } = await emitAcpxFailure({ - ctx, - prepared, - err, - phase: "ensure_session", - }); - return { - exitCode: 1, - signal: null, - timedOut: false, - errorMessage: message, - ...classified, - ...billingFields, - ...referencedProjectStagingFailuresField, - model: prepared.requestedModel || null, - clearSession, - resultJson: { phase: "ensure_session" }, - summary: message, - }; - } finally { - await settleRunResources(); - } + // Record how the settlement closes the runtime on a pre-turn handshake + // failure: a direct close that drops a matching warm entry, the same as + // the configuration-failure path. A warm-hit reuse already cleared the + // idle timer before the handshake ran, so the failure must close and + // remove the entry, never re-arm it. A cold `ensureSession` throw has no + // established handle, so the settlement closes the synthetic handle — the + // ledger now holds the runtime, so `endSession` closes it and the former + // cold-handshake leak is gone. The settlement stops the bridges, releases + // the staging lease, and flushes the child stderr on every exit path. + runtimeSettlement = { + mode: "direct", + handle: handle ?? syntheticCloseHandle(), + reason: "paperclip handshake cleanup", + discardPersistentState: false, + dropWarmEntry: true, + recordCloseError: true, + cancelTurnReason: null, + }; + await emitPhase("ensure_session", ensureSessionPhaseStart, "failed"); + const { classified, message } = await emitAcpxFailure({ + ctx, + prepared, + err, + phase: "ensure_session", + }); + capturedResult = { + exitCode: 1, + signal: null, + timedOut: false, + errorMessage: message, + ...classified, + ...billingFields, + ...referencedProjectStagingFailuresField, + model: prepared.requestedModel || null, + clearSession, + resultJson: { phase: "ensure_session" }, + summary: message, + }; + return settleFor("handshake", err); } if (!handle) { - // ensureSession returned no session handle. Close the runtime the run - // constructed so its child process cannot leak. There is no established - // handle, so build a minimal one from the prepared identity; the close is - // best effort and never blocks the error result. - await runtime.close({ - handle: { - sessionKey: prepared.sessionKey, - backend: prepared.acpxAgent, - runtimeSessionName: prepared.sessionKey, - }, + // ensureSession returned no session handle. The ledger holds the runtime + // the run constructed, so record a direct close of the synthetic handle + // for the settlement `endSession` step (the child process cannot leak). + runtimeSettlement = { + mode: "direct", + handle: syntheticCloseHandle(), reason: "paperclip missing-handle cleanup", discardPersistentState: false, - }).catch((closeErr) => recordTeardownError("runtime-close", closeErr)); - await discardStagedRuntime({ handles: stagedRuntimes, prepared }); - try { - return { - exitCode: 1, - signal: null, - timedOut: false, - errorMessage: "ACPX did not return a runtime session handle.", - errorCode: "acpx_runtime_error", - ...billingFields, - ...referencedProjectStagingFailuresField, - model: prepared.requestedModel || null, - resultJson: { phase: "ensure_session" }, - summary: "ACPX did not return a runtime session handle.", - }; - } finally { - await settleRunResources(); - } + dropWarmEntry: false, + recordCloseError: true, + cancelTurnReason: null, + }; + await emitPhase("ensure_session", ensureSessionPhaseStart, "failed"); + capturedResult = { + exitCode: 1, + signal: null, + timedOut: false, + errorMessage: "ACPX did not return a runtime session handle.", + errorCode: "acpx_runtime_error", + ...billingFields, + ...referencedProjectStagingFailuresField, + model: prepared.requestedModel || null, + resultJson: { phase: "ensure_session" }, + summary: "ACPX did not return a runtime session handle.", + }; + return settleFor( + "session_handle_missing", + new Error("ACPX did not return a runtime session handle."), + ); } sessionHandle = handle; startupFailed = false; + await emitPhase("ensure_session", ensureSessionPhaseStart, "ok"); } catch (err) { if (!buildRuntimeSettled) { // buildRuntime failed before it staged or bridged anything, so there is @@ -3672,40 +3710,38 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { throw err; } // The post-build runtime-creation window failed after buildRuntime returned - // live bridges and a held staging lease. Drop the staged runtime, then emit - // and settle: `settleRunResources` in the finally stops the bridges, - // releases the staging lease, and flushes the child stderr even if the - // emission throws. - await discardStagedRuntime({ handles: stagedRuntimes, prepared }); - try { - const { classified, message } = await emitAcpxFailure({ - ctx, - prepared, - err, - phase: "create_runtime", - }); - return { - exitCode: 1, - signal: null, - timedOut: false, - errorMessage: message, - ...classified, - ...billingFields, - ...referencedProjectStagingFailuresField, - model: prepared.requestedModel || null, - clearSession, - resultJson: { phase: "create_runtime" }, - summary: message, - }; - } finally { - await settleRunResources(); - } + // live bridges and a held staging lease. `createRuntime` throws before the + // ledger registers the runtime, so the settlement `endSession` step no-ops + // on the empty slot (create_runtime never closes a runtime). The settlement + // still stops the bridges, discards the staged runtime, releases the staging + // lease, and flushes the child stderr. + const { classified, message } = await emitAcpxFailure({ + ctx, + prepared, + err, + phase: "create_runtime", + }); + capturedResult = { + exitCode: 1, + signal: null, + timedOut: false, + errorMessage: message, + ...classified, + ...billingFields, + ...referencedProjectStagingFailuresField, + model: prepared.requestedModel || null, + clearSession, + resultJson: { phase: "create_runtime" }, + summary: message, + }; + return settleFor("runtime_create", err); } finally { // End the sandbox.startup span exactly once, on every return and on every // throw. It covers buildRuntime through acp.handshake and no further; the // agent turn runs after and is out of the span's scope. rootSpan.end(startupFailed); } + const configureSessionStart = now(); try { await applySessionConfigOptions({ runtime, @@ -3713,63 +3749,78 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { prepared, onLog: ctx.onLog, }); + await emitPhase("configure_session", configureSessionStart, "ok"); } catch (err) { - // Close the runtime and drop the matching warm entry and the staged runtime - // before the result emission, so a throwing emission never skips them. - // `settleRunResources` in the finally stops the bridges, releases the - // staging lease, and flushes the child stderr. - await runtime.close({ + // Record a direct close that drops the matching warm entry for the + // settlement `endSession` step. The settlement stops the bridges, discards + // the staged runtime, releases the staging lease, and flushes the child + // stderr on this exit path too. + runtimeSettlement = { + mode: "direct", handle: sessionHandle, reason: "paperclip config cleanup", discardPersistentState: false, - }).catch((closeErr) => recordTeardownError("runtime-close", closeErr)); - const existing = warmHandles.get(prepared.sessionKey); - if (warmHandleMatches(existing, runtime, sessionHandle) && existing) { - clearWarmHandleTimer(existing); - warmHandles.delete(prepared.sessionKey); - } - await discardStagedRuntime({ handles: stagedRuntimes, prepared }); - try { - const { classified, message } = await emitAcpxFailure({ - ctx, - prepared, - err, + dropWarmEntry: true, + recordCloseError: true, + cancelTurnReason: null, + }; + await emitPhase("configure_session", configureSessionStart, "failed"); + const { classified, message } = await emitAcpxFailure({ + ctx, + prepared, + err, + phase: "configure_session", + }); + capturedResult = { + exitCode: 1, + signal: null, + timedOut: false, + errorMessage: message, + ...classified, + ...billingFields, + ...referencedProjectStagingFailuresField, + model: prepared.requestedModel || null, + clearSession, + resultJson: { phase: "configure_session", - }); - return { - exitCode: 1, - signal: null, - timedOut: false, - errorMessage: message, - ...classified, - ...billingFields, - ...referencedProjectStagingFailuresField, - model: prepared.requestedModel || null, - clearSession, - resultJson: { - phase: "configure_session", - agent: prepared.acpxAgent, - requestedModel: prepared.requestedModel || null, - requestedThinkingEffort: prepared.requestedThinkingEffort || null, - fastMode: prepared.fastMode, - }, - summary: message, - }; - } finally { - await settleRunResources(); - } + agent: prepared.acpxAgent, + requestedModel: prepared.requestedModel || null, + requestedThinkingEffort: prepared.requestedThinkingEffort || null, + fastMode: prepared.fastMode, + }, + summary: message, + }; + return settleFor("session_configuration", err); } - let cancelActiveTurn: ((reason: string) => Promise) | null = null; - let controller: AbortController | null = null; - let timeout: NodeJS.Timeout | null = null; - let timedOut = false; + // Startup succeeded: seal the ledger (promotes the startup_rollback entries + // that survived to `per_run`) and hand the ready resources to the turn. The + // turn wrapper reads the run state through the shared closure locals in this + // phase, so the sealed view and the context are the contract carriers only. + return { + kind: "ready", + resources: runResourceLedger.seal([]), + context: { sessionKey: prepared.sessionKey } as unknown as AcpRunContext, + }; + }; + // The turn step: run the turn sequence and settle the runtime inline + // (today's behavior). The sequence owns the wall-clock timer and the abort + // controller and never rejects; it returns a `TurnCompletion`. The step + // bodies below record the external result for the coordinator to reproduce. + const runTurn = async (_ready: StartupReady): Promise => { const textParts: string[] = []; let eventBreakdown: AcpRuntimeUsageBreakdown | null = null; let eventCostUsd: number | null = null; - // `turnStarted` separates a pre-turn preparation failure (prompt build or - // metadata emit) from a failure of the running turn, so the turn catch below - // reports the right phase and settles the runtime on both. - let turnStarted = false; + // The turn-local state the sequence steps share. `promptBuild` sets the + // prompt, `preTurnUsage` sets the pre-turn status, `turnStart` sets the + // active turn, and `turnFinalize` reads all three. `activeTurn` is the run- + // scoped hoisted local, so the settlement `endSession` step can cancel it. + let runPrompt = ""; + let preTurnStatus: AcpRuntimeStatus | null = null; + // Phase-timing markers for the prepare_turn and turn phases. The prepare + // phase covers the prompt build and the pre-turn usage snapshot; the turn + // phase covers the started turn and the event relay. + let preparePhaseStart = now(); + let turnPhaseStart = now(); // Open the agent turn span as a child of the run root span. It wraps the // whole turn: the executor holds `turnSpan.parentContext` for later exec // parenting, and the `finally` below ends the span once on every path. The @@ -3779,12 +3830,13 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { // a detached exec during the turn parents to `agent.turn`. The turn // `finally` resets the holder to the `task.run` token. currentRunParentContext = turnSpan.parentContext; - try { + const stepPromptBuild = async (_signal: AbortSignal): Promise => { // Build the prompt and emit the run metadata inside the turn failure - // boundary. A failure here settles the runtime through the turn catch and - // returns an error result with phase `prepare_turn`. + // boundary. A failure here returns an error result with phase + // `prepare_turn`. + preparePhaseStart = now(); const { prompt, promptMetrics, commandNotes } = await buildPrompt(ctx, resumedSession, prepared.env); - const runPrompt = joinPromptSections([prepared.skillPromptInstructions, prompt]); + runPrompt = joinPromptSections([prepared.skillPromptInstructions, prompt]); await emitAcpxLog(ctx, { type: "acpx.session", agent: prepared.acpxAgent, @@ -3828,30 +3880,34 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { context: ctx.context, }); } + }; + const stepPreTurnUsage = async (): Promise => { // Snapshot pre-turn usage so cumulative agent-reported cost can be // attributed to this run alone. - const preTurnStatus = await readRuntimeStatus(runtime, sessionHandle); - const timeoutMs = prepared.timeoutSec > 0 ? prepared.timeoutSec * 1000 : undefined; - controller = new AbortController(); - if (timeoutMs) { - timeout = setTimeout(() => { - timedOut = true; - controller?.abort(); - void cancelActiveTurn?.(formatAdapterExecutionTimeoutErrorMessage(prepared.timeoutResolution)).catch(() => {}); - }, timeoutMs); - } + preTurnStatus = await readRuntimeStatus(runtime, sessionHandle); + // The prepare phase (prompt build + usage snapshot) finished; the turn + // phase starts next. + await emitPhase("prepare_turn", preparePhaseStart, "ok"); + turnPhaseStart = now(); + }; + const stepTurnStart = (signal: AbortSignal, startTimeoutMs: number | undefined): StartedTurn => { const turn = runtime.startTurn({ handle: sessionHandle, text: runPrompt, mode: "prompt", requestId: ctx.runId, - timeoutMs, - signal: controller?.signal, + timeoutMs: startTimeoutMs, + signal, }); - turnStarted = true; - cancelActiveTurn = async (reason: string) => { - await turn.cancel({ reason }); + activeTurn = turn; + return { + cancel: async (reason: string) => { + await turn.cancel({ reason }); + }, }; + }; + const stepEventRelay = async (): Promise => { + const turn = activeTurn as AcpRuntimeTurn; const toolTitles = new Map(); for await (const event of turn.events) { if (event.type === "text_delta") textParts.push(event.text); @@ -3861,9 +3917,15 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { } await emitRuntimeEvent(ctx, event, toolTitles); } - const terminal = await turn.result; - if (timeout) clearTimeout(timeout); - // Read usage before the close/warm-handle paths below can discard state. + return await turn.result; + }; + const stepTurnFinalize = async ( + input: TurnFinalizeInput, + ): Promise => { + if (input.kind === "terminal") { + const terminal = input.terminal; + const timedOut = input.timedOut; + // Read usage before the settlement can discard runtime state. const postTurnStatus = await readRuntimeStatus(runtime, sessionHandle); const turnUsage = summarizeAcpxTurnUsage({ preStatus: preTurnStatus, @@ -3871,81 +3933,29 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { eventBreakdown, eventCostUsd, }); - if (terminal.status === "failed" || terminal.status === "cancelled" || timedOut) { - const existing = warmHandles.get(prepared.sessionKey); - if (warmHandleMatches(existing, runtime, sessionHandle) && existing) { - await closeWarmHandle({ - handles: warmHandles, - key: prepared.sessionKey, - entry: existing, - reason: timedOut ? "paperclip timeout cleanup" : `paperclip turn ${terminal.status}`, - discardPersistentState: terminal.status === "cancelled" || timedOut, - }); - } else { - await runtime.close({ - handle: sessionHandle, - reason: timedOut ? "paperclip timeout cleanup" : `paperclip turn ${terminal.status}`, - discardPersistentState: terminal.status === "cancelled" || timedOut, - }).catch(() => {}); - } - } else if (prepared.mode === "persistent" && warmIdleMs > 0 && !prepared.processSessionBridge) { - const existing = warmHandles.get(prepared.sessionKey); - if (existing && !warmHandleMatches(existing, runtime, sessionHandle)) { - await runtime.close({ - handle: sessionHandle, - reason: "paperclip duplicate warm handle cleanup", - discardPersistentState: false, - }).catch(() => {}); - } else { - const entry: RuntimeCacheEntry = { - runtime, - handle: sessionHandle, - childStderrState, - processIdentitySink, - fingerprint: prepared.fingerprint, - lastUsedAt: now(), - }; - warmHandles.set(prepared.sessionKey, entry); - scheduleIdleHandleCleanup({ - handles: warmHandles, - key: prepared.sessionKey, - entry, - idleMs: warmIdleMs, - now, - }); - } - } else { - const existing = warmHandles.get(prepared.sessionKey); - if (warmHandleMatches(existing, runtime, sessionHandle) && existing) { - await closeWarmHandle({ - handles: warmHandles, - key: prepared.sessionKey, - entry: existing, - reason: "paperclip completed turn cleanup", - }); - } else { - await runtime.close({ - handle: sessionHandle, - reason: "paperclip completed turn cleanup", - discardPersistentState: false, - }).catch(() => {}); - } - } - - // PR 3: keep the staged runtime warm for the next compatible resume only - // after a clean turn; a failed/cancelled/timed-out turn discards it so the - // next run stages fresh instead of reusing a torn-down session's staged - // credentials. Copy-back still fires for every outcome via - // `cleanupRemoteBridges` below (unchanged from PR 2). - if (terminal.status === "completed" && !timedOut) { - saveStagedRuntimeAfterCleanTurn({ handles: stagedRuntimes, prepared, now: now() }); - } else { - await discardStagedRuntime({ handles: stagedRuntimes, prepared }); - } - // The turn path has closed the runtime and settled the staged runtime. Mark - // it done so a result-mapping throw below runs the turn teardown once, not a - // second time through the catch. - turnTeardownDone = true; + const failedTurn = terminal.status === "failed" || terminal.status === "cancelled" || timedOut; + // Record how the settlement `endSession` step closes the runtime for this + // outcome. A clean persistent host turn is save-eligible, but the + // Amendment B credential gate fails on the host lane (the run API key is + // never revoked), so the reuse decision discards and `endSession` closes it + // (close-and-relaunch). The sandbox lane always closes the runtime; its + // staged files are the reuse. A matching warm entry closes through the warm + // store (which also flushes its stderr); otherwise the runtime closes + // directly. The settlement then stops the bridges, saves or discards the + // staged runtime, releases the staging lease, and flushes the child stderr. + runtimeSettlement = { + mode: "warm_or_close", + handle: sessionHandle, + reason: timedOut + ? "paperclip timeout cleanup" + : failedTurn + ? `paperclip turn ${terminal.status}` + : "paperclip completed turn cleanup", + discardPersistentState: terminal.status === "cancelled" || timedOut, + dropWarmEntry: false, + recordCloseError: false, + cancelTurnReason: null, + }; const errorMessage = timedOut ? formatAdapterExecutionTimeoutErrorMessage(prepared.timeoutResolution) @@ -3957,17 +3967,10 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { stopReason: terminalStopReason, message: errorMessage, }); - // Settle the run resources once: stop the bridges, run the copy-back, - // release the staging lease (in a finally inside cleanupRemoteBridges), and - // flush the child stderr. Mark them settled first so a result-mapping throw - // below does not run the teardown again through the turn catch. - runResourcesSettled = true; - await cleanupRemoteBridges(prepared); - flushChildStderr(childStderrState); // The one clean-completion path clears the run failure flag; every other // path keeps it set, so the run root span closes with error status. runFailed = terminal.status === "completed" && !timedOut ? false : true; - return { + capturedResult = { exitCode: terminal.status === "completed" ? 0 : 1, signal: timedOut ? "SIGTERM" : null, timedOut, @@ -3997,64 +4000,117 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { summary: textParts.join("").trim() || terminalStopReason || terminal.status, clearSession, }; - } catch (err) { - if (timeout) clearTimeout(timeout); - // A failure before `startTurn` returned is a turn-preparation failure; a - // failure after it is a running-turn failure. The teardown is the same; - // only the reported phase differs. - const phase: AcpxExecutionPhase = turnStarted ? "turn" : "prepare_turn"; + // The turn phase finished. A completed, non-timed-out turn is `ok`; every + // other terminal outcome is `failed`. + await emitPhase( + "turn", + turnPhaseStart, + terminal.status === "completed" && !timedOut ? "ok" : "failed", + ); + // Return the typed turn completion so the coordinator settles for the right + // cause. The completion carries no live resources; the settlement claims the + // ledger and owns the release. A clean completed turn carries no cause, so + // the reuse decision can permit a save; a timed-out, failed, or cancelled + // turn carries its cause, which forbids the save. + if (timedOut) { + return { + kind: "timed_out", + cause: { kind: "turn_timed_out", timeoutSec: prepared.timeoutSec }, + resources: emptyConsumed, + }; + } + if (terminal.status === "failed") { + return { + kind: "failed", + cause: { + kind: "turn_failed", + error: terminal.error instanceof Error ? terminal.error : new Error(String(terminal.error)), + }, + resources: emptyConsumed, + }; + } + if (terminal.status === "cancelled") { + return { + kind: "cancelled", + cause: { kind: "turn_cancelled", reason: terminal.stopReason ?? "cancelled" }, + resources: emptyConsumed, + }; + } + return { kind: "finalized" }; + } + const err = input.error; + const timedOut = input.timedOut; + // The failure phase comes from the sequence: a failure before the turn + // started is `prepare_turn`; a failure after it is `turn`. The teardown is + // the same; only the reported phase differs. + const phase: AcpxExecutionPhase = input.phase; + // Emit the failed phase timing for the phase the sequence reported. + if (phase === "prepare_turn") { + await emitPhase("prepare_turn", preparePhaseStart, "failed"); + } else { + await emitPhase("turn", turnPhaseStart, "failed"); + } const messageOverride = timedOut ? formatAdapterExecutionTimeoutErrorMessage(prepared.timeoutResolution) : undefined; - const cancel = cancelActiveTurn as ((reason: string) => Promise) | null; const preEmitMessage = messageOverride ?? (err instanceof Error ? err.message : String(err)); - // Skip the cancel, close, and staged-runtime drop when the success path - // already ran them: a result-mapping throw after the close reaches this - // catch, and the turn teardown must not run twice. The close records its - // error and never blocks the later teardown; `settleRunResources` in the - // finally stops the bridges, releases the staging lease, and flushes the - // child stderr even if the emission throws. - if (!turnTeardownDone) { - turnTeardownDone = true; - if (cancel) await cancel(preEmitMessage).catch(() => {}); - await runtime.close({ - handle: sessionHandle, - reason: timedOut ? "paperclip timeout cleanup" : "paperclip error cleanup", - discardPersistentState: timedOut, - }).catch((closeErr) => recordTeardownError("runtime-close", closeErr)); - const existing = warmHandles.get(prepared.sessionKey); - if (warmHandleMatches(existing, runtime, sessionHandle) && existing) { - clearWarmHandleTimer(existing); - warmHandles.delete(prepared.sessionKey); - } - await discardStagedRuntime({ handles: stagedRuntimes, prepared }); - } + // Record a direct close for the settlement `endSession` step: cancel the + // running turn first (cancel-before-close), then close the runtime and drop + // a matching warm entry. The settlement discards the staged runtime, stops + // the bridges, releases the staging lease, and flushes the child stderr. + runtimeSettlement = { + mode: "direct", + handle: sessionHandle, + reason: timedOut ? "paperclip timeout cleanup" : "paperclip error cleanup", + discardPersistentState: timedOut, + dropWarmEntry: true, + recordCloseError: true, + cancelTurnReason: preEmitMessage, + }; + // Emit the failure best-effort. `turnFinalize` must not reject, so a + // failing emission never propagates: the settlement owns the teardown, and + // it runs only after this returns a completion. On an emission failure the + // run records a degraded result from the pre-emit message. + let emitted: Awaited> | null = null; try { - const { classified, message } = await emitAcpxFailure({ - ctx, - prepared, - err, - phase, - messageOverride, - }); - return { - exitCode: 1, - signal: timedOut ? "SIGTERM" : null, - timedOut, - errorMessage: message, - errorCode: timedOut ? "acpx_timeout" : classified.errorCode, - errorMeta: classified.errorMeta, - ...billingFields, - ...referencedProjectStagingFailuresField, - model: prepared.requestedModel || null, - clearSession: clearSession || timedOut, - resultJson: { phase }, - summary: message, - }; - } finally { - await settleRunResources(); + emitted = await emitAcpxFailure({ ctx, prepared, err, phase, messageOverride }); + } catch { + emitted = null; } + const message = emitted?.message ?? preEmitMessage; + capturedResult = { + exitCode: 1, + signal: timedOut ? "SIGTERM" : null, + timedOut, + errorMessage: message, + errorCode: timedOut ? "acpx_timeout" : (emitted?.classified.errorCode ?? null), + errorMeta: emitted?.classified.errorMeta, + ...billingFields, + ...referencedProjectStagingFailuresField, + model: prepared.requestedModel || null, + clearSession: clearSession || timedOut, + resultJson: { phase }, + summary: message, + }; + // Return a typed failed completion so the coordinator settles for a cause + // that forbids the save. The reported phase lives on the recorded result. + return { + kind: "failed", + cause: { kind: "turn_failed", error: err instanceof Error ? err : new Error(String(err)) }, + resources: emptyConsumed, + }; + }; + try { + return await runTurnSequence({ + timeoutMs: prepared.timeoutSec > 0 ? prepared.timeoutSec * 1000 : undefined, + timeoutMessage: formatAdapterExecutionTimeoutErrorMessage(prepared.timeoutResolution), + promptBuild: stepPromptBuild, + preTurnUsage: stepPreTurnUsage, + turnStart: stepTurnStart, + eventRelay: stepEventRelay, + turnFinalize: stepTurnFinalize, + }); } finally { // End the agent turn span exactly once, on every return and on a throw. // `runFailed` is `false` only on a completed, non-timed-out turn, so the @@ -4065,9 +4121,182 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { // exec after the turn parents to `task.run`. currentRunParentContext = runRootSpan.parentContext; } + }; + // A live run-scoped credential marker. The host-lane reuse candidate carries + // it, so the Amendment B credential gate blocks the host warm save: the + // run-minted API key is never revoked, so it stays valid and forces a + // close-and-relaunch. It is a non-secret marker; the settlement reads only its + // presence, and no report or reuse payload carries it. + const LIVE_RUN_SCOPED_API_KEY = "paperclip-run-scoped-api-key"; + // Record the final per-resource disposition report where a test can observe + // it. The engine never reads it back. + const recordDispositionReport = (report: SettlementDispositionReport): void => { + deps.onSettlementDisposition?.(report); + }; + // The settlement sequence is the one live cleanup owner for every settled path + // (Phase 22). It claims the ledger once, makes the pure reuse decision, then + // runs the ordered steps: `endSession`, `settleReuse`, `stopTransport`, + // `syncBack`, and `releaseStagingLease` (in a finally). Each step reads the run + // state through the shared closure locals and no-ops on an empty ledger slot. + // The Phase 3 error policy governs every step. The coordinator alone owns the + // startup rollback. + const settlementSteps: SettlementSteps = { + // Derive the reuse candidate. Only a clean turn (no cause) can save. The + // host lane would save the live runtime, but the Amendment B credential gate + // fails there: the run-minted API key is never revoked, so a live run-scoped + // credential stays valid and the decision discards (close-and-relaunch). The + // sandbox lane saves its staged files, which carry no credential. + reuseCandidate: (slots, cause) => { + const clean = cause === null; + if (prepared.processSessionBridge) { + // Sandbox lane: the staged files are the reuse. Save only after a clean + // turn and only when the run holds a staged runtime. + if (!clean || !slots.has("staged_runtime")) return null; + return { kind: "sandbox", causePermitsSave: true, liveRunScopedCredentials: [] }; + } + // Host lane: save a clean persistent warm-eligible turn, but carry the + // live run-scoped credential so the gate blocks the transfer. + if (!slots.has("acp_runtime")) return null; + const permits = clean && prepared.mode === "persistent" && warmIdleMs > 0; + if (!permits) return null; + return { kind: "host", causePermitsSave: true, liveRunScopedCredentials: [LIVE_RUN_SCOPED_API_KEY] }; + }, + // Close every runtime the decision did not transfer, and drop the warm entry + // on close. + endSession: (slots, decision) => timedPhase("end_session", async () => { + if (!slots.has("acp_runtime")) return; + if (decision.kind === "save" && decision.savedId === "acp_runtime") return; + const settlement: RuntimeSettlementPlan = runtimeSettlement ?? { + mode: "direct", + handle: syntheticCloseHandle(), + reason: "paperclip cleanup", + discardPersistentState: false, + dropWarmEntry: false, + recordCloseError: true, + cancelTurnReason: null, + }; + // Cancel a running turn before the close (the turn-error path). + if (settlement.cancelTurnReason && activeTurn) { + await activeTurn.cancel({ reason: settlement.cancelTurnReason }).catch(() => {}); + } + const existing = warmHandles.get(prepared.sessionKey); + if ( + settlement.mode === "warm_or_close" && + warmHandleMatches(existing, runtime, settlement.handle) && + existing + ) { + // A matching warm entry closes through the warm store, which also + // clears its idle timer and flushes its child stderr. + await closeWarmHandle({ + handles: warmHandles, + key: prepared.sessionKey, + entry: existing, + reason: settlement.reason, + discardPersistentState: settlement.discardPersistentState, + }); + return; + } + const onCloseError = settlement.recordCloseError + ? (closeErr: unknown) => recordTeardownError("runtime-close", closeErr) + : () => {}; + await runtime + .close({ + handle: settlement.handle, + reason: settlement.reason, + discardPersistentState: settlement.discardPersistentState, + }) + .catch(onCloseError); + if (settlement.dropWarmEntry && warmHandleMatches(existing, runtime, settlement.handle) && existing) { + clearWarmHandleTimer(existing); + warmHandles.delete(prepared.sessionKey); + } + }), + // Perform the reuse decision. A save transfers the staged files to the site + // store (which arms the idle policy); every other case discards the staged + // runtime. The host warm save is closed in `endSession`. + settleReuse: (slots, decision) => timedPhase("settle_reuse", async () => { + if (!slots.has("staged_runtime")) return; + if (decision.kind === "save" && decision.savedId === "staged_runtime") { + saveStagedRuntimeAfterCleanTurn({ handles: stagedRuntimes, prepared, now: now() }); + return; + } + await discardStagedRuntime({ handles: stagedRuntimes, prepared }); + }), + // Stop both bridges in one allSettled. + stopTransport: () => timedPhase("stop_transport", async () => { + await stopRunTransport(prepared); + }), + // The site sync-back (the managed-home copy-back). + syncBack: () => timedPhase("sync_back", async () => { + await syncBackManagedHome(prepared); + }), + // The staging lease releases as the run's final act, AFTER the coordinator + // reproduces the result, in the run root `finally` below. This step stays a + // no-op in the live engine: a same-session second run must stay blocked on + // the lease until this run fully returns, not merely until the settlement + // sync-back finishes, so the release cannot move earlier into settlement. + releaseStagingLease: () => {}, + recordError: async (step, error) => { + await recordTeardownError(`settlement-${step}`, error); + }, + }; + // Drive the attempt through the coordinator routing table. The startup step + // returns `ready` or `settle` (or throws on a build or partial-bridge + // failure); the coordinator runs the turn on the ready path, settles, then + // reproduces the recorded result. + const plan: RunPlan = { + ledger: runResourceLedger, + startup, + runTurn, + // The coordinator owns the startup rollback: `buildRuntime` runs its own + // partial-bring-up rollback and throws, so no ledger resource is left to + // release here. + rollbackStartup: () => {}, + recordDisposition: recordDispositionReport, + // The settlement sequence is the one live cleanup owner. It claims the + // ledger, releases every settled resource, and returns the final + // disposition report. The child stderr flushes after the sync-back on every + // settled exit path (it stays null before the run reads the warm entry). + settle: async (reason: SettlementReason) => { + const cause: SettlementCause | null = + reason.kind === "pre_turn" + ? reason.cause + : reason.completion.kind === "finalized" + ? null + : reason.completion.cause; + const report = await settleAcpRun(runResourceLedger, cause, settlementSteps); + recordDispositionReport(report); + if (childStderrState) flushChildStderr(childStderrState); + }, + reproduceResult: (): AdapterExecutionResult => { + if (!capturedResult) { + throw new Error("run coordinator reproduced a result before the run recorded one"); + } + return capturedResult; + }, + }; + return await runAttempt(plan); } finally { // End the run root span exactly once, on every return and on a throw. runRootSpan.end(runFailed); + // Release the per-session staging lease as the run's final act, AFTER the + // coordinator settled every other resource and reproduced the result. It runs + // last, in this `finally`, so a same-session second run stays blocked on the + // lease until this run fully returns (not merely until the settlement + // sync-back finishes) and an earlier teardown fault never strands the lease. + // It stays null on a build failure (where `buildRuntime` released its own + // partial lease) and on the host lane (no staging). The settlement stopped the + // bridges and ran the sync-back before this point, so the ordering is + // bridge-stop → sync-back → lease release. + const leaseRelease = releaseStagingLease as (() => void) | null; + if (leaseRelease) { + const leaseStart = now(); + leaseRelease(); + // Fire-and-forget the phase timing: the release is the run's final act, so + // the run must not await telemetry here. `emitRunPhaseTiming` swallows a + // sink failure, so this never rejects. + void emitRunPhaseTiming(ctx, "release_staging_lease", now() - leaseStart, "ok"); + } } }; } diff --git a/packages/adapter-utils/src/acpx-engine/run-contracts.test.ts b/packages/adapter-utils/src/acpx-engine/run-contracts.test.ts new file mode 100644 index 0000000000..14dcaea70f --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-contracts.test.ts @@ -0,0 +1,85 @@ +// Compile-time type tests for the run contracts. +// +// The real assertion in each test is that this file compiles. A `@ts-expect-error` +// marks an assignment that MUST fail to typecheck; if it starts to compile, tsc +// reports the unused directive and the typecheck fails. The runtime body only +// keeps vitest happy, so each test names the property it proves. + +import { describe, expect, it } from "vitest"; +import type { + AcquiredRunResources, + ConsumedRunResources, + PreTurnFailedCause, + ReadyRunResources, + SessionFingerprintIdentity, + TurnCause, +} from "./run-contracts.js"; + +// A type-level equality check. `Equal` is true only when A and B are the +// same type. +type Equal = + (() => T extends A ? 1 : 2) extends () => T extends B ? 1 : 2 ? true : false; +type Expect = T; + +describe("run contracts", () => { + it("test_a_turn_cause_is_not_assignable_where_pre_turn_failed_cause_is_required", () => { + function requiresPreTurn(_cause: PreTurnFailedCause): void {} + const turnCause: TurnCause = { kind: "turn_failed", error: new Error("boom") }; + + // A turn cause has a different `kind` and no `phase`, so it is not + // assignable where a `PreTurnFailedCause` is required. + // @ts-expect-error a TurnCause is not a PreTurnFailedCause. + requiresPreTurn(turnCause); + + expect(turnCause.kind).toBe("turn_failed"); + }); + + it("test_only_seal_produces_ready_run_resources_and_only_take_for_settlement_produces_consumed_run_resources", () => { + // The assertions live in an unexecuted function so tsc checks the types + // without a runtime call on a synthetic ledger. + function _assertBrands(ledger: AcquiredRunResources): void { + // `seal()` is the only source of a `ReadyRunResources`; + // `takeForSettlement()` is the only source of a `ConsumedRunResources`. + const ready: ReadyRunResources = ledger.seal([]); + const consumed: ConsumedRunResources = ledger.takeForSettlement(); + + // The two brands are distinct: neither result is assignable to the other. + // @ts-expect-error a ReadyRunResources is not a ConsumedRunResources. + const _crossA: ConsumedRunResources = ready; + // @ts-expect-error a ConsumedRunResources is not a ReadyRunResources. + const _crossB: ReadyRunResources = consumed; + + // A plain object cannot stand in for a branded result; it lacks the + // private brand that only `seal()` / `takeForSettlement()` can set. + // @ts-expect-error missing the private ready brand. + const _fakeReady: ReadyRunResources = { + get: () => undefined, + scopeOf: () => undefined, + has: () => false, + sealed: [], + }; + // @ts-expect-error missing the private consumed brand. + const _fakeConsumed: ConsumedRunResources = { entries: () => [] }; + + void ready; + void consumed; + void _crossA; + void _crossB; + void _fakeReady; + void _fakeConsumed; + } + + void _assertBrands; + expect(true).toBe(true); + }); + + it("test_session_fingerprint_identity_has_no_company_agent_or_task_member", () => { + // None of the outer-key parts appear as a fingerprint field. + type ForbiddenKey = "companyId" | "agentId" | "taskKey"; + type LeakedKey = Extract; + type _noLeak = Expect>; + + const _ok: _noLeak = true; + expect(_ok).toBe(true); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/run-contracts.ts b/packages/adapter-utils/src/acpx-engine/run-contracts.ts new file mode 100644 index 0000000000..a82501bd47 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-contracts.ts @@ -0,0 +1,509 @@ +// Contract types for the ACPX run coordinator refactor. +// +// This module declares the types only. No production code consumes them yet. +// Each placeholder type maps to a field that `AcpxPreparedRuntime` carries in +// `execute.ts` today. The later coordinator phases move the ownership of those +// fields onto these contracts one seam at a time. +// +// Two brands (`ReadyRunResources`, `ConsumedRunResources`) mark a resource set +// that only the resource ledger can produce. Two more brands mark a run-scoped +// launch contribution and the finalized launch environment. A brand is a private +// `unique symbol`, so no other module can build a branded value by a plain +// object literal. Only the module that owns the symbol returns the value. + +import type { AcpRuntime, AcpRuntimeHandle } from "acpx/runtime"; +import type { + AdapterExecutionTarget, + AdapterExecutionTargetPaperclipBridgeHandle, + AdapterExecutionTargetProcessSessionBridgeHandle, + PreparedAdapterExecutionTargetRuntime, +} from "@paperclipai/adapter-utils/execution-target"; + +// --------------------------------------------------------------------------- +// Resource identity +// --------------------------------------------------------------------------- + +/** + * The closed set of run resources the ledger can hold. The set is fixed; a new + * resource needs a new member here and a payload in `RunResourcePayloads`. + * + * `acp_runtime` is a composite of the runtime, the session handle, and the + * child process. `runtime.close()` is the one release boundary for all three. + * + * These are NOT resources and never enter the ledger: `sessionKey`, the + * environment lease, the realized execution target, the cleanup registrations, + * the reuse-store entries, the three OTel spans, the turn timer, the child + * stderr flush state, and the idle timers. + */ +export type ResourceId = + | "acp_runtime" + | "staged_runtime" + | "control_bridge" + | "agent_bridge" + | "managed_home" + | "staging_lease"; + +/** + * The outcome the ledger records for a resource when it leaves the ledger. + * `finalized` means the run released the resource here. `transferred` means the + * run handed ownership to settlement. + */ +export type ResourceDisposition = "finalized" | "transferred"; + +/** + * The lifetime a cleanup belongs to. + * + * A `startup_rollback` cleanup runs only when startup fails, to undo a partial + * bring-up. A `per_run` cleanup runs at settlement, at the end of the run. A + * successful `seal()` promotes each promotable `startup_rollback` entry to + * `per_run`, because the resource now lives through the whole run. + */ +export type CleanupScope = "startup_rollback" | "per_run"; + +// --------------------------------------------------------------------------- +// Resource payloads +// --------------------------------------------------------------------------- + +/** + * The `acp_runtime` composite. `runtime.close()` releases the runtime, the + * session handle, and the child process together. Maps to the runtime, the + * session handle, and the spawned child that `execute.ts` builds in + * `buildRuntime`. + */ +export interface AcpRuntimeResource { + readonly runtime: AcpRuntime; + readonly sessionHandle: AcpRuntimeHandle; + /** The pid of the spawned agent child, or null before the first spawn. */ + readonly childProcessPid: number | null; +} + +/** + * The staged in-sandbox runtime and its one-time host-side cleanup. Maps to + * `AcpxPreparedRuntime.stagedRuntime` and `remoteStagingDispose`. + */ +export interface StagedRuntimeResource { + readonly stagedRuntime: PreparedAdapterExecutionTargetRuntime; + /** Remove the host-side staged temp. Fired only when the stage is dropped. */ + readonly disposeStaged: () => Promise; +} + +/** + * The per-adapter managed-home seam. Maps to + * `AcpxPreparedRuntime.remoteManagedHomeTeardown` and `remoteStagingEnvDelta`. + */ +export interface ManagedHomeResource { + /** Per-run copy-back hook. Runs the auth copy-back on every exit path. */ + readonly teardown: () => Promise; + /** The env keys the seam mutated on this run. */ + readonly envDelta: Record; +} + +/** + * The per-session staging lease. Maps to + * `AcpxPreparedRuntime.sessionStagingLeaseRelease`. + */ +export interface StagingLeaseResource { + /** Release the staging lease so a later overlapping run can re-stage. */ + readonly release: () => void; +} + +/** + * Maps each resource id to its payload. The `control_bridge` is the Paperclip + * control-plane bridge; the `agent_bridge` is the agent process-session bridge. + */ +export interface RunResourcePayloads { + readonly acp_runtime: AcpRuntimeResource; + readonly staged_runtime: StagedRuntimeResource; + readonly control_bridge: AdapterExecutionTargetPaperclipBridgeHandle; + readonly agent_bridge: AdapterExecutionTargetProcessSessionBridgeHandle; + readonly managed_home: ManagedHomeResource; + readonly staging_lease: StagingLeaseResource; +} + +/** One resource the caller registers into the ledger. */ +export interface RunResourceRegistration { + readonly id: Id; + readonly payload: RunResourcePayloads[Id]; + readonly scope: CleanupScope; + /** + * Marks a `startup_rollback` entry that `seal()` promotes to `per_run`. + * Default true. Set false to pin the entry as rollback-only. Ignored for a + * `per_run` entry. + */ + readonly promotable?: boolean; + /** + * The ledger calls this once when it finalizes or transfers the resource. It + * records the disposition. It runs no release itself. + */ + readonly onDisposition?: (disposition: ResourceDisposition) => void; +} + +/** One resource the settlement claim carries out of the ledger. */ +export interface SettledResourceEntry { + readonly id: Id; + readonly payload: RunResourcePayloads[Id]; + readonly scope: CleanupScope; + readonly disposition: ResourceDisposition; +} + +// --------------------------------------------------------------------------- +// Ledger surface and its two branded results +// --------------------------------------------------------------------------- + +declare const READY_RUN_RESOURCES: unique symbol; +declare const CONSUMED_RUN_RESOURCES: unique symbol; + +/** + * The sealed, validated resource set. Only `AcquiredRunResources.seal()` + * produces this brand. The turn reads its resources through this handle. + */ +export interface ReadyRunResources { + readonly [READY_RUN_RESOURCES]: true; + /** The payload for a resource id, or undefined when the slot is empty. */ + get(id: Id): RunResourcePayloads[Id] | undefined; + /** The scope of a resource id after promotion, or undefined when empty. */ + scopeOf(id: ResourceId): CleanupScope | undefined; + /** True when the slot holds a resource. */ + has(id: ResourceId): boolean; + /** The required ids that `seal()` validated. */ + readonly sealed: readonly ResourceId[]; +} + +/** + * The one-time settlement claim. Only `AcquiredRunResources.takeForSettlement()` + * produces this brand. Settlement releases the resources this claim carries. + */ +export interface ConsumedRunResources { + readonly [CONSUMED_RUN_RESOURCES]: true; + /** The resources, each with its final scope and disposition. */ + entries(): readonly SettledResourceEntry[]; +} + +/** + * The write surface the startup path holds while it acquires resources. + * `register` is the only writer. `seal` and `takeForSettlement` end the write + * phase. The state machine is `open -> sealed -> consumed` or `open -> + * consumed`. + */ +export interface AcquiredRunResources { + register(registration: RunResourceRegistration): void; + seal(required: readonly ResourceId[]): ReadyRunResources; + takeForSettlement(): ConsumedRunResources; +} + +/** The error the ledger throws on an out-of-order or duplicate operation. */ +export class LedgerStateError extends Error { + constructor(message: string) { + super(message); + this.name = "LedgerStateError"; + } +} + +// --------------------------------------------------------------------------- +// Startup, turn, and settlement outcomes +// --------------------------------------------------------------------------- + +/** + * The four phases a pre-turn failure can occur in. A pre-turn failure happens + * after the ledger acquires resources but before the turn runs. + */ +export interface PreTurnFailedCause { + readonly kind: "pre_turn_failed"; + readonly phase: + | "runtime_create" + | "handshake" + | "session_handle_missing" + | "session_configuration"; + readonly error: Error; +} + +/** The turn ran and failed with an error. */ +export interface TurnFailedCause { + readonly kind: "turn_failed"; + readonly error: Error; +} + +/** The caller cancelled the turn. */ +export interface TurnCancelledCause { + readonly kind: "turn_cancelled"; + readonly reason: string; +} + +/** The turn hit its wall-clock timeout. */ +export interface TurnTimedOutCause { + readonly kind: "turn_timed_out"; + readonly timeoutSec: number; +} + +/** A cause that the turn itself produced. Disjoint from `PreTurnFailedCause`. */ +export type TurnCause = TurnFailedCause | TurnCancelledCause | TurnTimedOutCause; + +/** Any cause that leads a run into settlement. */ +export type SettlementCause = PreTurnFailedCause | TurnCause; + +/** + * Startup produced a ready runtime. The turn can run against these resources. + */ +export interface StartupReady { + readonly kind: "ready"; + readonly resources: ReadyRunResources; + readonly context: AcpRunContext; +} + +/** + * Startup failed after it acquired resources, so the run must settle them. A + * build failure and a partial-bridge failure do NOT reach here — they throw, + * and the coordinator owns that rollback (Phase 20). + */ +export interface StartupSettle { + readonly kind: "settle"; + readonly cause: PreTurnFailedCause; + readonly resources: ConsumedRunResources; +} + +export type StartupResult = StartupReady | StartupSettle; + +/** + * The turn finished and released everything in place. It carries no resources, + * so settlement has nothing to do. + */ +export interface FinalizedTurn { + readonly kind: "finalized"; +} + +/** The turn failed. Settlement releases the carried resources. */ +export interface FailedTurn { + readonly kind: "failed"; + readonly cause: TurnFailedCause; + readonly resources: ConsumedRunResources; +} + +/** The caller cancelled the turn. Settlement releases the carried resources. */ +export interface CancelledTurn { + readonly kind: "cancelled"; + readonly cause: TurnCancelledCause; + readonly resources: ConsumedRunResources; +} + +/** The turn timed out. Settlement releases the carried resources. */ +export interface TimedOutTurn { + readonly kind: "timed_out"; + readonly cause: TurnTimedOutCause; + readonly resources: ConsumedRunResources; +} + +/** The four ways a turn completes. Only `FinalizedTurn` is resourceless. */ +export type TurnCompletion = FinalizedTurn | FailedTurn | CancelledTurn | TimedOutTurn; + +// --------------------------------------------------------------------------- +// Run site +// --------------------------------------------------------------------------- + +/** The host site runs on the host. The sandbox site runs in a remote runner. */ +export type RunSiteKind = "host" | "sandbox"; + +/** + * The synchronous plan a site produces before the fingerprint build. `plan()` + * must be synchronous, because `sessionCwd` is a fingerprint input. + */ +export interface SitePlan { + /** + * The cwd the session binds to and the fingerprint reads. `placeWorkspace` is + * not its source; the site derives it before it places the workspace. + */ + readonly sessionCwd: string; + /** + * The host-only spawn cwd for the relay proxy on the remote process-session + * lane. Undefined on every other lane. Maps to `AcpxPreparedRuntime. + * hostSpawnCwd`. + */ + readonly spawnCwd: string | undefined; + /** True only for a sandbox target, where the run emits real OTel spans. */ + readonly tracingEnabled: boolean; +} + +/** One referenced project whose staging into the sandbox failed. */ +export interface ReferencedProjectStagingFailure { + readonly projectId: string; +} + +/** + * The result of placing the workspace. Carries the referenced-project staging + * failures the run reports back on its result. + */ +export interface PlacedWorkspace { + readonly referencedProjectStagingFailures: readonly ReferencedProjectStagingFailure[]; +} + +/** The transport handles the site started for the run. */ +export interface RunSiteTransport { + readonly controlBridge: AdapterExecutionTargetPaperclipBridgeHandle | null; + readonly agentBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null; +} + +/** What the host site's store can save: the live runtime and session handle. */ +export interface HostReuseCandidate { + readonly runtime: AcpRuntime; + readonly sessionHandle: AcpRuntimeHandle; +} + +/** What the sandbox site's store can save: the staged files. */ +export interface SandboxReuseCandidate { + readonly stagedRuntime: PreparedAdapterExecutionTargetRuntime; +} + +export type RunSiteReuseCandidate = HostReuseCandidate | SandboxReuseCandidate; + +/** Construction options for a `SessionReuseStore`. */ +export interface SessionReuseStoreOptions { + /** + * The optional per-entry idle timeout, in milliseconds. When set, `evictIdle` + * drops an entry that stayed idle longer than this bound. + */ + readonly idleMs?: number; +} + +/** + * The per-site store that saves a reuse candidate across runs. The store is + * generic in the saved type, so a generic `save` cannot know what a site saves. + * `RunSite.reuseCandidate()` names the concrete type per site. + */ +export interface SessionReuseStore { + borrow(sessionKey: string): T | undefined; + save(sessionKey: string, entry: T): void; + /** + * Identity-guarded removal that fires the idempotent release. It removes the + * key synchronously, so a later reuse never reads a discarded entry. It + * returns a promise the caller can await when the release must finish before + * the run releases a downstream resource (e.g. the staging lease). + */ + discard(sessionKey: string): void | Promise; + /** + * Run-start idle sweep. It removes each stale key synchronously, so + * `buildRuntime` never reuses an expired entry. It returns a promise the + * caller can await so the sweep completes before the run reads the cache. + */ + evictIdle(now: number): void | Promise; +} + +/** + * A run site. The host site and the sandbox site differ in what they place, + * what transport they start, and what their store saves. + */ +export interface RunSite { + readonly kind: RunSiteKind; + /** Synchronous. Runs before the fingerprint build. */ + plan(context: AcpRunContext): SitePlan; + placeWorkspace(context: AcpRunContext): Promise; + startTransport(context: AcpRunContext): Promise; + reuse(): SessionReuseStore; + /** Names the concrete candidate this site's store can save. */ + reuseCandidate(resources: ReadyRunResources): TReuse; + syncBack(context: AcpRunContext): Promise; +} + +// --------------------------------------------------------------------------- +// Identity, launch environment, and run context +// --------------------------------------------------------------------------- + +export type AcpxRunMode = "persistent" | "oneshot"; +export type AcpxPermissionMode = "approve-all" | "approve-reads" | "deny-all"; +export type AcpxNonInteractivePermissions = "deny" | "fail"; + +/** One MCP server, in the shape the fingerprint reads. */ +export interface McpServerIdentity { + readonly name: string; + readonly url: string; + readonly connectionId: string; +} + +/** The Paperclip Claude settings the fingerprint reads. */ +export interface PaperclipClaudeSettingsIdentity { + readonly allow: readonly string[]; + readonly additionalDirectories: readonly string[]; + readonly defaultMode: string; +} + +/** + * The 17 fields the session fingerprint hashes. Company, agent, and task + * identifiers are NOT here; they scope the outer session key only (see + * `SessionKeyIdentity`). A change to any field here invalidates a warm handle + * or a resumable session and forces a fresh launch. + */ +export interface SessionFingerprintIdentity { + readonly acpxAgent: string; + readonly agentCommand: string; + readonly cwd: string; + readonly mode: AcpxRunMode; + readonly permissionMode: AcpxPermissionMode; + readonly nonInteractivePermissions: AcpxNonInteractivePermissions; + readonly requestedModel: string; + readonly requestedThinkingEffort: string; + readonly fastMode: boolean; + readonly remoteExecutionIdentity: Record | null; + readonly additionalSourcesIdentity: Record; + readonly skillsIdentity: Record; + readonly skillPromptInstructions: string; + readonly paperclipClaudeSettings: PaperclipClaudeSettingsIdentity | null; + readonly mcpServers: readonly McpServerIdentity[]; + readonly secretManifestHash: string; + readonly adapterEnvHash: string; +} + +/** + * The company, agent, and task parts of the session key. The key form is + * `paperclip:companyId:agentId:taskKey:fingerprint`. These parts stay out of + * the fingerprint hash. + */ +export interface SessionKeyIdentity { + readonly companyId: string; + readonly agentId: string; + readonly taskKey: string; +} + +declare const RUN_SCOPED_CONTRIBUTION: unique symbol; +declare const LAUNCH_ENVIRONMENT: unique symbol; + +/** + * A launch-environment contribution that lives for one run only. The run API + * key and every bridge token are run-scoped. The brand makes this type + * non-transferable: no reuse payload type can hold it (Amendment B). + */ +export interface RunScopedContribution { + readonly scope: "run"; + readonly env: Record; + readonly [RUN_SCOPED_CONTRIBUTION]: true; +} + +/** + * A launch-environment contribution that lives for the whole session. The Codex + * auth copy-back material is session-scoped. + */ +export interface SessionScopedContribution { + readonly scope: "session"; + readonly env: Record; +} + +export type LaunchEnvironmentContribution = RunScopedContribution | SessionScopedContribution; + +/** + * The finalized launch environment. Only `finalizeLaunchEnvironment()` + * (Phase 17) produces this brand. No reuse payload type can hold it. + */ +export interface LaunchEnvironment { + readonly [LAUNCH_ENVIRONMENT]: true; + readonly env: Readonly>; +} + +/** + * The context the coordinator threads through the run. The site reads the + * execution target for its effective capabilities. + */ +export interface AcpRunContext { + readonly executionTarget: AdapterExecutionTarget; + readonly fingerprintIdentity: SessionFingerprintIdentity; + readonly keyIdentity: SessionKeyIdentity; + readonly fingerprint: string; + readonly sessionKey: string; + readonly launchEnvironment: LaunchEnvironment; +} diff --git a/packages/adapter-utils/src/acpx-engine/run-coordinator.test.ts b/packages/adapter-utils/src/acpx-engine/run-coordinator.test.ts new file mode 100644 index 0000000000..6818364b76 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-coordinator.test.ts @@ -0,0 +1,184 @@ +import { describe, expect, it } from "vitest"; +import { runAttempt, type RunPlan, type SettlementReason } from "./run-coordinator.js"; +import { createRunResourceLedger } from "./run-resource-ledger.js"; +import type { + AcquiredRunResources, + PreTurnFailedCause, + ReadyRunResources, + StartupReady, + StartupResult, + TurnCompletion, +} from "./run-contracts.js"; + +// The coordinator owns the run routing table. These tests drive it with fakes +// for the four steps (startup, turn, settle, reproduce) and assert the order and +// the routing, never the engine's real run state. + +const readyResources = {} as ReadyRunResources; + +function readyStartup(): StartupReady { + return { kind: "ready", resources: readyResources, context: {} as StartupReady["context"] }; +} + +function preTurnCause(): PreTurnFailedCause { + return { kind: "pre_turn_failed", phase: "handshake", error: new Error("handshake boom") }; +} + +function finalizedTurn(): TurnCompletion { + return { kind: "finalized" }; +} + +// A minimal plan whose steps push their names into `order`, so a test reads the +// exact sequence the coordinator ran. +function makePlan( + order: string[], + overrides: Partial> & { ledger?: AcquiredRunResources } = {}, +): RunPlan { + const ledger = overrides.ledger ?? createRunResourceLedger(); + return { + ledger, + startup: overrides.startup ?? (async (): Promise => { + order.push("startup"); + return readyStartup(); + }), + rollbackStartup: overrides.rollbackStartup ?? ((consumed) => { + order.push(`rollback:${consumed.entries().length}`); + }), + recordDisposition: overrides.recordDisposition ?? ((report) => { + order.push(`disposition:${report.records.map((r) => `${r.id}=${r.disposition}`).join(",")}`); + }), + runTurn: overrides.runTurn ?? (async (): Promise => { + order.push("turn"); + return finalizedTurn(); + }), + settle: overrides.settle ?? (async (reason: SettlementReason) => { + order.push(`settle:${reason.kind}`); + }), + reproduceResult: overrides.reproduceResult ?? (async (outcome) => { + order.push(`reproduce:${outcome.kind}`); + return "result"; + }), + }; +} + +describe("ACPX run coordinator", () => { + it("test_startup_throw_replays_startup_rollback_entries_and_rethrows", async () => { + const order: string[] = []; + // Fill a partial ledger with two startup_rollback entries that record their + // release order when the rollback replays them. + const ledger = createRunResourceLedger(); + const released: string[] = []; + ledger.register({ + id: "staged_runtime", + payload: { + stagedRuntime: {} as never, + disposeStaged: async () => { + released.push("staged_runtime"); + }, + }, + scope: "startup_rollback", + }); + ledger.register({ + id: "control_bridge", + payload: { stop: async () => { released.push("control_bridge"); } } as never, + scope: "startup_rollback", + }); + + const boom = new Error("build boom"); + const plan = makePlan(order, { + ledger, + startup: async (): Promise => { + order.push("startup"); + throw boom; + }, + // Replay every claimed entry through its own release, newest first. + rollbackStartup: async (consumed) => { + for (const entry of [...consumed.entries()].reverse()) { + if (entry.id === "staged_runtime") { + await (entry.payload as { disposeStaged: () => Promise }).disposeStaged(); + } else if (entry.id === "control_bridge") { + await (entry.payload as { stop: () => Promise }).stop(); + } + } + order.push("rollback"); + }, + }); + + // The coordinator rethrows the original startup error. + await expect(runAttempt(plan)).rejects.toBe(boom); + // It replayed the startup_rollback entries through the partial ledger... + expect(released).toEqual(["control_bridge", "staged_runtime"]); + // ...recorded a disposition report that marks each claimed entry finalized... + expect(order).toEqual([ + "startup", + "rollback", + "disposition:staged_runtime=finalized,control_bridge=finalized", + ]); + // ...and never ran the turn or the settlement. + expect(order).not.toContain("turn"); + expect(order.some((step) => step.startsWith("settle"))).toBe(false); + }); + + it("test_settle_path_returns_error_result_after_settlement", async () => { + const order: string[] = []; + const cause = preTurnCause(); + const plan = makePlan(order, { + startup: async (): Promise => { + order.push("startup"); + return { kind: "settle", cause, resources: createRunResourceLedger().takeForSettlement() }; + }, + settle: async (reason) => { + expect(reason.kind).toBe("pre_turn"); + order.push("settle"); + }, + reproduceResult: async (outcome) => { + expect(outcome.kind).toBe("pre_turn"); + order.push("reproduce"); + return "error-result"; + }, + }); + + const result = await runAttempt(plan); + expect(result).toBe("error-result"); + // The settle path never runs the turn; it settles first, then reproduces. + expect(order).toEqual(["startup", "settle", "reproduce"]); + }); + + it("test_ready_path_runs_turn_then_settles_then_reproduces_result", async () => { + const order: string[] = []; + const plan = makePlan(order); + + const result = await runAttempt(plan); + expect(result).toBe("result"); + // The ready path runs the turn, settles, then reproduces — in that order. + expect(order).toEqual(["startup", "turn", "settle:turn", "reproduce:turn"]); + }); + + it("test_coordinator_has_no_catch_around_the_turn", async () => { + // `runTurn` is contracted never to reject. This test proves the coordinator + // holds no catch around the turn: an (illegal) turn rejection propagates out + // of `runAttempt` unchanged, and neither settlement nor result reproduction + // runs. A catch around the turn would swallow the rejection and route it into + // settlement, so this rejection-propagation is the structural proof. + const order: string[] = []; + const boom = new Error("turn must never reject"); + const settleCalls: number[] = []; + const reproduceCalls: number[] = []; + const plan = makePlan(order, { + runTurn: async (): Promise => { + throw boom; + }, + settle: async () => { + settleCalls.push(1); + }, + reproduceResult: async () => { + reproduceCalls.push(1); + return "unreachable"; + }, + }); + + await expect(runAttempt(plan)).rejects.toBe(boom); + expect(settleCalls).toHaveLength(0); + expect(reproduceCalls).toHaveLength(0); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/run-coordinator.ts b/packages/adapter-utils/src/acpx-engine/run-coordinator.ts new file mode 100644 index 0000000000..487d2e1e6b --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-coordinator.ts @@ -0,0 +1,125 @@ +// The run coordinator. +// +// One `run()` coordinator owns the attempt. It sequences four steps: startup, +// turn, settlement, and result reproduction. It holds the routing table as +// data: +// +// * A build failure or a partial-bridge failure throws from startup. The +// coordinator's one catch replays the `startup_rollback` entries through the +// partial ledger, records the disposition report, and rethrows. +// * A runtime-create, handshake, missing-handle, or configuration failure +// produces a `settle` result. The coordinator settles the resources, then +// reproduces the error result. +// * Every other startup outcome produces a `ready` result. The coordinator +// runs the turn, settles the resources, then reproduces the turn result. +// +// The coordinator has no catch around the turn: the turn never rejects and +// returns a typed `TurnCompletion` (Phase 21). The external result is reproduced +// AFTER settlement, so a caller never observes a result before the run releases +// its resources. + +import type { + AcquiredRunResources, + ConsumedRunResources, + PreTurnFailedCause, + ResourceDisposition, + ResourceId, + StartupReady, + StartupResult, + TurnCompletion, +} from "./run-contracts.js"; + +/** One resource's final disposition, as the coordinator or settlement records it. */ +export interface ResourceDispositionRecord { + readonly id: ResourceId; + readonly disposition: ResourceDisposition; +} + +/** The per-resource disposition report the coordinator records at each end. */ +export interface SettlementDispositionReport { + readonly records: readonly ResourceDispositionRecord[]; +} + +/** The cause the coordinator hands to settlement. A clean turn carries no cause. */ +export type SettlementReason = + | { readonly kind: "pre_turn"; readonly cause: PreTurnFailedCause } + | { readonly kind: "turn"; readonly completion: TurnCompletion }; + +/** The outcome the coordinator hands to result reproduction. */ +export type ReproducibleOutcome = SettlementReason; + +/** + * The four steps the coordinator sequences, plus the run ledger. The engine + * supplies each step; the coordinator owns only the order and the startup + * rollback. This keeps the coordinator free of the engine's private run state. + */ +export interface RunPlan { + /** The one run ledger for the attempt. The coordinator creates it (see `runAttempt`). */ + readonly ledger: AcquiredRunResources; + /** + * Run the startup sequence. It returns a `ready` or a `settle` result, or it + * throws on a build failure or a partial-bridge failure. + */ + startup(): Promise; + /** + * Replay the `startup_rollback` entries the partial ledger carries. The + * coordinator calls it only on a startup throw, with the claimed entries. + */ + rollbackStartup(consumed: ConsumedRunResources): Promise | void; + /** Record the per-resource disposition report at the end of the run. */ + recordDisposition(report: SettlementDispositionReport): void; + /** Run the turn against the ready resources. It never rejects. */ + runTurn(ready: StartupReady): Promise; + /** + * Settle the run's resources for the given reason. The one cleanup owner. A + * synchronous return adds no awaited boundary, so a step that already settled + * inline keeps the caller's timing unchanged. + */ + settle(reason: SettlementReason): void | Promise; + /** Reproduce the external result after settlement. */ + reproduceResult(outcome: ReproducibleOutcome): TResult | Promise; +} + +/** + * Run the attempt through the routing table. The coordinator is the one owner of + * the startup-to-settlement order. + */ +export async function runAttempt(plan: RunPlan): Promise { + let startupResult: StartupResult; + try { + startupResult = await plan.startup(); + } catch (startupError) { + // The coordinator's only catch wraps startup. A build failure and a + // partial-bridge failure reach here. Claim the partial ledger, replay its + // `startup_rollback` entries, record the disposition report (each entry the + // rollback released is `finalized`), and rethrow the original error. + const consumed = plan.ledger.takeForSettlement(); + await plan.rollbackStartup(consumed); + plan.recordDisposition(finalizedReport(consumed)); + throw startupError; + } + + if (startupResult.kind === "settle") { + // A pre-turn failure after the ledger acquired resources. Settle the + // resources, then reproduce the error result AFTER settlement. + const reason: SettlementReason = { kind: "pre_turn", cause: startupResult.cause }; + const settled = plan.settle(reason); + if (settled) await settled; + return plan.reproduceResult(reason); + } + + // The ready path: run the turn, settle, then reproduce the result. There is no + // catch around the turn — `runTurn` never rejects. + const completion = await plan.runTurn(startupResult); + const reason: SettlementReason = { kind: "turn", completion }; + const settled = plan.settle(reason); + if (settled) await settled; + return plan.reproduceResult(reason); +} + +/** Build a disposition report that marks every claimed entry `finalized`. */ +function finalizedReport(consumed: ConsumedRunResources): SettlementDispositionReport { + return { + records: consumed.entries().map((entry) => ({ id: entry.id, disposition: "finalized" as const })), + }; +} diff --git a/packages/adapter-utils/src/acpx-engine/run-fault-matrix.test.ts b/packages/adapter-utils/src/acpx-engine/run-fault-matrix.test.ts new file mode 100644 index 0000000000..eccb29c366 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-fault-matrix.test.ts @@ -0,0 +1,883 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + prepareAdapterExecutionTargetRuntime, + startAdapterExecutionTargetPaperclipBridge, + startAdapterExecutionTargetProcessSessionBridge, +} from "@paperclipai/adapter-utils/execution-target"; +import { runChildProcess } from "../server-utils.js"; + +// The composed fault matrix. +// +// Each case drives the whole ACP run and asserts the run's final settlement +// disposition report. The report is the authority on which resources the run +// acquired and how each one left the run: `finalized` (the run released it) or +// `transferred` (the run saved it for reuse). For each case the file asserts: +// +// 1. the report keys equal the acquired subset of the six resource ids; +// 2. each resource carries exactly one disposition; +// 3. the external result equals the composed baseline for that exit path. +// +// The six resource ids are: acp_runtime, staged_runtime, control_bridge, +// agent_bridge, managed_home, staging_lease. The host lane acquires only +// acp_runtime; the sandbox lane acquires the staging and transport resources too. + +// Wrap the staging seam and both sandbox bridges so a test can stub them without +// changing behavior for the other tests. +vi.mock("@paperclipai/adapter-utils/execution-target", async (importActual) => { + const actual = await importActual(); + return { + ...actual, + prepareAdapterExecutionTargetRuntime: vi.fn(actual.prepareAdapterExecutionTargetRuntime), + startAdapterExecutionTargetPaperclipBridge: vi.fn(actual.startAdapterExecutionTargetPaperclipBridge), + startAdapterExecutionTargetProcessSessionBridge: vi.fn(actual.startAdapterExecutionTargetProcessSessionBridge), + }; +}); + +import { + createAcpxEngineExecutor, + type AcpxEngineExecutorOptions, +} from "./execute.js"; +import { settleAcpRun, type SettlementSteps } from "./settlement-sequence.js"; +import { createRunResourceLedger } from "./run-resource-ledger.js"; +import { LedgerStateError, type ResourceId } from "./run-contracts.js"; +import type { SettlementDispositionReport } from "./run-coordinator.js"; + +const tempRoots: string[] = []; + +async function makeTempRoot() { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-acpx-fault-")); + tempRoots.push(root); + return root; +} + +afterEach(async () => { + await Promise.all( + tempRoots.splice(0).map((root) => + fs.rm(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 }), + ), + ); +}); + +function createLocalSandboxRunner() { + let counter = 0; + return { + execute: async (input: { + command: string; + args?: string[]; + cwd?: string; + env?: Record; + stdin?: string; + timeoutMs?: number; + onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise; + onSpawn?: (meta: { pid: number; startedAt: string }) => Promise; + }) => { + counter += 1; + const command = input.command === "bash" ? "/bin/bash" : input.command; + return await runChildProcess(`acpx-sandbox-run-${counter}`, command, input.args ?? [], { + cwd: input.cwd ?? process.cwd(), + env: input.env ?? {}, + stdin: input.stdin, + timeoutSec: Math.max(1, Math.ceil((input.timeoutMs ?? 30_000) / 1000)), + graceSec: 5, + onLog: input.onLog ?? (async () => {}), + onSpawn: input.onSpawn + ? async (meta) => input.onSpawn?.({ pid: meta.pid, startedAt: meta.startedAt }) + : undefined, + }); + }, + }; +} + +const okHandle = { + backendSessionId: "backend-session", + agentSessionId: "agent-session", + runtimeSessionName: "runtime-session", +}; + +function completedTurn() { + return { + events: (async function* () {})(), + result: Promise.resolve({ status: "completed", stopReason: "end_turn" }), + cancel: async () => {}, + }; +} + +function throwingTurn() { + return { + events: (async function* () { + throw new Error("turn upstream boom"); + })(), + result: Promise.resolve({ status: "completed", stopReason: "end_turn" }), + cancel: async () => {}, + }; +} + +function cancelledTurn() { + return { + events: (async function* () {})(), + result: Promise.resolve({ status: "cancelled", stopReason: "cancelled" }), + cancel: async () => {}, + }; +} + +// A runtime whose completed turn keeps the session handshake and turn simple. +function completedRuntime() { + return { + ensureSession: async () => okHandle, + startTurn: () => completedTurn(), + setConfigOption: async () => {}, + close: async () => {}, + } as never; +} + +// A context whose only prompt-read field throws when read, so the prompt build +// fails after the session handshake and before the turn starts. +function throwingHandoffContext(): Record { + const context: Record = {}; + Object.defineProperty(context, "paperclipSessionHandoffMarkdown", { + enumerable: false, + get() { + throw new Error("prompt build boom"); + }, + }); + return context; +} + +// Stub both sandbox bridges, so a case reaches the post-build window with live +// handles without running the real bridge transport. `stopRejects` makes the +// bridge stop reject during settlement. +function stubBridges(options: { stopRejects?: boolean } = {}) { + const stop = options.stopRejects + ? vi.fn(async () => { + throw new Error("bridge stop boom"); + }) + : vi.fn(async () => {}); + vi.mocked(startAdapterExecutionTargetPaperclipBridge).mockImplementation(async () => { + return { env: {}, stop } as never; + }); + vi.mocked(startAdapterExecutionTargetProcessSessionBridge).mockImplementation(async () => { + return { agentCommand: null, stop } as never; + }); +} + +async function setupRemoteSandbox() { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + const remoteCwd = path.join(root, "remote-workspace"); + await fs.mkdir(localCwd, { recursive: true }); + await fs.mkdir(remoteCwd, { recursive: true }); + await fs.writeFile(path.join(localCwd, "hello.txt"), "hi", "utf8"); + const executionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "fake-plugin", + remoteCwd, + runner: createLocalSandboxRunner(), + }; + return { root, stateDir, localCwd, remoteCwd, executionTarget }; +} + +function remoteArgs( + stateDir: string, + localCwd: string, + executionTarget: unknown, + overrides: Record = {}, +) { + return { + agent: { id: "agent-1", companyId: "company-1" }, + runtime: {}, + config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + context: {}, + authToken: "real-run-jwt", + executionTarget, + onLog: async () => {}, + onMeta: async () => {}, + onEvent: async () => {}, + ...overrides, + }; +} + +// Seed the managed home through the staging seam and return a per-run copy-back +// so the run registers the managed_home resource. `teardownRejects` makes the +// copy-back reject during the settlement sync-back. +function managedHomeSeed(options: { teardownRejects?: boolean } = {}): AcpxEngineExecutorOptions["prepareRemoteManagedHome"] { + return async (input) => { + const stagedRuntime = await input.stage([]); + return { + stagedRuntime, + teardown: options.teardownRejects + ? async () => { + throw new Error("copy-back boom"); + } + : async () => {}, + }; + }; +} + +// Capture the final settlement disposition report the run records. +function captureDisposition() { + const reports: SettlementDispositionReport[] = []; + return { + onSettlementDisposition: (report: SettlementDispositionReport) => { + reports.push(report); + }, + last(): SettlementDispositionReport | null { + return reports.length > 0 ? reports[reports.length - 1]! : null; + }, + }; +} + +// The shared per-case assertion. It proves the report keys equal the acquired +// subset, each resource carries exactly one disposition, and the transferred id +// (if any) is the only `transferred` entry. +function assertDispositionReport( + report: SettlementDispositionReport | null, + expected: { acquired: ResourceId[]; transferred?: ResourceId | null }, +) { + expect(report, "the run must record a disposition report").not.toBeNull(); + const records = report!.records; + const ids = records.map((record) => record.id); + // Each resource carries exactly one disposition (no duplicate id). + expect(ids.length, "no resource is reported twice").toBe(new Set(ids).size); + // The report keys equal the acquired subset of the six resource ids. + expect(new Set(ids)).toEqual(new Set(expected.acquired)); + const transferred = expected.transferred ?? null; + for (const record of records) { + expect(["finalized", "transferred"]).toContain(record.disposition); + if (record.id === transferred) { + expect(record.disposition, `${record.id} must be transferred`).toBe("transferred"); + } else { + expect(record.disposition, `${record.id} must be finalized`).toBe("finalized"); + } + } +} + +// The sandbox-lane resource set before the runtime is created (staging and +// transport), and after (adds the runtime). Without a managed-home seed the +// managed_home resource is absent. +const SANDBOX_PRE_RUNTIME: ResourceId[] = ["staged_runtime", "staging_lease", "control_bridge", "agent_bridge"]; +const SANDBOX_WITH_RUNTIME: ResourceId[] = [...SANDBOX_PRE_RUNTIME, "acp_runtime"]; +const SANDBOX_WITH_MANAGED_HOME: ResourceId[] = [ + "staged_runtime", + "managed_home", + "staging_lease", + "control_bridge", + "agent_bridge", + "acp_runtime", +]; + +describe("composed ACPX run fault matrix", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + // Case 1 — a build failure throws before the ledger acquires any resource. + it("case_01_build_failure_throws_and_acquires_nothing", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + vi.mocked(prepareAdapterExecutionTargetRuntime).mockImplementationOnce(async () => { + throw new Error("staging boom"); + }); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => completedRuntime(), + onSettlementDisposition: capture.onSettlementDisposition, + }); + + await expect( + execute({ runId: "fault-build", ...remoteArgs(stateDir, localCwd, executionTarget) } as never), + ).rejects.toThrow("staging boom"); + + assertDispositionReport(capture.last(), { acquired: [] }); + }); + + // Case 2 — a partial bridge failure throws; the staged entry and the lease are + // finalized (the rollback removed the borrowed staged entry). + it("case_02_partial_bridge_failure_finalizes_staged_and_lease", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + const capture = captureDisposition(); + const stop = vi.fn(async () => {}); + vi.mocked(startAdapterExecutionTargetPaperclipBridge).mockImplementationOnce(async () => { + throw new Error("paperclip bridge boom"); + }); + vi.mocked(startAdapterExecutionTargetProcessSessionBridge).mockImplementationOnce( + async () => ({ agentCommand: null, stop }) as never, + ); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => completedRuntime(), + onSettlementDisposition: capture.onSettlementDisposition, + }); + + await expect( + execute({ runId: "fault-bridge", ...remoteArgs(stateDir, localCwd, executionTarget) } as never), + ).rejects.toThrow("paperclip bridge boom"); + + assertDispositionReport(capture.last(), { acquired: ["staged_runtime", "staging_lease"] }); + }); + + // Case 3 — a runtime-create failure settles the staging and transport resources. + // The runtime never enters the ledger. + it("case_03_runtime_create_failure_finalizes_staging_and_transport", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => { + throw new Error("createRuntime boom"); + }, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-create", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("create_runtime"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_PRE_RUNTIME }); + }); + + // Case 4 — a handshake failure closes the runtime; every resource finalizes. + it("case_04_handshake_failure_closes_runtime", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => + ({ + ensureSession: async () => { + throw new Error("ensureSession boom"); + }, + startTurn: () => completedTurn(), + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-handshake", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("ensure_session"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }); + + // Case 5 — a missing session handle closes the runtime; the result carries a + // null clearSession and no errorMeta. + it("case_05_missing_handle_closes_runtime_null_error", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => + ({ + ensureSession: async () => undefined, + startTurn: () => completedTurn(), + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-missing-handle", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("ensure_session"); + expect(result.errorCode).toBe("acpx_runtime_error"); + expect(result.clearSession).toBeUndefined(); + expect(result.errorMeta).toBeUndefined(); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }); + + // Case 6 — a configuration failure closes the runtime. + it("case_06_configuration_failure_closes_runtime", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => + ({ + ensureSession: async () => okHandle, + startTurn: () => completedTurn(), + setConfigOption: async () => { + throw new Error("setConfigOption boom"); + }, + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-configure", + ...remoteArgs(stateDir, localCwd, executionTarget, { + config: { + agent: "custom", + agentCommand: "node ./fake-acp.js", + model: "custom-model-x", + stateDir, + cwd: localCwd, + }, + }), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("configure_session"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }); + + // Case 7 — a prompt-build failure fails the turn in the prepare phase. + it("case_07_prompt_build_failure_finalizes_every_resource", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => completedRuntime(), + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-prepare", + ...remoteArgs(stateDir, localCwd, executionTarget, { context: throwingHandoffContext() }), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("prepare_turn"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }); + + // Case 8 — a resume retry runs ensureSession twice but registers the runtime + // once, so the acquired set has no doubled resource. + it("case_08_resume_retry_does_not_double_resources", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const cwd = path.join(root, "worktree"); + await fs.mkdir(cwd, { recursive: true }); + const config = { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd }; + + // A first clean host run mints the session params the second run resumes from. + const firstExecute = createAcpxEngineExecutor({ createRuntime: () => completedRuntime() }); + const first = await firstExecute({ + runId: "fault-resume-a", + agent: { id: "agent-1", companyId: "company-1" }, + runtime: {}, + config, + context: {}, + onLog: async () => {}, + onMeta: async () => {}, + } as never); + expect(first.exitCode).toBe(0); + + // The second run resumes: the resume ensureSession fails, the fresh retry + // succeeds. The runtime is created once, so acp_runtime registers once. + const capture = captureDisposition(); + let created = 0; + const secondExecute = createAcpxEngineExecutor({ + onSettlementDisposition: capture.onSettlementDisposition, + createRuntime: () => { + created += 1; + return { + ensureSession: async (session: Record) => { + if (session.resumeSessionId) throw new Error("resume session not found"); + return okHandle; + }, + startTurn: () => completedTurn(), + setConfigOption: async () => {}, + close: async () => {}, + } as never; + }, + }); + const second = await secondExecute({ + runId: "fault-resume-b", + agent: { id: "agent-1", companyId: "company-1" }, + runtime: { sessionParams: (first as { sessionParams?: unknown }).sessionParams }, + config, + context: {}, + onLog: async () => {}, + onMeta: async () => {}, + } as never); + + expect(second.exitCode).toBe(0); + // The runtime was created exactly once across the resume and the fresh retry. + expect(created).toBe(1); + // The host lane acquires only the runtime, and it appears once. + assertDispositionReport(capture.last(), { acquired: ["acp_runtime"] }); + }); + + // Case 9 — a turn exception finalizes every resource (the staged runtime is + // discarded, not transferred). + it("case_09_turn_exception_discards_staged_runtime", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => + ({ + ensureSession: async () => okHandle, + startTurn: () => throwingTurn(), + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-turn", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("turn"); + expect(result.errorCode).toBe("acpx_turn_failed"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }); + + // Case 10 — a turn timeout finalizes every resource. + it("case_10_turn_timeout_finalizes_every_resource", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + let releaseTurn: (() => void) | null = null; + const turnCancelled = new Promise((resolve) => { + releaseTurn = resolve; + }); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => + ({ + ensureSession: async () => okHandle, + startTurn: () => ({ + events: (async function* () { + await turnCancelled; + })(), + result: turnCancelled.then(() => ({ status: "cancelled", stopReason: "cancelled" })), + cancel: async () => { + releaseTurn?.(); + }, + }), + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-timeout", + ...remoteArgs(stateDir, localCwd, executionTarget, { + config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd, timeoutSec: 1 }, + }), + } as never); + + expect(result.timedOut).toBe(true); + expect(result.errorCode).toBe("acpx_timeout"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }, 15_000); + + // Case 11 — a cancelled turn finalizes every resource. + it("case_11_turn_cancellation_finalizes_every_resource", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => + ({ + ensureSession: async () => okHandle, + startTurn: () => cancelledTurn(), + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-cancel", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.status).toBe("cancelled"); + assertDispositionReport(capture.last(), { acquired: SANDBOX_WITH_RUNTIME }); + }); + + // Case 12 — a clean success saves the staged runtime (transferred); every other + // resource finalizes. A managed-home seed exercises the full six-resource set. + it("case_12_clean_success_transfers_staged_runtime", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => completedRuntime(), + prepareRemoteManagedHome: managedHomeSeed(), + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-success", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(0); + expect(result.resultJson?.status).toBe("completed"); + assertDispositionReport(capture.last(), { + acquired: SANDBOX_WITH_MANAGED_HOME, + transferred: "staged_runtime", + }); + }); + + // Case 13 — a bridge stop that rejects during settlement never changes the + // result or the report (allSettled absorbs the fault). + it("case_13_bridge_stop_reject_does_not_change_result", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges({ stopRejects: true }); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => completedRuntime(), + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-bridge-stop", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(0); + expect(result.resultJson?.status).toBe("completed"); + assertDispositionReport(capture.last(), { + acquired: SANDBOX_WITH_RUNTIME, + transferred: "staged_runtime", + }); + }); + + // Case 14 — a sync-back copy-back that rejects during settlement never changes + // the result or the report. + it("case_14_sync_back_failure_does_not_change_result", async () => { + const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox(); + stubBridges(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + stagedRuntimes: new Map(), + stagingLocks: new Map(), + createRuntime: () => completedRuntime(), + prepareRemoteManagedHome: managedHomeSeed({ teardownRejects: true }), + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-sync-back", + ...remoteArgs(stateDir, localCwd, executionTarget), + } as never); + + expect(result.exitCode).toBe(0); + expect(result.resultJson?.status).toBe("completed"); + assertDispositionReport(capture.last(), { + acquired: SANDBOX_WITH_MANAGED_HOME, + transferred: "staged_runtime", + }); + }); + + // Case 15 — a concurrent double settlement is a structural error: the second + // claim of the ledger throws a LedgerStateError. + it("case_15_concurrent_double_settlement_throws_ledger_state_error", async () => { + const ledger = createRunResourceLedger(); + ledger.register({ + id: "acp_runtime", + payload: { runtime: {}, sessionHandle: {}, childProcessPid: null } as never, + scope: "per_run", + }); + const noopSteps: SettlementSteps = { + reuseCandidate: () => null, + endSession: () => {}, + settleReuse: () => {}, + stopTransport: () => {}, + syncBack: () => {}, + releaseStagingLease: () => {}, + recordError: () => {}, + }; + + // The first settlement claims the ledger and produces the report. + const report = await settleAcpRun(ledger, null, noopSteps); + assertDispositionReport(report, { acquired: ["acp_runtime"] }); + + // A second settlement of the same ledger is a structural error. + await expect(settleAcpRun(ledger, null, noopSteps)).rejects.toBeInstanceOf(LedgerStateError); + }); + + // Case 16 — the host-lane credential gate. Constructed gate-passed: the runtime + // transfers and never closes. Live host path (gate failed): the runtime closes + // and never transfers. + it("case_16_host_credential_gate_transfer_versus_close", async () => { + // Constructed gate-passed condition: a host candidate with no live run-scoped + // credential. The reuse decision saves, so the runtime transfers and endSession + // never closes it. + const ledger = createRunResourceLedger(); + ledger.register({ + id: "acp_runtime", + payload: { runtime: {}, sessionHandle: {}, childProcessPid: null } as never, + scope: "per_run", + }); + let closed = false; + const gatePassedSteps: SettlementSteps = { + reuseCandidate: () => ({ kind: "host", causePermitsSave: true, liveRunScopedCredentials: [] }), + endSession: (_slots, decision) => { + if (decision.kind !== "save") closed = true; + }, + settleReuse: () => {}, + stopTransport: () => {}, + syncBack: () => {}, + releaseStagingLease: () => {}, + recordError: () => {}, + }; + const gatePassedReport = await settleAcpRun(ledger, null, gatePassedSteps); + expect(closed).toBe(false); + assertDispositionReport(gatePassedReport, { acquired: ["acp_runtime"], transferred: "acp_runtime" }); + + // Live host path: the engine populates the run-scoped credential, so the gate + // fails, the runtime closes, and it never transfers. + const root = await makeTempRoot(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + createRuntime: () => completedRuntime(), + onSettlementDisposition: capture.onSettlementDisposition, + }); + const result = await execute({ + runId: "fault-host-gate", + agent: { id: "agent-1", companyId: "company-1" }, + runtime: {}, + config: { + agent: "custom", + agentCommand: "node ./fake-acp.js", + stateDir: path.join(root, "state"), + mode: "persistent", + warmHandleIdleMs: 60_000, + }, + context: {}, + onLog: async () => {}, + onMeta: async () => {}, + } as never); + + expect(result.exitCode).toBe(0); + // The live host path closes and never transfers the runtime. + assertDispositionReport(capture.last(), { acquired: ["acp_runtime"], transferred: null }); + }); + + // Case 17 — a host-lane turn failure finalizes (closes) the runtime. + it("case_17_host_turn_failure_finalizes_runtime", async () => { + const root = await makeTempRoot(); + const capture = captureDisposition(); + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + createRuntime: () => + ({ + ensureSession: async () => okHandle, + startTurn: () => throwingTurn(), + setConfigOption: async () => {}, + close: async () => {}, + }) as never, + onSettlementDisposition: capture.onSettlementDisposition, + }); + + const result = await execute({ + runId: "fault-host-turn-fail", + agent: { id: "agent-1", companyId: "company-1" }, + runtime: {}, + config: { + agent: "custom", + agentCommand: "node ./fake-acp.js", + stateDir: path.join(root, "state"), + mode: "persistent", + warmHandleIdleMs: 60_000, + }, + context: {}, + onLog: async () => {}, + onMeta: async () => {}, + } as never); + + expect(result.exitCode).toBe(1); + expect(result.resultJson?.phase).toBe("turn"); + assertDispositionReport(capture.last(), { acquired: ["acp_runtime"], transferred: null }); + }); +}); + +describe("composed ACPX run per-phase telemetry", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it("emits a run.phase.timing event for each lifecycle phase a clean host run reaches", async () => { + const root = await makeTempRoot(); + const events: Array<{ eventType: string; payload?: Record }> = []; + const execute = createAcpxEngineExecutor({ + warmHandles: new Map(), + createRuntime: () => completedRuntime(), + }); + + const result = await execute({ + runId: "phase-telemetry", + agent: { id: "agent-1", companyId: "company-1" }, + runtime: {}, + config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir: path.join(root, "state") }, + context: {}, + onLog: async () => {}, + onMeta: async () => {}, + onEvent: async (event: { eventType: string; payload?: Record }) => { + events.push(event); + }, + } as never); + + expect(result.exitCode).toBe(0); + const phases = events + .filter((event) => event.eventType === "run.phase.timing") + .map((event) => String(event.payload?.phase)); + // A clean host run runs the runtime, session, prepare, turn, and settlement + // phases. Each phase emits a phase-timing event. + for (const phase of ["create_runtime", "ensure_session", "prepare_turn", "turn", "end_session"]) { + expect(phases, `missing phase telemetry for ${phase}`).toContain(phase); + } + // Every phase-timing event carries exactly the three closed fields. + for (const event of events.filter((entry) => entry.eventType === "run.phase.timing")) { + expect(Object.keys(event.payload ?? {}).sort()).toEqual(["durationMs", "outcome", "phase"]); + } + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/run-resource-ledger.test.ts b/packages/adapter-utils/src/acpx-engine/run-resource-ledger.test.ts new file mode 100644 index 0000000000..20c8a1fda2 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-resource-ledger.test.ts @@ -0,0 +1,185 @@ +import { describe, expect, it, vi } from "vitest"; +import { createRunResourceLedger, createScopedCleanupRegistry } from "./run-resource-ledger.js"; +import { LedgerStateError } from "./run-contracts.js"; +import type { RunResourcePayloads } from "./run-contracts.js"; + +// A typed placeholder payload for a resource whose value the test never reads. +function stub(): T { + return undefined as unknown as T; +} + +describe("run resource ledger", () => { + it("test_register_throws_on_filled_slot", () => { + const ledger = createRunResourceLedger(); + ledger.register({ id: "staging_lease", scope: "per_run", payload: { release: () => {} } }); + + expect(() => + ledger.register({ id: "staging_lease", scope: "per_run", payload: { release: () => {} } }), + ).toThrow(LedgerStateError); + }); + + it("test_register_throws_after_seal_and_after_take", () => { + const sealed = createRunResourceLedger(); + sealed.register({ id: "staging_lease", scope: "per_run", payload: { release: () => {} } }); + sealed.seal(["staging_lease"]); + expect(() => + sealed.register({ + id: "managed_home", + scope: "per_run", + payload: stub(), + }), + ).toThrow(LedgerStateError); + + const taken = createRunResourceLedger(); + taken.register({ id: "staging_lease", scope: "per_run", payload: { release: () => {} } }); + taken.takeForSettlement(); + expect(() => + taken.register({ + id: "managed_home", + scope: "per_run", + payload: stub(), + }), + ).toThrow(LedgerStateError); + }); + + it("test_seal_validates_site_declared_required_slots", () => { + const ledger = createRunResourceLedger(); + ledger.register({ + id: "acp_runtime", + scope: "per_run", + payload: stub(), + }); + + // The site declares control_bridge required, but it is not registered, so + // seal throws and leaves the ledger open. + expect(() => ledger.seal(["acp_runtime", "control_bridge"])).toThrow(LedgerStateError); + + // The failed seal left the ledger open, so the missing slot can be filled. + ledger.register({ + id: "control_bridge", + scope: "per_run", + payload: stub(), + }); + const ready = ledger.seal(["acp_runtime", "control_bridge"]); + + expect(ready.has("acp_runtime")).toBe(true); + expect(ready.has("control_bridge")).toBe(true); + expect([...ready.sealed]).toEqual(["acp_runtime", "control_bridge"]); + }); + + it("test_seal_promotes_startup_rollback_entries_to_per_run", () => { + const ledger = createRunResourceLedger(); + // Promotable by default. + ledger.register({ + id: "staged_runtime", + scope: "startup_rollback", + payload: stub(), + }); + // Pinned as rollback-only. + ledger.register({ + id: "control_bridge", + scope: "startup_rollback", + promotable: false, + payload: stub(), + }); + ledger.register({ + id: "acp_runtime", + scope: "per_run", + payload: stub(), + }); + + const ready = ledger.seal(["staged_runtime", "control_bridge", "acp_runtime"]); + + expect(ready.scopeOf("staged_runtime")).toBe("per_run"); + expect(ready.scopeOf("control_bridge")).toBe("startup_rollback"); + expect(ready.scopeOf("acp_runtime")).toBe("per_run"); + }); + + it("test_take_for_settlement_is_synchronous_one_time_claim", () => { + const ledger = createRunResourceLedger(); + ledger.register({ + id: "acp_runtime", + scope: "per_run", + payload: stub(), + }); + + const consumed = ledger.takeForSettlement(); + + // Synchronous: the claim comes back directly, not as a promise. + expect(consumed).not.toBeInstanceOf(Promise); + expect(consumed.entries().map((entry) => entry.id)).toContain("acp_runtime"); + + // One-time: a second claim throws. + expect(() => ledger.takeForSettlement()).toThrow(LedgerStateError); + }); + + it("test_second_take_throws_ledger_state_error_before_any_side_effect", () => { + const ledger = createRunResourceLedger(); + const onDisposition = vi.fn(); + ledger.register({ + id: "staging_lease", + scope: "per_run", + payload: { release: () => {} }, + onDisposition, + }); + + const first = ledger.takeForSettlement(); + expect(first.entries().map((entry) => entry.id)).toEqual(["staging_lease"]); + expect(onDisposition).toHaveBeenCalledTimes(1); + expect(onDisposition).toHaveBeenCalledWith("transferred"); + + let thrown: unknown; + try { + ledger.takeForSettlement(); + } catch (err) { + thrown = err; + } + + expect(thrown).toBeInstanceOf(LedgerStateError); + // The second claim threw before it marked a disposition a second time. + expect(onDisposition).toHaveBeenCalledTimes(1); + }); + + it("scoped cleanup registry runs and promotes cleanups by scope", async () => { + const registry = createScopedCleanupRegistry(); + const order: string[] = []; + registry.add("startup_rollback", () => { + order.push("rollback-a"); + }); + registry.add("startup_rollback", async () => { + order.push("rollback-b"); + }); + registry.add("per_run", () => { + order.push("per-run"); + }); + expect(registry.size("startup_rollback")).toBe(2); + + registry.promote("startup_rollback", "per_run"); + expect(registry.size("startup_rollback")).toBe(0); + expect(registry.size("per_run")).toBe(3); + + await registry.run("per_run"); + + // Reverse registration order (last in, first out). + expect(order).toEqual(["rollback-b", "rollback-a", "per-run"]); + expect(registry.size("per_run")).toBe(0); + }); + + it("scoped cleanup registry runs every cleanup even when one fails", async () => { + const registry = createScopedCleanupRegistry(); + const ran: string[] = []; + registry.add("per_run", () => { + ran.push("first"); + }); + registry.add("per_run", () => { + throw new Error("boom"); + }); + registry.add("per_run", () => { + ran.push("third"); + }); + + await expect(registry.run("per_run")).rejects.toBeInstanceOf(AggregateError); + // The failing cleanup did not skip the others. + expect(ran).toEqual(["third", "first"]); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/run-resource-ledger.ts b/packages/adapter-utils/src/acpx-engine/run-resource-ledger.ts new file mode 100644 index 0000000000..3e1e978039 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-resource-ledger.ts @@ -0,0 +1,201 @@ +// The run resource ledger and the scoped cleanup registry. +// +// The ledger is the single owner of the run resources between acquisition and +// settlement. `register` is the only writer. `seal` ends acquisition and +// validates the required set. `takeForSettlement` makes the one-time claim that +// hands the resources to settlement. The state machine is `open -> sealed -> +// consumed` or `open -> consumed`. +// +// The scoped cleanup registry holds cleanup thunks by lifetime. The coordinator +// runs the `startup_rollback` scope when startup fails and the `per_run` scope +// at settlement. A successful seal promotes the rollback cleanups to per_run. + +import { LedgerStateError } from "./run-contracts.js"; +import type { + AcquiredRunResources, + CleanupScope, + ConsumedRunResources, + ReadyRunResources, + ResourceDisposition, + ResourceId, + RunResourcePayloads, + RunResourceRegistration, + SettledResourceEntry, +} from "./run-contracts.js"; + +type LedgerState = "open" | "sealed" | "consumed"; + +interface LedgerSlot { + readonly id: ResourceId; + readonly payload: RunResourcePayloads[ResourceId]; + scope: CleanupScope; + readonly promotable: boolean; + readonly onDisposition?: (disposition: ResourceDisposition) => void; +} + +/** + * Create a run resource ledger. The ledger starts `open`. `register` fills at + * most one slot per resource id. `seal` validates the required set and promotes + * promotable `startup_rollback` entries to `per_run`. `takeForSettlement` makes + * the one-time settlement claim. + */ +export function createRunResourceLedger(): AcquiredRunResources { + const slots = new Map(); + let state: LedgerState = "open"; + let sealedRequired: readonly ResourceId[] = []; + + function readyView(): ReadyRunResources { + const view = { + get(id: ResourceId): RunResourcePayloads[ResourceId] | undefined { + return slots.get(id)?.payload; + }, + scopeOf(id: ResourceId): CleanupScope | undefined { + return slots.get(id)?.scope; + }, + has(id: ResourceId): boolean { + return slots.has(id); + }, + sealed: sealedRequired, + }; + return view as unknown as ReadyRunResources; + } + + return { + register(registration: RunResourceRegistration): void { + if (state !== "open") { + throw new LedgerStateError( + state === "sealed" + ? "cannot register a resource after the ledger is sealed" + : "cannot register a resource after the ledger is taken for settlement", + ); + } + if (slots.has(registration.id)) { + throw new LedgerStateError(`resource slot "${registration.id}" is already filled`); + } + slots.set(registration.id, { + id: registration.id, + payload: registration.payload, + scope: registration.scope, + promotable: registration.promotable ?? true, + onDisposition: registration.onDisposition, + }); + }, + + seal(required: readonly ResourceId[]): ReadyRunResources { + if (state !== "open") { + throw new LedgerStateError( + state === "sealed" + ? "the ledger is already sealed" + : "cannot seal the ledger after it is taken for settlement", + ); + } + // Validate the required set first. A failed seal must leave the ledger + // open and unchanged, so the caller can register the missing slot and + // seal again. + const missing = required.filter((id) => !slots.has(id)); + if (missing.length > 0) { + throw new LedgerStateError(`seal is missing required resource(s): ${missing.join(", ")}`); + } + // Promote every promotable startup_rollback entry to per_run in one pass. + // The resource now lives through the whole run, so settlement owns it. + for (const slot of slots.values()) { + if (slot.scope === "startup_rollback" && slot.promotable) { + slot.scope = "per_run"; + } + } + state = "sealed"; + sealedRequired = [...required]; + return readyView(); + }, + + takeForSettlement(): ConsumedRunResources { + // The one-time claim. Check the state first, so a second claim throws + // before it can mark a disposition or hand the resources out again. + if (state === "consumed") { + throw new LedgerStateError("run resources are already taken for settlement"); + } + state = "consumed"; + const entries: SettledResourceEntry[] = []; + for (const slot of slots.values()) { + slot.onDisposition?.("transferred"); + entries.push({ + id: slot.id, + payload: slot.payload, + scope: slot.scope, + disposition: "transferred", + }); + } + const consumed = { entries: () => entries }; + return consumed as unknown as ConsumedRunResources; + }, + }; +} + +/** + * A registry of cleanup thunks, grouped by lifetime scope. The coordinator adds + * a cleanup under a scope, promotes a scope's cleanups to another scope, and + * runs a scope's cleanups. + */ +export interface ScopedCleanupRegistry { + /** Add a cleanup under a scope. */ + add(scope: CleanupScope, cleanup: () => void | Promise): void; + /** Move every cleanup from one scope to another, in registration order. */ + promote(from: CleanupScope, to: CleanupScope): void; + /** The count of cleanups in a scope. */ + size(scope: CleanupScope): number; + /** + * Run and clear a scope's cleanups in reverse registration order. Run every + * cleanup even when one fails. Throw an `AggregateError` at the end when one + * or more failed, so no error is lost and no cleanup is skipped. + */ + run(scope: CleanupScope): Promise; +} + +/** Create an empty scoped cleanup registry. */ +export function createScopedCleanupRegistry(): ScopedCleanupRegistry { + const byScope = new Map void | Promise>>(); + + function bucket(scope: CleanupScope): Array<() => void | Promise> { + let list = byScope.get(scope); + if (!list) { + list = []; + byScope.set(scope, list); + } + return list; + } + + return { + add(scope: CleanupScope, cleanup: () => void | Promise): void { + bucket(scope).push(cleanup); + }, + + promote(from: CleanupScope, to: CleanupScope): void { + if (from === to) return; + const source = byScope.get(from); + if (!source || source.length === 0) return; + const target = bucket(to); + target.push(...source); + byScope.set(from, []); + }, + + size(scope: CleanupScope): number { + return byScope.get(scope)?.length ?? 0; + }, + + async run(scope: CleanupScope): Promise { + const list = byScope.get(scope) ?? []; + byScope.set(scope, []); + const errors: unknown[] = []; + for (let index = list.length - 1; index >= 0; index -= 1) { + try { + await list[index](); + } catch (err) { + errors.push(err); + } + } + if (errors.length > 0) { + throw new AggregateError(errors, `cleanup scope "${scope}" had ${errors.length} failure(s)`); + } + }, + }; +} diff --git a/packages/adapter-utils/src/acpx-engine/run-site-host.test.ts b/packages/adapter-utils/src/acpx-engine/run-site-host.test.ts new file mode 100644 index 0000000000..681c6552f4 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-site-host.test.ts @@ -0,0 +1,113 @@ +// Tests for the host run site (Phase 18). +// +// The host site is the null path with a warm-runtime store. These tests pin its +// warm-hit and warm-save behavior against the baseline warm sets, and confirm +// the null-path operations: host cwd plan, no staging failures, no transport, +// and no sync-back. + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { createHostRunSite, type RuntimeCacheEntry } from "./run-site-host.js"; +import type { AcpRunContext, ReadyRunResources } from "./run-contracts.js"; + +function makeEntry(id: string, lastUsedAt: number): RuntimeCacheEntry { + return { + runtime: { id, close: async () => {} } as never, + handle: { sessionKey: id } as never, + childStderrState: { logPath: null, pendingLiveLine: "" }, + processIdentitySink: { current: undefined, latest: null }, + fingerprint: `fp-${id}`, + lastUsedAt, + }; +} + +describe("host run site", () => { + beforeEach(() => { + vi.useFakeTimers(); + }); + afterEach(() => { + vi.useRealTimers(); + }); + + it("test_host_site_warm_hit_and_warm_save_match_baseline_sets", async () => { + const warmHandles = new Map(); + const closed: RuntimeCacheEntry[] = []; + const site = createHostRunSite({ + warmHandles, + now: () => Date.now(), + idleMs: 60_000, + closeWarmEntry: async (entry) => { + closed.push(entry); + }, + }); + const store = site.reuse(); + + // Warm save: the entry lands in the caller's map as the sole warm entry, the + // same set the baseline warm-save path produces (`warmHandles.size` is 1). + const entry = makeEntry("paperclip:c:a:t:fp", Date.now()); + store.save(entry.handle.sessionKey, entry); + expect(new Set(warmHandles.keys())).toEqual(new Set(["paperclip:c:a:t:fp"])); + expect(warmHandles.get("paperclip:c:a:t:fp")).toBe(entry); + // The save armed a per-entry idle timer. + expect(entry.cleanupTimer).toBeDefined(); + + // Warm hit: a borrow returns the same entry and leaves it in the map, so an + // overlapping run of the same session still reads it. The borrow cleared the + // idle timer, so the reused runtime does not expire while it is in use. + expect(store.borrow("paperclip:c:a:t:fp")).toBe(entry); + expect(warmHandles.size).toBe(1); + await vi.advanceTimersByTimeAsync(120_000); + expect(closed).toEqual([]); + expect(warmHandles.get("paperclip:c:a:t:fp")).toBe(entry); + + // A discard drops the entry and closes its runtime once, matching the + // baseline drop set (empty map, one close). + await store.discard("paperclip:c:a:t:fp"); + expect(warmHandles.size).toBe(0); + expect(closed).toEqual([entry]); + }); + + it("test_host_site_plan_and_null_path_operations", async () => { + const site = createHostRunSite({ + warmHandles: new Map(), + now: () => 0, + idleMs: 0, + closeWarmEntry: async () => {}, + }); + + const context = { + fingerprintIdentity: { cwd: "/work/dir" }, + } as unknown as AcpRunContext; + + // The plan binds the session to the host cwd, names no relay-proxy spawn cwd, + // and disables sandbox tracing. + expect(site.plan(context)).toEqual({ + sessionCwd: "/work/dir", + spawnCwd: undefined, + tracingEnabled: false, + }); + + // The host runs in place, so it places nothing, starts no transport, and + // syncs nothing back. + expect(await site.placeWorkspace(context)).toEqual({ referencedProjectStagingFailures: [] }); + expect(await site.startTransport(context)).toEqual({ controlBridge: null, agentBridge: null }); + await expect(site.syncBack(context)).resolves.toBeUndefined(); + }); + + it("test_host_site_reuse_candidate_names_runtime_and_session_handle", () => { + const site = createHostRunSite({ + warmHandles: new Map(), + now: () => 0, + idleMs: 0, + closeWarmEntry: async () => {}, + }); + + const runtime = { id: "rt" }; + const sessionHandle = { id: "handle" }; + const resources = { + get: (id: string) => + id === "acp_runtime" ? { runtime, sessionHandle, childProcessPid: 42 } : undefined, + } as unknown as ReadyRunResources; + + expect(site.reuseCandidate(resources)).toEqual({ runtime, sessionHandle }); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/run-site-host.ts b/packages/adapter-utils/src/acpx-engine/run-site-host.ts new file mode 100644 index 0000000000..6ed9d921a0 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-site-host.ts @@ -0,0 +1,146 @@ +// The host run site. +// +// The host site is the literal null path: it runs the agent on the host, with +// no sandbox, no staging, and no bridges. Its store keeps a live ACP runtime and +// session handle warm across runs of the same session, so a compatible resume +// reuses the running agent instead of a fresh spawn. This is today's warm-handle +// cache, factored behind the generic reuse store. +// +// The site owns the warm store only. The engine still drives the run body; the +// site's `plan`, `placeWorkspace`, `startTransport`, and `syncBack` are the null +// operations the host lane always did (host cwd, no staging failures, no +// transport, no sync-back). A later phase routes the whole lane through the site. + +import type { AdapterExecutionContext } from "@paperclipai/adapter-utils"; +import type { AcpRuntime, AcpRuntimeHandle } from "acpx/runtime"; +import type { + AcpRunContext, + HostReuseCandidate, + PlacedWorkspace, + ReadyRunResources, + RunSiteTransport, + SessionReuseStore, + SitePlan, +} from "./run-contracts.js"; +import { createSessionReuseStore } from "./session-reuse-store.js"; + +/** The agent child's identity, as the runtime reports it on each spawn. */ +export type AcpxAgentProcessIdentity = { pid: number; startedAt: string }; + +/** + * The mutable spawn-callback target and the last spawn identity a warm runtime + * carries across runs. A warm handle can outlive the run that created it, so the + * `current` callback stays mutable: a later heartbeat repoints it to its own + * `onSpawn`, and `latest` lets that heartbeat re-announce the running agent. + */ +export type AcpxProcessIdentitySink = { + current: AdapterExecutionContext["onSpawn"]; + latest: AcpxAgentProcessIdentity | null; +}; + +/** The live-line buffer and log path a warm runtime's child stderr carries. */ +export interface ChildStderrState { + logPath: string | null; + pendingLiveLine: string; +} + +/** + * One warm host runtime the store keeps for the next compatible resume. It is + * the value the host warm-handle map holds. The entry carries its own last-used + * time and its own optional idle timer, which the store reads and writes. + */ +export interface RuntimeCacheEntry { + runtime: AcpRuntime; + handle: AcpRuntimeHandle; + childStderrState: ChildStderrState; + processIdentitySink: AcpxProcessIdentitySink; + fingerprint: string; + lastUsedAt: number; + cleanupTimer?: ReturnType; +} + +/** Construction options for the host site. */ +export interface HostRunSiteOptions { + /** + * The warm-handle map the store operates over. The engine owns it and reuses + * it across runs, so a saved runtime stays warm for the next resume. + */ + readonly warmHandles: Map; + readonly now: () => number; + /** The warm-idle bound, in milliseconds. `0` or less disables the store. */ + readonly idleMs: number; + /** + * Close a warm runtime and flush its child stderr. The store fires it when the + * per-entry timer expires, when the run-start sweep evicts an idle entry, and + * on an explicit discard. The engine injects it, because the close reason and + * the stderr flush live in the engine module. + */ + readonly closeWarmEntry: (entry: RuntimeCacheEntry) => Promise; +} + +/** + * The host run site. Its store keeps warm runtimes; every other operation is the + * null path the host lane always ran. + */ +export interface HostRunSite { + readonly kind: "host"; + plan(context: AcpRunContext): SitePlan; + placeWorkspace(context: AcpRunContext): Promise; + startTransport(context: AcpRunContext): Promise; + reuse(): SessionReuseStore; + reuseCandidate(resources: ReadyRunResources): HostReuseCandidate; + syncBack(context: AcpRunContext): Promise; +} + +/** Create the host run site over the engine's warm-handle map. */ +export function createHostRunSite(options: HostRunSiteOptions): HostRunSite { + const store = createSessionReuseStore({ + entries: options.warmHandles, + now: options.now, + idleMs: options.idleMs, + lastUsedAt: (entry) => entry.lastUsedAt, + getTimer: (entry) => entry.cleanupTimer, + setTimer: (entry, timer) => { + entry.cleanupTimer = timer; + }, + // The host lane arms a per-entry idle timer on every warm save, so an idle + // runtime closes on its own without a later run. + perEntryTimer: true, + release: (entry) => options.closeWarmEntry(entry), + }); + + return { + kind: "host", + plan(context: AcpRunContext): SitePlan { + // The host binds the session to the run cwd. There is no relay-proxy spawn + // cwd off the host, and the host emits no sandbox trace spans. + return { + sessionCwd: context.fingerprintIdentity.cwd, + spawnCwd: undefined, + tracingEnabled: false, + }; + }, + async placeWorkspace(): Promise { + // The host runs in place, so it stages nothing and reports no referenced- + // project staging failure. + return { referencedProjectStagingFailures: [] }; + }, + async startTransport(): Promise { + // The host lane starts no control bridge and no agent bridge. + return { controlBridge: null, agentBridge: null }; + }, + reuse(): SessionReuseStore { + return store; + }, + reuseCandidate(resources: ReadyRunResources): HostReuseCandidate { + const runtime = resources.get("acp_runtime"); + if (!runtime) { + throw new Error("host run site cannot name a reuse candidate without an acp_runtime resource"); + } + return { runtime: runtime.runtime, sessionHandle: runtime.sessionHandle }; + }, + async syncBack(): Promise { + // The host runs in place, so there is nothing to sync back. + }, + }; +} diff --git a/packages/adapter-utils/src/acpx-engine/run-site-sandbox.test.ts b/packages/adapter-utils/src/acpx-engine/run-site-sandbox.test.ts new file mode 100644 index 0000000000..14ad2ff1cc --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-site-sandbox.test.ts @@ -0,0 +1,257 @@ +// Tests for the sandbox run site (Phase 19). +// +// The sandbox site owns staging, the staging lease, both bridges, the launch- +// environment contribution, and sync-back. These tests drive the site in +// isolation with fakes: they pin the ledger registrations, the bridge overlap and +// the callback-environment sequencing point, the staged-files reuse (which still +// creates the runtime), the synchronous plan, and Amendment B — no run-scoped +// contribution or bridge token escapes into the ledger or the reuse payload. + +import { describe, expect, it } from "vitest"; +import { + createSandboxRunSite, + type ManagedHomeSeamResult, + type SandboxRunSiteOptions, + type StagedRuntimeStoreEntry, +} from "./run-site-sandbox.js"; +import type { + AcpRunContext, + AcquiredRunResources, + LaunchEnvironmentContribution, + ReadyRunResources, + ResourceId, + RunResourceRegistration, + SandboxReuseCandidate, + StagedRuntimeResource, +} from "./run-contracts.js"; +import type { PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target"; + +function makeStagedRuntime(id: string): PreparedAdapterExecutionTargetRuntime { + return { + runtimeRootDir: `/remote/${id}/.paperclip-runtime/acpx`, + additionalSourceDirs: {}, + additionalSourceFailures: [], + } as unknown as PreparedAdapterExecutionTargetRuntime; +} + +function makeLedger(): { ledger: AcquiredRunResources; registered: RunResourceRegistration[] } { + const registered: RunResourceRegistration[] = []; + const ledger: AcquiredRunResources = { + register(registration) { + registered.push(registration as RunResourceRegistration); + }, + seal() { + throw new Error("seal is not exercised by these unit tests"); + }, + takeForSettlement() { + throw new Error("takeForSettlement is not exercised by these unit tests"); + }, + }; + return { ledger, registered }; +} + +function makeContext(sessionKey: string): AcpRunContext { + return { sessionKey } as unknown as AcpRunContext; +} + +/** Build a site over injectable fakes. Every option has a sensible default. */ +function makeSite(overrides: Partial = {}) { + const { ledger, registered } = makeLedger(); + const env: Record = { BASE: "1" }; + const stagedRuntimes = new Map(); + const stagingLocks = new Map>(); + const freshStaged = makeStagedRuntime("fresh"); + const bridgeCalls: string[] = []; + const options: SandboxRunSiteOptions = { + ledger, + stagedRuntimes, + stagingLocks, + now: () => 1000, + idleMs: 60_000, + sessionCwd: "/remote/workspace", + spawnCwd: "/host/workspace", + target: { kind: "remote" } as unknown as SandboxRunSiteOptions["target"], + env, + isCompatibleResume: false, + stage: async () => freshStaged, + seedManagedHome: async (stage) => { + const stagedRuntime = await stage([]); + // Repoint a managed-home env var, so the site captures the delta. + env.CODEX_HOME = "/remote/home"; + return { stagedRuntime, teardown: async () => {}, dispose: async () => {} } satisfies ManagedHomeSeamResult; + }, + disposeFreshStagedRuntime: async () => {}, + measureStageStep: (run) => run(), + publishStagedProjectHints: () => {}, + onReuseLog: async () => {}, + startPaperclipBridge: async () => { + bridgeCalls.push("paperclip:start"); + return { env: { PAPERCLIP_API_KEY: "run-token" }, stop: async () => {} } as never; + }, + startProcessSessionBridge: async ({ launchEnv }) => { + bridgeCalls.push("process-session:start"); + await launchEnv(); + return { agentCommand: null, stop: async () => {} } as never; + }, + measureBridgeStep: (_step, run) => run(), + finalizeLaunchEnv: (contributions) => { + const merged = { ...env }; + for (const contribution of contributions) Object.assign(merged, contribution.env); + return merged; + }, + onPaperclipBridgeLog: async () => {}, + stopBridges: async () => {}, + ...overrides, + }; + const site = createSandboxRunSite(options); + return { site, ledger, registered, env, stagedRuntimes, stagingLocks, freshStaged, bridgeCalls, options }; +} + +/** A minimal `ReadyRunResources` that resolves a single staged runtime. */ +function makeReady(stagedRuntime: PreparedAdapterExecutionTargetRuntime): ReadyRunResources { + const staged: StagedRuntimeResource = { stagedRuntime, disposeStaged: async () => {} }; + return { + get: (id: ResourceId) => (id === "staged_runtime" ? staged : undefined), + scopeOf: () => undefined, + has: (id: ResourceId) => id === "staged_runtime", + sealed: ["staged_runtime"], + } as unknown as ReadyRunResources; +} + +describe("sandbox run site", () => { + it("test_sandbox_site_registers_staging_bridges_and_lease_in_ledger", async () => { + const { site, registered } = makeSite(); + await site.placeWorkspace(makeContext("paperclip:c:a:t:fp")); + await site.startTransport(makeContext("paperclip:c:a:t:fp")); + + const byId = new Set(registered.map((entry) => entry.id)); + // Staging registers the staged runtime, the managed-home copy-back, and the + // per-session staging lease. Both bridge starts register their handles. + expect(byId).toEqual( + new Set(["staged_runtime", "managed_home", "staging_lease", "control_bridge", "agent_bridge"]), + ); + // The staged runtime is startup-rollback until a clean turn promotes it; the + // copy-back, the lease, and the bridges live through the whole run. + const scopeOf = (id: ResourceId) => registered.find((entry) => entry.id === id)?.scope; + expect(scopeOf("staged_runtime")).toBe("startup_rollback"); + expect(scopeOf("managed_home")).toBe("per_run"); + expect(scopeOf("staging_lease")).toBe("per_run"); + expect(scopeOf("control_bridge")).toBe("per_run"); + expect(scopeOf("agent_bridge")).toBe("per_run"); + }); + + it("test_sandbox_site_preserves_bridge_overlap_and_callback_sequencing", async () => { + const events: string[] = []; + let releasePaperclip!: () => void; + const paperclipGate = new Promise((resolve) => { + releasePaperclip = resolve; + }); + let processLaunchEnv: Record | null = null; + const { site } = makeSite({ + startPaperclipBridge: async () => { + events.push("paperclip:start"); + await paperclipGate; + events.push("paperclip:env-ready"); + return { env: { PAPERCLIP_API_KEY: "run-token" }, stop: async () => {} } as never; + }, + startProcessSessionBridge: async ({ launchEnv }) => { + events.push("process-session:start"); + // The launch awaits the sequencing point, so it observes the merged env. + processLaunchEnv = await launchEnv(); + events.push("process-session:launch"); + return { agentCommand: null, stop: async () => {} } as never; + }, + }); + + await site.placeWorkspace(makeContext("s")); + const transportPromise = site.startTransport(makeContext("s")); + // Both bridges start before the paperclip env resolves: their setup overlaps. + await Promise.resolve(); + await Promise.resolve(); + expect(events).toContain("paperclip:start"); + expect(events).toContain("process-session:start"); + expect(events).not.toContain("process-session:launch"); + + // Release the paperclip env; the process-session launch now observes the + // merged run-scoped env (the single sequencing point). + releasePaperclip(); + const transport = await transportPromise; + expect(events.indexOf("paperclip:env-ready")).toBeLessThan(events.indexOf("process-session:launch")); + expect(processLaunchEnv).toEqual({ BASE: "1", CODEX_HOME: "/remote/home", PAPERCLIP_API_KEY: "run-token" }); + expect(transport.launchEnv).toEqual(processLaunchEnv); + }); + + it("test_sandbox_site_borrow_yields_files_and_runtime_is_still_created", async () => { + const cachedRuntime = makeStagedRuntime("cached"); + const stagedRuntimes = new Map([ + [ + "paperclip:c:a:t:fp", + { + stagedRuntime: cachedRuntime, + envDelta: { CODEX_HOME: "/remote/home" }, + teardown: async () => {}, + dispose: async () => {}, + lastUsedAt: 500, + }, + ], + ]); + const { site, env } = makeSite({ stagedRuntimes, isCompatibleResume: true }); + + // A borrow reads the staged entry without removal, so an overlapping run of + // the same session still reads it. + const borrowed = site.reuse().borrow("paperclip:c:a:t:fp"); + expect(borrowed?.stagedRuntime).toBe(cachedRuntime); + expect(stagedRuntimes.has("paperclip:c:a:t:fp")).toBe(true); + + // A reuse hit yields the staged FILES (not a live runtime): `placeWorkspace` + // reports `reused` and re-applies the seam env delta, and the reuse candidate + // names the staged files only. So the run still creates the runtime and runs + // the handshake against the reused files. + await site.placeWorkspace(makeContext("paperclip:c:a:t:fp")); + expect(site.staged?.reused).toBe(true); + expect(site.staged?.stagedRuntime).toBe(cachedRuntime); + expect(env.CODEX_HOME).toBe("/remote/home"); + const candidate: SandboxReuseCandidate = site.reuseCandidate(makeReady(cachedRuntime)); + expect(candidate).toEqual({ stagedRuntime: cachedRuntime }); + expect("runtime" in candidate).toBe(false); + expect("sessionHandle" in candidate).toBe(false); + }); + + it("test_site_plan_supplies_session_cwd_spawn_cwd_and_tracing_before_the_fingerprint_build", () => { + const { site } = makeSite(); + // `plan()` is synchronous and reads no staged state, so it runs before the + // fingerprint build. It binds the session to the in-sandbox cwd, redirects the + // relay-proxy spawn to the host cwd, and enables sandbox tracing. + expect(site.plan(makeContext("s"))).toEqual({ + sessionCwd: "/remote/workspace", + spawnCwd: "/host/workspace", + tracingEnabled: true, + }); + }); + + it("test_bridge_contribution_is_not_retained_by_site_ledger_or_reuse_payload", async () => { + const { site, registered } = makeSite(); + await site.placeWorkspace(makeContext("s")); + const transport = await site.startTransport(makeContext("s")); + + // The ledger holds the bridge HANDLES, never the run-scoped token env. + const control = registered.find((entry) => entry.id === "control_bridge"); + expect(control?.payload).toBe(transport.controlBridge); + // The staged reuse payload carries the staged files only — no credential + // material and no bridge token. + const candidate = site.reuseCandidate(makeReady(site.staged!.stagedRuntime)); + expect(Object.keys(candidate)).toEqual(["stagedRuntime"]); + + // Compile-time (Amendment B): a run-scoped contribution is not assignable to + // the staged reuse payload. This function never runs; the assertion holds at + // typecheck only. A runtime call on a synthetic value would crash. + // eslint-disable-next-line @typescript-eslint/no-unused-vars + function _staged_reuse_payload_rejects_a_run_scoped_contribution( + contribution: LaunchEnvironmentContribution, + ): void { + // @ts-expect-error a launch-environment contribution is not a staged reuse payload + const payload: SandboxReuseCandidate = contribution; + void payload; + } + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/run-site-sandbox.ts b/packages/adapter-utils/src/acpx-engine/run-site-sandbox.ts new file mode 100644 index 0000000000..58f7a0f2c3 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/run-site-sandbox.ts @@ -0,0 +1,459 @@ +// The sandbox run site. +// +// The sandbox site runs the agent in a remote runner. It owns the four concerns +// the host site never has: it stages the workspace and the managed home into the +// sandbox, it holds the per-session staging lease, it starts both host-side +// bridges, and it syncs the managed home back on teardown. Its store keeps the +// staged files warm across runs of the same session, so a compatible resume +// reuses the already-staged in-sandbox runtime instead of re-shipping. +// +// The site owns the orchestration and the policy: the staging lease, the reuse +// and stale decision, the ledger registrations, the launch-environment envelope, +// the bridge overlap, and the single callback-environment sequencing point. The +// engine injects the leaf input/output primitives — the workspace stage call, the +// managed-home seam, the two bridge starts, and the step timer — so this module +// stays free of the engine's private staging helpers and the unit tests drive it +// with fakes. +// +// Amendment B (credential scope): `startTransport` builds the run-scoped bridge +// contribution and hands it to the injected `finalizeLaunchEnv`, the sole +// consumer of a contribution. The site, its ledger entries, and its reuse +// candidate never retain a run-scoped contribution or a bridge token. The staged +// reuse payload carries the staged files only. + +import type { + AdapterExecutionTarget, + AdapterExecutionTargetPaperclipBridgeHandle, + AdapterExecutionTargetProcessSessionBridgeHandle, + AdapterManagedRuntimeAsset, + PreparedAdapterExecutionTargetRuntime, +} from "@paperclipai/adapter-utils/execution-target"; +import type { + AcpRunContext, + AcquiredRunResources, + LaunchEnvironmentContribution, + PlacedWorkspace, + ReadyRunResources, + RunScopedContribution, + RunSiteTransport, + SandboxReuseCandidate, + SessionReuseStore, + SitePlan, +} from "./run-contracts.js"; +import { createSessionReuseStore } from "./session-reuse-store.js"; + +/** + * One staged in-sandbox runtime the store keeps for the next compatible resume. + * It is the value the staged-runtime map holds. The engine still owns this shape; + * the store reads its `lastUsedAt` and its `dispose` through the config + * accessors. The staged cache has no per-entry idle timer, so the entry carries + * none — the run-start sweep is the only eviction path. + */ +export interface StagedRuntimeStoreEntry { + stagedRuntime: PreparedAdapterExecutionTargetRuntime; + envDelta: Record; + teardown: (() => Promise) | null; + dispose: (() => Promise) | null; + lastUsedAt: number; +} + +/** + * The result of one workspace stage into the sandbox. The site records these + * fields on the run and, after a clean turn, into the staged store. + */ +export interface StagedWorkspace { + readonly stagedRuntime: PreparedAdapterExecutionTargetRuntime; + readonly teardown: (() => Promise) | null; + readonly dispose: (() => Promise) | null; + readonly envDelta: Record; + /** True when the run reused an already-staged runtime rather than staging fresh. */ + readonly reused: boolean; +} + +/** The leaf staging primitive the engine injects. It ships one asset set. */ +export type StageWorkspace = (assets: AdapterManagedRuntimeAsset[]) => Promise; + +/** + * The managed-home seam result, in the shape the site consumes. The engine maps + * its adapter seam onto this before it passes the seam in. + */ +export interface ManagedHomeSeamResult { + readonly stagedRuntime: PreparedAdapterExecutionTargetRuntime; + readonly teardown: (() => Promise) | null; + readonly dispose: (() => Promise) | null; +} + +/** The construction options for the sandbox site. Every value is per run. */ +export interface SandboxRunSiteOptions { + /** The run ledger. `executeAcpxEngine` creates it and passes it in (one ledger). */ + readonly ledger: AcquiredRunResources; + /** The staged-runtime map the store operates over. The engine owns it across runs. */ + readonly stagedRuntimes: Map; + /** The per-key staging mutex the lease and the sweep share. */ + readonly stagingLocks: Map>; + readonly now: () => number; + /** The staged-idle bound, in milliseconds. `0` or less disables the sweep. */ + readonly idleMs: number; + /** The in-sandbox workspace dir the session binds to (a fingerprint input). */ + readonly sessionCwd: string; + /** The host spawn cwd for the relay proxy on the remote process-session lane. */ + readonly spawnCwd: string; + /** The remote runner-backed sandbox target. */ + readonly target: AdapterExecutionTarget; + /** + * The run env the site mutates in place. Staging repoints the managed-home env + * var onto the in-sandbox home here, and the site captures the delta from it. + */ + readonly env: Record; + /** + * True when the supplied session params resume THIS staged session. Only a + * compatible resume may reuse the already-staged runtime. + */ + readonly isCompatibleResume: boolean; + /** Ship the workspace (and, via the seam, the managed home) into the sandbox. */ + readonly stage: StageWorkspace; + /** + * Seed the managed home through `stage`, repoint the home env var, and wire the + * per-run copy-back. Absent for custom agents and the shared-engine tests, where + * the site stages the workspace with no home asset. + */ + readonly seedManagedHome?: (stage: StageWorkspace) => Promise; + /** + * Remove a freshly staged in-sandbox runtime after the seam threw. The site + * fires it when the seam fails after a successful stage. + */ + readonly disposeFreshStagedRuntime: (stagedRuntime: PreparedAdapterExecutionTargetRuntime) => Promise; + /** Wrap the `stage.sync` step in the run's startup step timer. */ + readonly measureStageStep: (run: () => Promise) => Promise; + /** Publish the referenced-project workspace hints after a fresh stage. */ + readonly publishStagedProjectHints: (stagedProjectDirs: Record) => void; + readonly onReuseLog: () => Promise; + + /** Start the host-side paperclip callback bridge. */ + readonly startPaperclipBridge: ( + runtimeRootDir: string | null, + ) => Promise; + /** Start the host-side process-session bridge with a deferred launch env. */ + readonly startProcessSessionBridge: (input: { + runtimeRootDir: string | null; + launchEnv: () => Promise>; + }) => Promise; + /** Wrap each concurrent bridge start in the run's startup step timer. */ + readonly measureBridgeStep: (step: "bridge.paperclip" | "bridge.process-session", run: () => Promise) => Promise; + /** + * Finalize the run's branded launch environment from the bridge contribution. + * The engine owns `finalizeLaunchEnvironment`, so it stays the sole consumer of + * a contribution; the site calls this capability at the sequencing point. + */ + readonly finalizeLaunchEnv: (contributions: readonly LaunchEnvironmentContribution[]) => Record; + readonly onPaperclipBridgeLog: () => Promise; + + /** Stop both bridges and run the managed-home copy-back on teardown. */ + readonly stopBridges: (input: { + controlBridge: AdapterExecutionTargetPaperclipBridgeHandle | null; + agentBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null; + }) => Promise; +} + +/** The transport the sandbox site started, plus the run-scoped launch env. */ +export interface SandboxRunSiteTransport extends RunSiteTransport { + /** The finalized launch env for the process launch. */ + readonly launchEnv: Record; +} + +/** The sandbox run site. It owns staging, the lease, both bridges, and sync-back. */ +export interface SandboxRunSite { + readonly kind: "sandbox"; + plan(context: AcpRunContext): SitePlan; + placeWorkspace(context: AcpRunContext): Promise; + startTransport(context: AcpRunContext): Promise; + reuse(): SessionReuseStore; + reuseCandidate(resources: ReadyRunResources): SandboxReuseCandidate; + syncBack(context: AcpRunContext): Promise; + /** The staged workspace the run installed or reused, or null before `placeWorkspace`. */ + readonly staged: StagedWorkspace | null; + /** + * The paperclip callback bridge the run started, or null before + * `startTransport` or on the host lane. `startTransport` sets it before it + * rethrows a partial-bring-up failure, so an abandon path can stop the bridge + * that started when its sibling threw. + */ + readonly controlBridge: AdapterExecutionTargetPaperclipBridgeHandle | null; + /** The process-session bridge the run started, or null before `startTransport`. */ + readonly agentBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null; + /** + * Release the per-session staging lease `placeWorkspace` acquired, or null + * before `placeWorkspace`. An abandon path calls it to free the lease when the + * run never reaches sync-back. + */ + readonly stagingLeaseRelease: (() => void) | null; +} + +/** Create the sandbox run site over the engine's staged-runtime map and ledger. */ +export function createSandboxRunSite(options: SandboxRunSiteOptions): SandboxRunSite { + const store = createSessionReuseStore({ + entries: options.stagedRuntimes, + now: options.now, + idleMs: options.idleMs, + lastUsedAt: (entry) => entry.lastUsedAt, + // The staged cache arms no per-entry timer; the run-start sweep is the only + // eviction path, and it runs under the per-key staging lease. + release: async (entry) => { + if (entry.dispose) await entry.dispose().catch(() => {}); + }, + withEvictLease: async (key, critical) => { + const lease = await withStagingLease(options.stagingLocks, key, critical); + lease.release(); + }, + }); + + // The lease the run holds from the stage-or-reuse decision through the active + // turn until `syncBack` releases it. Held on the run, not on the store, because + // a run keeps its lease across its whole lifetime. + let leaseRelease: (() => void) | null = null; + let staged: StagedWorkspace | null = null; + let controlBridge: AdapterExecutionTargetPaperclipBridgeHandle | null = null; + let agentBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null = null; + + return { + kind: "sandbox", + + plan(): SitePlan { + // The sandbox binds the session to the in-sandbox workspace dir, redirects + // the relay proxy spawn to the host cwd, and emits real sandbox trace spans. + return { + sessionCwd: options.sessionCwd, + spawnCwd: options.spawnCwd, + tracingEnabled: true, + }; + }, + + async placeWorkspace(context: AcpRunContext): Promise { + const sessionKey = context.sessionKey; + const lease = await withStagingLease(options.stagingLocks, sessionKey, async (): Promise => { + const cached = options.isCompatibleResume ? store.borrow(sessionKey) : undefined; + if (cached) { + // Reuse the already-staged in-sandbox workspace and managed home. + // Re-apply the env keys the seam repointed and reuse the per-run + // copy-back so the copy-back cadence stays exactly per run. + Object.assign(options.env, cached.envDelta); + cached.lastUsedAt = options.now(); + await options.onReuseLog(); + return { + stagedRuntime: cached.stagedRuntime, + teardown: cached.teardown, + dispose: cached.dispose, + envDelta: cached.envDelta, + reused: true, + }; + } + // Not a compatible resume: stage fresh. Drop and dispose a stale entry + // that collides on this key first, so a fresh stage neither reuses nor + // leaks it. + const stale = options.stagedRuntimes.get(sessionKey); + if (stale) { + options.stagedRuntimes.delete(sessionKey); + if (stale.dispose) await stale.dispose().catch(() => {}); + } + const envBeforeStage = { ...options.env }; + let freshlyStaged: PreparedAdapterExecutionTargetRuntime | null = null; + const stage: StageWorkspace = async (assets) => { + const result = await options.stage(assets); + freshlyStaged = result; + return result; + }; + const seam = await options.measureStageStep(async (): Promise => { + if (options.seedManagedHome) { + try { + return await options.seedManagedHome(stage); + } catch (seamErr) { + // The seam threw after a possible successful stage. Dispose the fresh + // staged runtime so the abandoned in-sandbox managed home never leaks, + // then rethrow the seam error. + if (freshlyStaged) await options.disposeFreshStagedRuntime(freshlyStaged); + throw seamErr; + } + } + return { stagedRuntime: await stage([]), teardown: null, dispose: null }; + }); + const envDelta: Record = {}; + for (const [key, value] of Object.entries(options.env)) { + if (envBeforeStage[key] !== value) envDelta[key] = value; + } + return { + stagedRuntime: seam.stagedRuntime, + teardown: seam.teardown, + dispose: seam.dispose, + envDelta, + reused: false, + }; + }); + leaseRelease = lease.release; + staged = lease.value; + // Publish the referenced-project workspace hints (known only after staging). + const stagedProjectDirs = staged.stagedRuntime.additionalSourceDirs ?? {}; + if (Object.keys(stagedProjectDirs).length > 0) { + options.publishStagedProjectHints(stagedProjectDirs); + } + // Register the acquired resources into the ledger. A per-run copy-back and a + // staging lease live through the whole run; the staged runtime is + // startup-rollback until a clean turn promotes it. + options.ledger.register({ + id: "staged_runtime", + payload: { stagedRuntime: staged.stagedRuntime, disposeStaged: staged.dispose ?? (async () => {}) }, + scope: "startup_rollback", + }); + if (staged.teardown) { + options.ledger.register({ + id: "managed_home", + payload: { teardown: staged.teardown, envDelta: staged.envDelta }, + scope: "per_run", + }); + } + if (leaseRelease) { + const release = leaseRelease; + options.ledger.register({ + id: "staging_lease", + payload: { release }, + scope: "per_run", + }); + } + return { + referencedProjectStagingFailures: (staged.stagedRuntime.additionalSourceFailures ?? []).map( + (failure) => ({ projectId: failure.projectId }), + ), + }; + }, + + async startTransport(): Promise { + // Bring up both host-side bridges concurrently. Their remote subtrees are + // disjoint, so their env-independent setup overlaps. The one real + // dependency — the paperclip bridge's returned env must reach the + // process-session launch — is sequenced by `launchEnv`, a memoized thunk the + // process-session bridge awaits right before its launch. + const stagedRootDir = staged?.stagedRuntime.runtimeRootDir ?? null; + const paperclipStart = options.measureBridgeStep("bridge.paperclip", () => + options.startPaperclipBridge(stagedRootDir), + ); + // The single sequencing point (paperclip env → process-session launch), + // memoized so the merge runs exactly once whether the process-session bridge + // consumes it at launch or `startTransport` finalizes it below. + let launchEnvPromise: Promise> | null = null; + let launchEnv: Record = {}; + const finalizeLaunchEnv = (): Promise> => + (launchEnvPromise ??= (async () => { + const paperclip = await paperclipStart; + // The paperclip bridge token is run-scoped: it lives for this run only and + // never enters a reuse payload (Amendment B). The site hands it to the + // engine's `finalizeLaunchEnv`, the sole consumer of a contribution, and + // retains nothing. + const contributions: LaunchEnvironmentContribution[] = []; + if (paperclip) { + contributions.push({ scope: "run", env: paperclip.env } as unknown as RunScopedContribution); + await options.onPaperclipBridgeLog(); + } + launchEnv = options.finalizeLaunchEnv(contributions); + return launchEnv; + })()); + const processSessionStart = options.measureBridgeStep("bridge.process-session", () => + options.startProcessSessionBridge({ runtimeRootDir: stagedRootDir, launchEnv: finalizeLaunchEnv }), + ); + // Settle BOTH starts, so a partial failure can stop whichever bridge started. + const [paperclip, processSession] = await Promise.allSettled([paperclipStart, processSessionStart]); + controlBridge = paperclip.status === "fulfilled" ? paperclip.value : null; + agentBridge = processSession.status === "fulfilled" ? processSession.value : null; + const failure = + paperclip.status === "rejected" + ? paperclip.reason + : processSession.status === "rejected" + ? processSession.reason + : null; + if (failure) throw failure; + // Guarantee the merge ran even if the process-session bridge returned without + // consuming the launch env (memoized, so a no-op if it did). + await finalizeLaunchEnv(); + if (controlBridge) { + options.ledger.register({ id: "control_bridge", payload: controlBridge, scope: "per_run" }); + } + if (agentBridge) { + options.ledger.register({ id: "agent_bridge", payload: agentBridge, scope: "per_run" }); + } + return { controlBridge, agentBridge, launchEnv }; + }, + + reuse(): SessionReuseStore { + return store; + }, + + reuseCandidate(resources: ReadyRunResources): SandboxReuseCandidate { + const stagedResource = resources.get("staged_runtime"); + if (!stagedResource) { + throw new Error("sandbox run site cannot name a reuse candidate without a staged_runtime resource"); + } + // The reuse payload carries the staged files only — no credential material. + return { stagedRuntime: stagedResource.stagedRuntime }; + }, + + async syncBack(): Promise { + try { + // Stop both bridges, then run the managed-home copy-back (mirrors the CLI + // finally: stop bridge, then restore workspace). + await options.stopBridges({ controlBridge, agentBridge }); + } finally { + // Release the per-session staging lease in a finally, so an earlier + // teardown fault never strands it and deadlocks the next same-session run. + leaseRelease?.(); + leaseRelease = null; + } + }, + + get staged(): StagedWorkspace | null { + return staged; + }, + + get controlBridge(): AdapterExecutionTargetPaperclipBridgeHandle | null { + return controlBridge; + }, + + get agentBridge(): AdapterExecutionTargetProcessSessionBridgeHandle | null { + return agentBridge; + }, + + get stagingLeaseRelease(): (() => void) | null { + return leaseRelease; + }, + }; +} + +// Per-`sessionKey` async lease: chains each caller after the previous one so the +// stage-or-reuse decision for a session runs serially, then keeps the lease held +// until the run releases it. Two overlapping runs of the same session can never +// stage into the same remote workspace at once: the loser waits, then re-checks +// the cache before it decides. +async function withStagingLease( + locks: Map>, + key: string, + fn: () => Promise, +): Promise<{ value: T; release: () => void }> { + const prev = locks.get(key) ?? Promise.resolve(); + let releaseGate!: () => void; + const gate = new Promise((resolve) => { + releaseGate = resolve; + }); + const mine: Promise = prev.then(() => gate); + locks.set(key, mine); + await prev.catch(() => {}); + let released = false; + const release = () => { + if (released) return; + released = true; + releaseGate(); + if (locks.get(key) === mine) locks.delete(key); + }; + try { + return { value: await fn(), release }; + } catch (error) { + if (!released) release(); + throw error; + } +} diff --git a/packages/adapter-utils/src/acpx-engine/session-reuse-store.test.ts b/packages/adapter-utils/src/acpx-engine/session-reuse-store.test.ts new file mode 100644 index 0000000000..0af143f389 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/session-reuse-store.test.ts @@ -0,0 +1,160 @@ +// Tests for the generic per-session reuse store (Phase 18). +// +// The store backs both site caches over a caller-provided map. These tests pin +// the four behaviors the two caches had before the refactor: borrow reads +// without removal and clears the per-entry idle timer; discard is +// identity-guarded and releases once; save arms a per-entry idle timer that +// discards without a run; and the run-start sweep runs each eviction under the +// optional per-key lease. + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { createSessionReuseStore } from "./session-reuse-store.js"; + +interface Entry { + readonly id: string; + lastUsedAt: number; + cleanupTimer?: ReturnType; +} + +function hostConfig(entries: Map, released: Entry[], idleMs: number) { + return { + entries, + now: () => Date.now(), + idleMs, + lastUsedAt: (entry: Entry) => entry.lastUsedAt, + getTimer: (entry: Entry) => entry.cleanupTimer, + setTimer: (entry: Entry, timer: ReturnType | undefined) => { + entry.cleanupTimer = timer; + }, + perEntryTimer: true, + release: (entry: Entry) => { + released.push(entry); + }, + }; +} + +describe("session reuse store", () => { + beforeEach(() => { + vi.useFakeTimers(); + }); + afterEach(() => { + vi.useRealTimers(); + }); + + it("test_borrow_reads_without_removal_and_clears_the_per_entry_idle_timer", async () => { + const entries = new Map(); + const released: Entry[] = []; + const store = createSessionReuseStore(hostConfig(entries, released, 1000)); + + const entry: Entry = { id: "a", lastUsedAt: Date.now() }; + store.save("k", entry); + expect(entry.cleanupTimer).toBeDefined(); + + // A borrow reads the entry and leaves it in the store, so a second borrow + // still finds it. + expect(store.borrow("k")).toBe(entry); + expect(store.borrow("k")).toBe(entry); + + // The borrow cleared the per-entry idle timer, so the entry does not expire + // while it is in use. Advance well past the idle bound. + await vi.advanceTimersByTimeAsync(5000); + expect(released).toEqual([]); + expect(store.borrow("k")).toBe(entry); + // The entry stays in the caller's map for a later run to inspect. + expect(entries.get("k")).toBe(entry); + }); + + it("test_discard_is_identity_guarded_and_fires_release_once", async () => { + const entries = new Map(); + const released: Entry[] = []; + const store = createSessionReuseStore({ + entries, + now: () => Date.now(), + idleMs: 0, + lastUsedAt: (entry) => entry.lastUsedAt, + release: (entry) => { + released.push(entry); + }, + }); + + const first: Entry = { id: "first", lastUsedAt: Date.now() }; + store.save("k", first); + + // The first discard removes the entry and fires the release once. + await store.discard("k"); + expect(released).toEqual([first]); + expect(store.borrow("k")).toBeUndefined(); + + // A second discard of the same key finds no entry, so it releases nothing. + await store.discard("k"); + expect(released).toEqual([first]); + + // A re-saved entry is a new entry. A discard of the key releases the new + // entry, never the already-released first entry again. + const second: Entry = { id: "second", lastUsedAt: Date.now() }; + store.save("k", second); + await store.discard("k"); + expect(released).toEqual([first, second]); + }); + + it("test_save_arms_a_per_entry_idle_timer_that_discards_without_a_run", async () => { + const entries = new Map(); + const released: Entry[] = []; + const store = createSessionReuseStore(hostConfig(entries, released, 1000)); + + const entry: Entry = { id: "a", lastUsedAt: Date.now() }; + store.save("k", entry); + + // Before the idle deadline the entry is still present and unreleased. + await vi.advanceTimersByTimeAsync(999); + expect(released).toEqual([]); + + // At the idle deadline the armed timer discards the entry with no run and + // no explicit discard call. + await vi.advanceTimersByTimeAsync(1); + expect(released).toEqual([entry]); + expect(store.borrow("k")).toBeUndefined(); + expect(entries.has("k")).toBe(false); + }); + + it("test_evict_idle_sweep_runs_under_per_key_lease", async () => { + const entries = new Map(); + const released: Entry[] = []; + const leaseOrder: string[] = []; + let now = 0; + const store = createSessionReuseStore({ + entries, + now: () => now, + idleMs: 1000, + lastUsedAt: (entry) => entry.lastUsedAt, + // The sandbox lane leaves the per-entry timer off and relies on the + // run-start sweep only. + release: (entry) => { + released.push(entry); + }, + withEvictLease: async (key, critical) => { + leaseOrder.push(`lease:${key}`); + await critical(); + leaseOrder.push(`release:${key}`); + }, + }); + + store.save("stale", { id: "stale", lastUsedAt: 0 }); + store.save("fresh", { id: "fresh", lastUsedAt: 0 }); + + // Move the clock so only the "stale" entry crosses the idle bound. Touch + // "fresh" so it stays inside the window. + now = 1500; + store.borrow("fresh"); + store.save("fresh", { id: "fresh2", lastUsedAt: now }); + + await store.evictIdle(now); + + // Only the stale entry evicts, and its critical section ran inside the + // per-key lease (lease acquired, critical ran, lease released, in order). + expect(released.map((entry) => entry.id)).toEqual(["stale"]); + expect(leaseOrder).toEqual(["lease:stale", "release:stale"]); + expect(store.borrow("stale")).toBeUndefined(); + expect(store.borrow("fresh")).toEqual({ id: "fresh2", lastUsedAt: 1500 }); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/session-reuse-store.ts b/packages/adapter-utils/src/acpx-engine/session-reuse-store.ts new file mode 100644 index 0000000000..a8ca1f8994 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/session-reuse-store.ts @@ -0,0 +1,189 @@ +// The generic per-session reuse store. +// +// One `SessionReuseStore` implementation backs both site caches. The host +// site saves a live runtime and session handle. The sandbox site saves the +// staged files. The store is generic in the saved type, so a generic `save` +// cannot read what a site saves. `RunSite.reuseCandidate()` names the concrete +// type per site. +// +// The store operates OVER a caller-provided map, not a private internal map. +// The host lane passes its warm-handle map; the sandbox lane passes its staged- +// runtime map. Both maps persist across runs, so a saved entry stays visible to +// a later run of the same session. Each entry carries its own last-used time and +// its own optional per-entry idle timer, so the store reads and writes those +// through the caller-supplied accessors. This keeps the exact shape both caches +// had before this refactor, so a caller that inspects the map still reads the +// same entries. +// +// The store keeps the exact behavior the two caches had before this refactor: +// +// * `borrow` reads without removal. It keeps today's visibility for +// overlapping runs of the same session. It clears the entry's per-entry +// idle timer, because an in-use entry must not expire under its own timer. +// * `save` sets the entry in the map. When the store has a per-entry idle +// timer, `save` arms an unref'd timer that discards the entry at the idle +// deadline without a run (today's `scheduleIdleHandleCleanup`). +// * `discard` is an identity-guarded removal that fires the idempotent +// release once. It acts on the entry that is at the key now, so a re-saved +// entry is safe and a second discard is a no-op. +// * `evictIdle` is the run-start sweep (today's `cleanupIdleHandles` and +// `cleanupIdleStagedRuntimes`). The host lane runs the critical section +// directly. The sandbox lane runs it under its per-key staging lease and +// re-checks the idle window inside the lease. + +import type { SessionReuseStore } from "./run-contracts.js"; + +/** + * The construction options for a reuse store. The public `SessionReuseStore` + * surface stays generic; these options carry the per-lane behavior the store + * needs but the interface does not name: the backing map, the clock, the idle + * bound, how to read an entry's last-used time and per-entry timer, how to + * release an entry, whether to arm a per-entry timer, and the optional eviction + * lease. + */ +export interface SessionReuseStoreConfig { + /** + * The map the store reads and writes. The caller owns it, so a caller that + * inspects the map still reads the same entries. The map persists across runs. + */ + readonly entries: Map; + /** The clock. The store reads it for the idle deadline and the sweep. */ + readonly now: () => number; + /** + * The idle bound, in milliseconds. When it is `0` or less, the store arms no + * per-entry timer and the sweep evicts nothing. + */ + readonly idleMs: number; + /** Read an entry's last-used time. The store compares it to the idle bound. */ + readonly lastUsedAt: (entry: T) => number; + /** + * Release an entry's resources. The store calls it once per entry on a + * discard, on a per-entry timer fire, or on an idle sweep. The store guards + * every release with a map-identity check, so a discarded entry never + * releases twice. + */ + readonly release: (entry: T) => void | Promise; + /** + * Read an entry's per-entry idle timer, or `undefined` when it holds none. + * The host lane supplies this; the sandbox lane omits it and arms no timer. + */ + readonly getTimer?: (entry: T) => ReturnType | undefined; + /** + * Write (or clear with `undefined`) an entry's per-entry idle timer. The host + * lane supplies this; the sandbox lane omits it. + */ + readonly setTimer?: (entry: T, timer: ReturnType | undefined) => void; + /** + * Arm a per-entry idle timer on `save`. The host lane sets it true. The + * sandbox lane leaves it false and relies on the run-start sweep only. + */ + readonly perEntryTimer?: boolean; + /** + * Run each eviction critical section under a per-key lease. The sandbox lane + * supplies its per-key staging lease so the sweep re-checks the entry inside + * the lease. The host lane omits it and the sweep runs the critical section + * directly. + */ + readonly withEvictLease?: (key: string, critical: () => Promise) => Promise; +} + +/** + * Create a generic per-session reuse store over the caller-provided map. The + * store holds at most one entry per session key. Each entry carries its saved + * value, its last-used time, and an optional per-entry idle timer, all read + * through the config accessors. + */ +export function createSessionReuseStore( + config: SessionReuseStoreConfig, +): SessionReuseStore { + const { entries } = config; + + function clearEntryTimer(entry: T): void { + const timer = config.getTimer?.(entry); + if (timer === undefined) return; + clearTimeout(timer); + config.setTimer?.(entry, undefined); + } + + // Release an entry once and drop it from the map when it is still the current + // entry at the key. The map-identity guard makes the release idempotent: the + // synchronous `delete` runs before the awaited release, so a second release of + // the same entry finds a different or absent entry at the key and stops. + async function releaseEntry(key: string, entry: T): Promise { + clearEntryTimer(entry); + if (entries.get(key) !== entry) return; + entries.delete(key); + await config.release(entry); + } + + function armTimer(key: string, entry: T): void { + if (!config.perEntryTimer || config.idleMs <= 0) return; + clearEntryTimer(entry); + const delayMs = Math.max(1, config.lastUsedAt(entry) + config.idleMs - config.now()); + const timer = setTimeout(() => { + void (async () => { + if (entries.get(key) !== entry) return; + // A later touch can move `lastUsedAt` forward, so re-check the idle + // window and re-arm the timer when the entry is still in use. + if (config.now() - config.lastUsedAt(entry) < config.idleMs) { + armTimer(key, entry); + return; + } + await releaseEntry(key, entry); + })(); + }, delayMs); + timer.unref?.(); + config.setTimer?.(entry, timer); + } + + return { + borrow(sessionKey: string): T | undefined { + const entry = entries.get(sessionKey); + if (entry === undefined) return undefined; + // The entry is in use now, so clear its per-entry idle timer. The next + // `save` re-arms a fresh timer after the run finishes. + clearEntryTimer(entry); + return entry; + }, + + save(sessionKey: string, entry: T): void { + // The caller sets the entry's last-used time before it saves, so `save` + // sets the entry as-is and arms its timer from that time. + entries.set(sessionKey, entry); + armTimer(sessionKey, entry); + }, + + discard(sessionKey: string): Promise { + const entry = entries.get(sessionKey); + // `releaseEntry` deletes the key synchronously and then fires the release, + // so the returned promise lets the caller await the release when it must + // finish before a downstream resource release. + if (entry === undefined) return Promise.resolve(); + return releaseEntry(sessionKey, entry); + }, + + async evictIdle(now: number): Promise { + if (config.idleMs <= 0) return; + const stale: Array<[string, T]> = []; + for (const [key, entry] of entries.entries()) { + if (now - config.lastUsedAt(entry) >= config.idleMs) stale.push([key, entry]); + } + for (const [key, entry] of stale) { + const critical = async (): Promise => { + if (entries.get(key) !== entry) return; + // Under a lease the sweep re-checks the idle window, so a run that + // re-saved the key while the lease was held keeps its entry. + if (config.withEvictLease && config.now() - config.lastUsedAt(entry) < config.idleMs) { + return; + } + await releaseEntry(key, entry); + }; + if (config.withEvictLease) { + await config.withEvictLease(key, critical); + } else { + await critical(); + } + } + }, + }; +} diff --git a/packages/adapter-utils/src/acpx-engine/settlement-characterization.test.ts b/packages/adapter-utils/src/acpx-engine/settlement-characterization.test.ts index d5f9754a88..eb4aba4b1c 100644 --- a/packages/adapter-utils/src/acpx-engine/settlement-characterization.test.ts +++ b/packages/adapter-utils/src/acpx-engine/settlement-characterization.test.ts @@ -354,9 +354,11 @@ describe("ACP settlement — Layer A: engine teardown orchestration", () => { expect(stagingLocks.size).toBe(0); }); - it("test_clean_persistent_local_run_warm_saves_instead_of_closing", async () => { - // execute.ts:3886 keeps a clean persistent local turn warm (no runtime.close) - // when warmIdleMs>0 and there is no process-session bridge. + it("test_clean_persistent_local_run_closes_and_relaunches_amendment_b", async () => { + // Amendment B: a clean persistent local turn closes the runtime instead of + // warm-saving it. The host-lane reuse candidate carries the run-minted API key, + // which is never revoked, so the credential gate fails and the settlement + // discards (close-and-relaunch). const root = await makeTempRoot(); const closeSpy = vi.fn(async () => {}); const warmHandles = new Map(); @@ -393,16 +395,9 @@ describe("ACP settlement — Layer A: engine teardown orchestration", () => { } as never); expect(result.exitCode).toBe(0); - // The runtime stayed warm: no close, one warm entry. - expect(closeSpy).not.toHaveBeenCalled(); - expect(warmHandles.size).toBe(1); - - // Clear the scheduled idle-cleanup timer so no timer outlives the test. The - // long idle window kept the timer from racing the assertions above; clearing - // it here makes cleanup deterministic instead of a timed drain. - for (const entry of warmHandles.values()) { - if (entry.cleanupTimer) clearTimeout(entry.cleanupTimer); - } + // The runtime closed and relaunched: one close, no warm entry. + expect(closeSpy).toHaveBeenCalledTimes(1); + expect(warmHandles.size).toBe(0); }); it("test_completed_non_persistent_local_run_closes_runtime_once", async () => { @@ -543,12 +538,12 @@ describe("ACP settlement — Layer A: engine teardown orchestration", () => { } }); - it("test_handshake_failure_after_warm_hit_closes_runtime_and_removes_warm_entry", async () => { - // Mirror execute.test.ts F2 test_handshake_failure_closes_runtime_and_removes_warm_entry - // (:4666). The first run warms a handle; the second run reuses it and fails - // while persisting process identity (onSpawn throws). Because the warm hit - // already holds the handle, the handshake failure closes the reused runtime - // (execute.ts:3591) and removes the warm entry. + it("test_handshake_failure_on_second_run_closes_the_recreated_runtime", async () => { + // Amendment B: the first clean persistent turn closes and relaunches instead of + // warm-saving, so the second run finds no warm handle. The second run re-creates + // the runtime and fails while persisting process identity (onSpawn throws); the + // handshake failure closes the re-created runtime. The host-lane warm-hit + // teardown path itself is characterized by the composed fault matrix. const root = await makeTempRoot(); const stateDir = path.join(root, "state"); const startedAt = "2026-01-01T00:00:00.000Z"; @@ -591,7 +586,8 @@ describe("ACP settlement — Layer A: engine teardown orchestration", () => { onSpawn: async () => {}, } as never); expect(first.exitCode).toBe(0); - expect(warmHandles.size).toBe(1); + // Amendment B: no warm entry survives the first clean persistent turn. + expect(warmHandles.size).toBe(0); expect(created).toBe(1); const second = await execute({ @@ -607,11 +603,12 @@ describe("ACP settlement — Layer A: engine teardown orchestration", () => { }, } as never); - // The reused runtime (no new create) is closed and the warm entry removed. - expect(created).toBe(1); + // The second run re-created the runtime (no warm reuse) and closed it on the + // failure, reporting the ensure_session phase. Both runtimes closed. + expect(created).toBe(2); expect(second.exitCode).toBe(1); expect(second.resultJson?.phase).toBe("ensure_session"); - expect(closeSpy).toHaveBeenCalledTimes(1); + expect(closeSpy).toHaveBeenCalledTimes(2); expect(warmHandles.size).toBe(0); }); diff --git a/packages/adapter-utils/src/acpx-engine/settlement-sequence.test.ts b/packages/adapter-utils/src/acpx-engine/settlement-sequence.test.ts new file mode 100644 index 0000000000..6f3578e79a --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/settlement-sequence.test.ts @@ -0,0 +1,323 @@ +import { describe, expect, it } from "vitest"; +import { + decideReuse, + settleAcpRun, + type ReuseCandidateInput, + type ReuseDecision, + type SettlementSlots, + type SettlementSteps, +} from "./settlement-sequence.js"; +import { createRunResourceLedger } from "./run-resource-ledger.js"; +import type { AcquiredRunResources, ResourceId, SettlementCause } from "./run-contracts.js"; + +// The settlement sequence is the one cleanup owner. These tests drive it with +// fakes for the steps and assert the claim, the order, the Phase 3 error policy, +// the staging-lease finally, and the Amendment B credential gate. They never +// touch the engine's real run state. + +// Register a per-run resource for each id, with an optional payload the steps +// read. The tests do not seal the ledger; `takeForSettlement` runs from `open`. +function ledgerWith( + ids: readonly ResourceId[], + payloads: Partial> = {}, +): AcquiredRunResources { + const ledger = createRunResourceLedger(); + const register = ledger.register as (registration: { + id: ResourceId; + payload: unknown; + scope: "per_run"; + }) => void; + for (const id of ids) { + register({ id, payload: payloads[id] ?? {}, scope: "per_run" }); + } + return ledger; +} + +// Count the claims so a test proves the sequence claims the ledger once. +function countingLedger(inner: AcquiredRunResources): { + ledger: AcquiredRunResources; + claims: () => number; +} { + let claims = 0; + const ledger: AcquiredRunResources = { + register: inner.register.bind(inner), + seal: inner.seal.bind(inner), + takeForSettlement: () => { + claims += 1; + return inner.takeForSettlement(); + }, + }; + return { ledger, claims: () => claims }; +} + +// A base set of steps whose bodies record their name. Each test overrides the +// steps it drives. +function baseSteps(order: string[], overrides: Partial = {}): SettlementSteps { + return { + reuseCandidate: overrides.reuseCandidate ?? (() => null), + endSession: overrides.endSession ?? (() => { + order.push("end_session"); + }), + settleReuse: overrides.settleReuse ?? (() => { + order.push("settle_reuse"); + }), + stopTransport: overrides.stopTransport ?? (() => { + order.push("stop_transport"); + }), + syncBack: overrides.syncBack ?? (() => { + order.push("sync_back"); + }), + releaseStagingLease: overrides.releaseStagingLease ?? (() => { + order.push("release_staging_lease"); + }), + recordError: overrides.recordError ?? (() => {}), + }; +} + +const cleanTurnCause: SettlementCause | null = null; + +describe("ACPX settlement sequence", () => { + it("test_settlement_claims_ledger_once_then_runs_the_steps_in_order", async () => { + const order: string[] = []; + const { ledger, claims } = countingLedger( + ledgerWith(["acp_runtime", "control_bridge", "agent_bridge", "staging_lease"]), + ); + + await settleAcpRun(ledger, cleanTurnCause, baseSteps(order)); + + // The sequence claims the ledger exactly once. + expect(claims()).toBe(1); + // It runs the effect steps in the fixed order; the lease release runs last. + expect(order).toEqual([ + "end_session", + "settle_reuse", + "stop_transport", + "sync_back", + "release_staging_lease", + ]); + }); + + it("test_every_step_noops_on_empty_slot", async () => { + const order: string[] = []; + // An empty ledger: no slot holds a resource. Each step reads its slot and + // no-ops when the slot is empty. + const ledger = ledgerWith([]); + const noopFor = (name: string, id: ResourceId) => (slots: SettlementSlots) => { + order.push(slots.has(id) ? `${name}:work` : `${name}:noop`); + }; + const steps = baseSteps(order, { + endSession: noopFor("end_session", "acp_runtime"), + settleReuse: noopFor("settle_reuse", "acp_runtime"), + stopTransport: noopFor("stop_transport", "control_bridge"), + syncBack: noopFor("sync_back", "staged_runtime"), + releaseStagingLease: (slots) => { + order.push(slots.has("staging_lease") ? "release_staging_lease:work" : "release_staging_lease:noop"); + }, + }); + + const report = await settleAcpRun(ledger, cleanTurnCause, steps); + + // Every step saw an empty slot and no-opped. + expect(order).toEqual([ + "end_session:noop", + "settle_reuse:noop", + "stop_transport:noop", + "sync_back:noop", + "release_staging_lease:noop", + ]); + // An empty ledger yields an empty disposition report. + expect(report.records).toEqual([]); + }); + + it("test_transferred_runtime_is_never_closed_and_discarded_entry_is_finalized", async () => { + const order: string[] = []; + const ledger = ledgerWith(["acp_runtime", "staging_lease"], { + acp_runtime: { runtime: {}, sessionHandle: {}, childProcessPid: null }, + }); + const steps = baseSteps(order, { + // A clean host turn with no live run-scoped credential is save-eligible. + reuseCandidate: (): ReuseCandidateInput => ({ + kind: "host", + causePermitsSave: true, + liveRunScopedCredentials: [], + }), + endSession: (_slots, decision) => { + // The transferred runtime is never closed. + order.push(decision.kind === "save" ? "kept" : "closed"); + }, + settleReuse: (_slots, decision) => { + order.push(decision.kind === "save" ? "transferred" : "discarded"); + }, + }); + + const report = await settleAcpRun(ledger, cleanTurnCause, steps); + + // The runtime transferred; it was never closed. + expect(order).toContain("kept"); + expect(order).not.toContain("closed"); + expect(order).toContain("transferred"); + // The report marks the transferred runtime `transferred` and the discarded + // staging lease `finalized`. + expect(report.records).toEqual([ + { id: "acp_runtime", disposition: "transferred" }, + { id: "staging_lease", disposition: "finalized" }, + ]); + }); + + it("test_step_error_is_recorded_and_later_steps_still_run", async () => { + const order: string[] = []; + const recorded: string[] = []; + const boom = new Error("end_session boom"); + const steps = baseSteps(order, { + endSession: () => { + throw boom; + }, + recordError: (step, error) => { + recorded.push(step); + expect(error).toBe(boom); + }, + }); + + await settleAcpRun(ledgerWith(["acp_runtime", "staging_lease"]), cleanTurnCause, steps); + + // The failed step is recorded... + expect(recorded).toContain("end_session"); + // ...and every later step still runs. + expect(order).toEqual([ + "settle_reuse", + "stop_transport", + "sync_back", + "release_staging_lease", + ]); + }); + + it("test_staging_lease_release_runs_in_finally", async () => { + const order: string[] = []; + const recorded: string[] = []; + const steps = baseSteps(order, { + syncBack: () => { + throw new Error("sync_back boom"); + }, + recordError: (step) => { + recorded.push(step); + }, + }); + + await settleAcpRun(ledgerWith(["staging_lease"]), cleanTurnCause, steps); + + // The sync-back failure is recorded, and the lease release still runs. + expect(recorded).toContain("sync_back"); + expect(order).toContain("release_staging_lease"); + }); + + it("test_two_compatible_runs_do_not_share_run_scoped_credentials", async () => { + // Two compatible runs each own their ledger, bridge token, and run API key. + // Run B must never observe run A's run-scoped values. + const observed: Array<{ run: string; token: unknown; credentials: readonly string[] }> = []; + const settleRun = async (run: string, token: string, credential: string) => { + const ledger = ledgerWith(["acp_runtime", "control_bridge"], { + control_bridge: { token }, + }); + const steps = baseSteps([], { + reuseCandidate: (): ReuseCandidateInput => ({ + kind: "host", + causePermitsSave: true, + liveRunScopedCredentials: [credential], + }), + stopTransport: (slots) => { + const bridge = slots.get("control_bridge") as { token: unknown } | undefined; + observed.push({ run, token: bridge?.token, credentials: [credential] }); + }, + }); + return settleAcpRun(ledger, cleanTurnCause, steps); + }; + + await settleRun("A", "token-A", "run-api-key-A"); + await settleRun("B", "token-B", "run-api-key-B"); + + const runA = observed.find((o) => o.run === "A"); + const runB = observed.find((o) => o.run === "B"); + // Run B observed only its own bridge token and credential, never run A's. + expect(runB?.token).toBe("token-B"); + expect(runB?.token).not.toBe(runA?.token); + expect(runB?.credentials).not.toContain("run-api-key-A"); + }); + + it("test_no_run_scoped_credential_appears_in_reuse_payload_or_telemetry", async () => { + const runScopedCredential = "run-api-key-secret"; + let savedPayload: Record | null = null; + // A save-eligible sandbox candidate: the staged-files payload carries no + // credential, and the bridge tokens already died with the bridges, so the + // live run-scoped credential list is empty. + const ledger = ledgerWith(["staged_runtime", "staging_lease"], { + staged_runtime: { stagedRuntime: { files: ["a", "b"] } }, + }); + const steps = baseSteps([], { + reuseCandidate: (): ReuseCandidateInput => ({ + kind: "sandbox", + causePermitsSave: true, + liveRunScopedCredentials: [], + }), + settleReuse: (slots, decision) => { + if (decision.kind === "save") { + const staged = slots.get("staged_runtime") as unknown as { + stagedRuntime: Record; + }; + savedPayload = staged.stagedRuntime; + } + }, + }); + + const report = await settleAcpRun(ledger, cleanTurnCause, steps); + + // The saved reuse payload carries no run-scoped credential. + expect(savedPayload).not.toBeNull(); + expect(JSON.stringify(savedPayload)).not.toContain(runScopedCredential); + // The disposition report (telemetry) carries no run-scoped credential. + expect(JSON.stringify(report)).not.toContain(runScopedCredential); + // The staged files transferred. + expect(report.records).toContainEqual({ id: "staged_runtime", disposition: "transferred" }); + }); + + it("test_warm_save_requires_credential_gate_pass_else_close_and_relaunch", async () => { + const runWithCredentials = async (liveRunScopedCredentials: readonly string[]) => { + const order: string[] = []; + const ledger = ledgerWith(["acp_runtime"]); + const steps = baseSteps(order, { + reuseCandidate: (): ReuseCandidateInput => ({ + kind: "host", + causePermitsSave: true, + liveRunScopedCredentials, + }), + endSession: (_slots, decision) => { + order.push(decision.kind === "save" ? "kept" : "closed"); + }, + }); + const report = await settleAcpRun(ledger, cleanTurnCause, steps); + return { order, report }; + }; + + // A live run-scoped credential fails the gate: close-and-relaunch, finalized. + const gated = await runWithCredentials(["run-api-key"]); + expect(gated.order).toContain("closed"); + expect(gated.report.records).toContainEqual({ id: "acp_runtime", disposition: "finalized" }); + + // No live run-scoped credential passes the gate: the runtime transfers. + const cleared = await runWithCredentials([]); + expect(cleared.order).toContain("kept"); + expect(cleared.report.records).toContainEqual({ id: "acp_runtime", disposition: "transferred" }); + }); + + it("decideReuse blocks a save when the Amendment B credential gate fails", () => { + // A direct unit check on the pure decision, so the gate is legible on its own. + const base: ReuseCandidateInput = { kind: "host", causePermitsSave: true, liveRunScopedCredentials: [] }; + expect(decideReuse(base)).toEqual({ kind: "save", savedId: "acp_runtime" } satisfies ReuseDecision); + expect(decideReuse({ ...base, liveRunScopedCredentials: ["k"] }).kind).toBe("discard"); + expect(decideReuse({ ...base, causePermitsSave: false }).kind).toBe("discard"); + expect(decideReuse(null).kind).toBe("discard"); + expect(decideReuse({ ...base, kind: "sandbox" })).toEqual({ + kind: "save", + savedId: "staged_runtime", + } satisfies ReuseDecision); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/settlement-sequence.ts b/packages/adapter-utils/src/acpx-engine/settlement-sequence.ts new file mode 100644 index 0000000000..2db0fda9ea --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/settlement-sequence.ts @@ -0,0 +1,204 @@ +// The settlement sequence. +// +// `settleAcpRun(ledger, cause, steps)` is the one cleanup owner for every site +// and every settled path. The one exception is the startup rollback, which the +// coordinator owns (Phase 20). +// +// The sequence claims the ledger once, makes the pure reuse decision, then runs +// the ordered steps: +// +// 1. endSession — close every runtime the decision did not transfer, and +// drop the warm entry when it closes. +// 2. settleReuse — perform the decision: a save transfers the candidate to the +// site store (which arms the per-entry idle timer); everything +// else discards. +// 3. stopTransport — stop both bridges in one `allSettled`. +// 4. syncBack — the site sync-back. +// 5. releaseStagingLease — runs in a `finally`, so an earlier step fault never +// strands the lease. +// +// The Phase 3 error policy governs every step: a step records its error and the +// later steps still run. Every step no-ops on an empty slot. +// +// Amendment B — the credential gate. A runtime is save-eligible only when no +// run-scoped credential issued to it remains valid. The decision is pure and +// runs before any close. Settlement stops both bridges before a sandbox-lane +// transfer, so the bridge tokens die with the bridges. The run API key is not +// invalidated on run end (grounded server-side), so the host-lane warm save +// discards instead of transfers (close-and-relaunch) until a credential-rebind +// protocol exists. The sandbox staged-files reuse is unaffected: its payload +// carries no credentials. + +import type { + AcquiredRunResources, + ResourceDisposition, + ResourceId, + RunResourcePayloads, + RunSiteKind, + SettledResourceEntry, + SettlementCause, +} from "./run-contracts.js"; + +/** The save-versus-discard decision the settlement makes before any close. */ +export type ReuseDecision = + | { readonly kind: "save"; readonly savedId: ResourceId } + | { readonly kind: "discard"; readonly reason: string }; + +/** + * The inputs the reuse decision reads. The site derives the candidate from the + * settled slots. A null candidate means the run holds nothing to save. + */ +export interface ReuseCandidateInput { + /** The site kind. "host" saves the runtime; "sandbox" saves the staged files. */ + readonly kind: RunSiteKind; + /** True when the settlement cause permits a save (a clean persistent turn). */ + readonly causePermitsSave: boolean; + /** + * Amendment B: the run-scoped credentials the candidate would still carry into + * the store. A save is eligible only when this list is empty — no run-scoped + * credential issued to the runtime remains valid. + */ + readonly liveRunScopedCredentials: readonly string[]; +} + +/** One resource's final disposition, as the settlement records it. */ +export interface ResourceDispositionRecord { + readonly id: ResourceId; + readonly disposition: ResourceDisposition; +} + +/** The per-resource disposition report the settlement produces at the end. */ +export interface SettlementDispositionReport { + readonly records: readonly ResourceDispositionRecord[]; +} + +/** The settled slots view a step reads. `get` returns undefined for an empty slot. */ +export interface SettlementSlots { + get(id: Id): RunResourcePayloads[Id] | undefined; + has(id: ResourceId): boolean; +} + +/** + * The effectful steps the sequence runs. The engine implements each step over + * its own run state. Each step no-ops on an empty slot. A step must not throw + * for the sequence to record its error and continue; a throw is caught and + * recorded under the Phase 3 error policy. + */ +export interface SettlementSteps { + /** + * Derive the reuse candidate from the settled slots and the cause, or null + * when the run holds nothing to save. + */ + reuseCandidate(slots: SettlementSlots, cause: SettlementCause | null): ReuseCandidateInput | null; + /** Close every runtime the decision did not transfer; drop the warm entry on close. */ + endSession(slots: SettlementSlots, decision: ReuseDecision): Promise | void; + /** Perform the decision: a save transfers to the site store; everything else discards. */ + settleReuse(slots: SettlementSlots, decision: ReuseDecision): Promise | void; + /** Stop both bridges in one `allSettled`. */ + stopTransport(slots: SettlementSlots): Promise | void; + /** The site sync-back. */ + syncBack(slots: SettlementSlots): Promise | void; + /** Release the staging lease. Runs in a `finally`. */ + releaseStagingLease(slots: SettlementSlots): void; + /** Record a per-step error under the Phase 3 error policy. */ + recordError(step: string, error: unknown): Promise | void; +} + +/** + * The pure reuse decision. A save is eligible only when a candidate exists, the + * cause permits a save, and Amendment B holds: no run-scoped credential issued + * to the candidate remains valid. Everything else discards, with a reason. + */ +export function decideReuse(candidate: ReuseCandidateInput | null): ReuseDecision { + if (!candidate) { + return { kind: "discard", reason: "no reuse candidate" }; + } + if (!candidate.causePermitsSave) { + return { kind: "discard", reason: "settlement cause forbids save" }; + } + if (candidate.liveRunScopedCredentials.length > 0) { + // Amendment B credential gate: a live run-scoped credential blocks the save. + return { kind: "discard", reason: "run-scoped credential still valid" }; + } + const savedId: ResourceId = candidate.kind === "host" ? "acp_runtime" : "staged_runtime"; + return { kind: "save", savedId }; +} + +/** + * Settle the run's resources. Claims the ledger once, makes the reuse decision, + * runs the ordered steps under the Phase 3 error policy, releases the staging + * lease in a `finally`, and returns the final per-resource disposition report. + */ +export async function settleAcpRun( + ledger: AcquiredRunResources, + cause: SettlementCause | null, + steps: SettlementSteps, +): Promise { + // Claim the ledger once. Every later step reads the claimed slots. + const consumed = ledger.takeForSettlement(); + const entries = consumed.entries(); + const slots = makeSlots(entries); + + // The reuse decision is pure and runs before any close. + const candidate = steps.reuseCandidate(slots, cause); + const decision = decideReuse(candidate); + + const runStep = async (name: string, run: () => Promise | void): Promise => { + try { + await run(); + } catch (error) { + // The Phase 3 error policy: record the step error and let later steps run. + await steps.recordError(name, error); + } + }; + + try { + await runStep("end_session", () => steps.endSession(slots, decision)); + await runStep("settle_reuse", () => steps.settleReuse(slots, decision)); + await runStep("stop_transport", () => steps.stopTransport(slots)); + await runStep("sync_back", () => steps.syncBack(slots)); + } finally { + // The staging lease release runs in a `finally`, so an earlier step fault + // never strands the lease. + try { + steps.releaseStagingLease(slots); + } catch (error) { + await steps.recordError("release_staging_lease", error); + } + } + + return buildReport(entries, decision); +} + +/** Build the settled slots view from the claimed entries. */ +function makeSlots(entries: readonly SettledResourceEntry[]): SettlementSlots { + const byId = new Map(); + for (const entry of entries) byId.set(entry.id, entry); + return { + get(id: Id): RunResourcePayloads[Id] | undefined { + return byId.get(id)?.payload as RunResourcePayloads[Id] | undefined; + }, + has(id: ResourceId): boolean { + return byId.has(id); + }, + }; +} + +/** + * Build the final per-resource disposition report. The claim-time value the + * ledger stamps ("transferred") is not the final report; the settlement decides + * it here. A saved resource is `transferred`; every other resource is + * `finalized`. + */ +function buildReport( + entries: readonly SettledResourceEntry[], + decision: ReuseDecision, +): SettlementDispositionReport { + const savedId = decision.kind === "save" ? decision.savedId : null; + return { + records: entries.map((entry) => ({ + id: entry.id, + disposition: entry.id === savedId ? ("transferred" as const) : ("finalized" as const), + })), + }; +} diff --git a/packages/adapter-utils/src/acpx-engine/startup-characterization.test.ts b/packages/adapter-utils/src/acpx-engine/startup-characterization.test.ts index 8f55ec0ae6..9367f933e8 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-characterization.test.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-characterization.test.ts @@ -629,7 +629,7 @@ describe("ACPX engine startup characterization", () => { expect(stop).toHaveBeenCalledTimes(1); }); - it("cold ensure_session throw: ensure_session error, and the handshake-catch close does NOT fire (no handle yet)", async () => { + it("cold ensure_session throw: ensure_session error, and the runtime closes (the cold-handshake leak is fixed)", async () => { const root = await makeTempRoot(); const closeSpy = vi.fn(async () => {}); const execute = createAcpxEngineExecutor({ @@ -655,11 +655,10 @@ describe("ACPX engine startup characterization", () => { expect(result.exitCode).toBe(1); expect(result.resultJson?.phase).toBe("ensure_session"); - // Current behavior (execute.ts ~:3590): the handshake catch closes the runtime - // only `if (handle)`. A cold `ensureSession` throw never assigns a handle, so - // this close never fires. The warm-hit path below has a cached handle and does - // close. This pins the real cold-path behavior, not an aspiration. - expect(closeSpy).not.toHaveBeenCalled(); + // The runtime enters the ledger the moment it is created, so a cold + // `ensureSession` throw (before any handle exists) still closes it through the + // settlement `endSession` step. This closes the former cold-handshake leak. + expect(closeSpy).toHaveBeenCalledTimes(1); }); it("missing session handle: ensure_session error and the minimal runtime closes", async () => { @@ -728,7 +727,7 @@ describe("ACPX engine startup characterization", () => { expect(closeSpy).toHaveBeenCalledTimes(1); }); - it("warm-hit failure: reuses the runtime, fails with ensure_session, closes it, drops the warm entry", async () => { + it("persistent host failure: the first run closes and relaunches, so a failing second run re-creates and closes", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); const startedAt = "2026-01-01T00:00:00.000Z"; @@ -771,10 +770,13 @@ describe("ACPX engine startup characterization", () => { onSpawn: async () => {}, } as never); expect(first.exitCode).toBe(0); - expect(warmHandles.size).toBe(1); + // Amendment B: the first clean persistent turn closes and relaunches instead + // of warm-saving the runtime, so no warm handle survives. + expect(warmHandles.size).toBe(0); expect(created).toBe(1); - // The warm-hit reuses runtime #1 and fails while persisting process identity. + // The second run finds no warm handle, so it re-creates the runtime and fails + // while persisting process identity during ensure_session. const second = await execute({ runId: "warm-2", agent: { id: "agent-1", companyId: "company-1" }, @@ -788,12 +790,12 @@ describe("ACPX engine startup characterization", () => { }, } as never); - // No new runtime was created; the reused runtime is closed and the warm entry - // removed, with the failure reported on the ensure_session phase. - expect(created).toBe(1); + // The second run built runtime #2 (no warm reuse), closed it on the failure, + // and reported the ensure_session phase. Both runtimes closed (one per run). + expect(created).toBe(2); expect(second.exitCode).toBe(1); expect(second.resultJson?.phase).toBe("ensure_session"); - expect(closeSpy).toHaveBeenCalledTimes(1); + expect(closeSpy).toHaveBeenCalledTimes(2); expect(warmHandles.size).toBe(0); }); }); @@ -820,7 +822,7 @@ describe("ACPX engine startup characterization", () => { expect(runtimeOptions[0]?.cwd).toBe(localCwd); }); - it("persistent host lane: warm-saves the handle so a second run reuses the runtime", async () => { + it("persistent host lane: closes and relaunches the handle, so a second run re-creates the runtime", async () => { const root = await makeTempRoot(); const stateDir = path.join(root, "state"); let created = 0; @@ -857,9 +859,11 @@ describe("ACPX engine startup characterization", () => { expect(first.exitCode).toBe(0); expect(second.exitCode).toBe(0); - // The warm handle survives, so the second run reuses runtime #1. - expect(created).toBe(1); - expect(warmHandles.size).toBe(1); + // Amendment B: the host lane never warm-saves a live runtime (the run-minted + // API key is never revoked), so the first run closes and relaunches. No warm + // handle survives, so the second run re-creates runtime #2. + expect(created).toBe(2); + expect(warmHandles.size).toBe(0); }); it("remote process-session lane: does NOT warm-save the handle, so a second run re-creates the runtime", async () => { diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts index 14c7e0bfe2..ea0decb916 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts @@ -4,12 +4,15 @@ import type { StartupSpan, StartupTraceContext, StartupTracer } from "./startup- import { clampSpanLabel, createRuntimeSpanRunner, + emitRunPhaseTiming, emitSkippedStartupStep, getActiveStepContext, measureStartupStep, NOOP_STARTUP_SPAN, NOOP_STARTUP_TRACE_CONTEXT, normalizeProviderFamily, + RUN_PHASE_NAMES, + RUN_PHASE_TIMING_EVENT_TYPE, runWithoutActiveStep, runWithRuntimeParent, SANDBOX_STARTUP_SPAN_ATTR_PREFIX, @@ -712,3 +715,59 @@ describe("createRuntimeSpanRunner", () => { expect(childStep).toBeNull(); }); }); + +describe("run phase timing telemetry", () => { + it("test_phase_telemetry_fields_are_exactly_phase_duration_outcome", async () => { + const events: AdapterRuntimeEvent[] = []; + const ctx = { onEvent: async (event: AdapterRuntimeEvent) => void events.push(event) }; + + await emitRunPhaseTiming(ctx, "create_runtime", 12, "ok"); + + expect(events).toHaveLength(1); + const event = events[0]!; + expect(event.eventType).toBe(RUN_PHASE_TIMING_EVENT_TYPE); + // The payload carries exactly the three closed fields and nothing else. + expect(Object.keys(event.payload ?? {}).sort()).toEqual(["durationMs", "outcome", "phase"]); + expect(event.payload).toEqual({ phase: "create_runtime", durationMs: 12, outcome: "ok" }); + }); + + it("test_phase_telemetry_emits_no_command_path_environment_or_identifier_value", async () => { + const events: AdapterRuntimeEvent[] = []; + const ctx = { onEvent: async (event: AdapterRuntimeEvent) => void events.push(event) }; + + // Every allowlisted phase emits exactly one closed-shape event. + for (const phase of RUN_PHASE_NAMES) { + await emitRunPhaseTiming(ctx, phase, 5, "ok"); + } + // A phase name outside the closed allowlist emits nothing, so a free-form + // label (a command, a path, an environment value, or a raw identifier) can + // never reach the stream. + await emitRunPhaseTiming(ctx, "/usr/bin/node --flag /secret/path", 5, "ok"); + await emitRunPhaseTiming(ctx, "PAPERCLIP_API_KEY=abc123", 5, "ok"); + await emitRunPhaseTiming(ctx, "run-7f3a-agent-42", 5, "ok"); + + expect(events).toHaveLength(RUN_PHASE_NAMES.length); + for (const event of events) { + // The phase field is only ever an allowlisted name. + expect(RUN_PHASE_NAMES).toContain(event.payload?.phase); + // No field carries a command, a path, an environment value, or an identifier. + const serialized = JSON.stringify(event); + expect(serialized).not.toContain("/usr/bin/node"); + expect(serialized).not.toContain("/secret/path"); + expect(serialized).not.toContain("PAPERCLIP_API_KEY"); + expect(serialized).not.toContain("abc123"); + expect(serialized).not.toContain("run-7f3a-agent-42"); + } + }); + + it("test_telemetry_failure_does_not_fail_the_run", async () => { + const ctx = { + onEvent: async () => { + throw new Error("telemetry sink boom"); + }, + }; + + // A throwing telemetry sink never propagates, so the run continues. + await expect(emitRunPhaseTiming(ctx, "turn", 9, "failed")).resolves.toBeUndefined(); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index ce91ca6a5b..5e70bdbdb3 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -786,3 +786,72 @@ export async function emitSkippedStartupStep( // Observability must not change startup control flow. } } + +/** + * Structured event emitted once per named run-lifecycle phase, so the duration + * and the outcome of each phase land in the run-events stream. It is + * observability-only and rides the existing `ctx.onEvent` bridge. The payload is + * a closed shape: exactly `phase`, `durationMs`, and `outcome`. The phase name is + * from a closed allowlist, so the event never carries a command, an argument, a + * path, an environment value, or a raw identifier. + */ +export const RUN_PHASE_TIMING_EVENT_TYPE = "run.phase.timing"; + +/** + * The closed set of run-lifecycle phase names. A phase-timing event may name only + * one of these. The list is fixed and low-cardinality; it never derives from run + * or user data. + */ +export const RUN_PHASE_NAMES = [ + "place_workspace", + "start_transport", + "create_runtime", + "ensure_session", + "configure_session", + "prepare_turn", + "turn", + "end_session", + "settle_reuse", + "stop_transport", + "sync_back", + "release_staging_lease", +] as const; + +/** One run-lifecycle phase name from the closed allowlist. */ +export type RunPhaseName = (typeof RUN_PHASE_NAMES)[number]; + +const RUN_PHASE_NAME_SET: ReadonlySet = new Set(RUN_PHASE_NAMES); + +/** The closed outcome set for a phase-timing event. */ +export type RunPhaseOutcome = "ok" | "failed"; + +/** + * Emit exactly one `run.phase.timing` event for a run-lifecycle phase. The + * payload carries only `phase`, `durationMs`, and `outcome`. It never carries a + * command, an argument, a path, an environment value, or a raw identifier. The + * phase name must be one member of the closed allowlist; a name outside the + * allowlist emits nothing, so a free-form label can never reach the stream. A + * negative or a non-finite duration clamps to 0. Every sink call sits inside an + * error swallow, so a throwing telemetry sink never fails the run. + */ +export async function emitRunPhaseTiming( + ctx: Pick, + phase: string, + durationMs: number, + outcome: RunPhaseOutcome, +): Promise { + if (!RUN_PHASE_NAME_SET.has(phase)) return; + const safeDuration = Number.isFinite(durationMs) && durationMs > 0 ? durationMs : 0; + const event: AdapterRuntimeEvent = { + eventType: RUN_PHASE_TIMING_EVENT_TYPE, + stream: "system", + level: "info", + message: `run phase: ${phase} (${safeDuration}ms)`, + payload: { phase, durationMs: safeDuration, outcome }, + }; + try { + await ctx.onEvent?.(event); + } catch { + // Telemetry never fails the run. + } +} diff --git a/packages/adapter-utils/src/acpx-engine/turn-sequence.test.ts b/packages/adapter-utils/src/acpx-engine/turn-sequence.test.ts new file mode 100644 index 0000000000..aef9c72a91 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/turn-sequence.test.ts @@ -0,0 +1,190 @@ +import { describe, expect, it, vi } from "vitest"; +import { runTurn, type StartedTurn, type TurnFinalizeInput, type TurnSteps } from "./turn-sequence.js"; +import { createRunResourceLedger } from "./run-resource-ledger.js"; +import type { TurnCompletion } from "./run-contracts.js"; + +// The turn sequence owns the run order, the wall-clock timer, and the abort +// controller. These tests drive it with fakes for the five steps and assert the +// order, the never-reject guarantee, and the timer ownership. They never touch +// the engine's real run state. + +function finalizedTurn(): TurnCompletion { + return { kind: "finalized" }; +} + +function failedTurn(error: unknown): TurnCompletion { + return { + kind: "failed", + cause: { kind: "turn_failed", error: error instanceof Error ? error : new Error(String(error)) }, + resources: createRunResourceLedger().takeForSettlement(), + }; +} + +function noopStarted(): StartedTurn { + return { cancel: async () => {} }; +} + +// A base set of steps whose bodies do nothing. Each test overrides the steps it +// drives and records the calls it cares about. +function baseSteps(order: string[], overrides: Partial> = {}): TurnSteps { + return { + timeoutMs: overrides.timeoutMs, + timeoutMessage: overrides.timeoutMessage ?? "timed out", + promptBuild: overrides.promptBuild ?? (async () => { + order.push("promptBuild"); + }), + preTurnUsage: overrides.preTurnUsage ?? (async () => { + order.push("preTurnUsage"); + }), + turnStart: overrides.turnStart ?? (() => { + order.push("turnStart"); + return noopStarted(); + }), + eventRelay: overrides.eventRelay ?? (async () => { + order.push("eventRelay"); + return "terminal"; + }), + turnFinalize: overrides.turnFinalize ?? (async (input) => { + order.push(`turnFinalize:${input.kind}`); + return input.kind === "terminal" ? finalizedTurn() : failedTurn(input.error); + }), + }; +} + +describe("ACPX turn sequence", () => { + it("test_run_turn_never_rejects_every_error_is_a_completion", async () => { + // A step error must become a `TurnCompletion`, never a rejection. The event + // relay throws after the turn started, so the sequence reports a `turn` + // failure and returns a completion. + const finalizeInputs: TurnFinalizeInput[] = []; + const boom = new Error("relay boom"); + const steps = baseSteps([], { + eventRelay: async () => { + throw boom; + }, + turnFinalize: async (input) => { + finalizeInputs.push(input); + return input.kind === "terminal" ? finalizedTurn() : failedTurn(input.error); + }, + }); + + // The returned promise resolves; it never rejects. + const completion = await runTurn(steps); + expect(completion.kind).toBe("failed"); + // The error routed through the one finalize step with the `turn` phase. + expect(finalizeInputs).toHaveLength(1); + expect(finalizeInputs[0]).toMatchObject({ kind: "error", error: boom, phase: "turn" }); + }); + + it("test_run_turn_owns_timer_and_abort_controller", async () => { + vi.useFakeTimers(); + try { + const captured: { signal: AbortSignal | null; cancelReason: string | null } = { + signal: null, + cancelReason: null, + }; + let releaseRelay: (value: string) => void = () => {}; + const relayPromise = new Promise((resolve) => { + releaseRelay = resolve; + }); + const finalizeInputs: TurnFinalizeInput[] = []; + const steps = baseSteps([], { + timeoutMs: 1000, + timeoutMessage: "wall-clock timeout", + turnStart: (signal) => { + captured.signal = signal; + return { + cancel: async (reason) => { + captured.cancelReason = reason; + }, + }; + }, + eventRelay: async () => relayPromise, + turnFinalize: async (input) => { + finalizeInputs.push(input); + return input.kind === "terminal" ? finalizedTurn() : failedTurn(input.error); + }, + }); + + const runPromise = runTurn(steps); + // Let promptBuild, preTurnUsage, and turnStart run. + await vi.advanceTimersByTimeAsync(0); + expect(captured.signal?.aborted).toBe(false); + // Fire the wall-clock timeout the sequence owns. + await vi.advanceTimersByTimeAsync(1000); + // The sequence owns the abort controller: the shared signal is aborted... + expect(captured.signal?.aborted).toBe(true); + // ...and it cancelled the started turn with the timeout message. + expect(captured.cancelReason).toBe("wall-clock timeout"); + // Complete the relay so the sequence finalizes the timed-out turn. + releaseRelay("terminal"); + const completion = await runPromise; + expect(completion).toEqual({ kind: "finalized" }); + // The finalize step saw the timeout flag the sequence tracked. + expect(finalizeInputs).toHaveLength(1); + expect(finalizeInputs[0]).toMatchObject({ kind: "terminal", timedOut: true }); + } finally { + vi.useRealTimers(); + } + }); + + it("test_prompt_build_failure_is_failed_completion_with_prepare_turn", async () => { + // A prompt-build failure happens before the turn starts, so the sequence + // reports a `prepare_turn` failure and never runs the later steps. + const order: string[] = []; + const finalizeInputs: TurnFinalizeInput[] = []; + const boom = new Error("prompt build boom"); + const steps = baseSteps(order, { + promptBuild: async () => { + throw boom; + }, + turnFinalize: async (input) => { + finalizeInputs.push(input); + return input.kind === "terminal" ? finalizedTurn() : failedTurn(input.error); + }, + }); + + const completion = await runTurn(steps); + expect(completion.kind).toBe("failed"); + expect(finalizeInputs).toHaveLength(1); + expect(finalizeInputs[0]).toMatchObject({ kind: "error", error: boom, phase: "prepare_turn" }); + // The later steps never ran. + expect(order).not.toContain("preTurnUsage"); + expect(order).not.toContain("turnStart"); + expect(order).not.toContain("eventRelay"); + }); + + it("test_run_turn_finalizes_no_resource", async () => { + // Structural proof: the sequence borrows the resources and finalizes none. + // The `TurnSteps` contract has no resource-release method; the sequence runs + // only the five ordered steps. A test-side resource whose release the + // sequence could never call stays untouched, and the recorded call order is + // exactly the five steps. The coordinator settles the resources after the + // turn, so a resource release inside the sequence would be a double release. + const order: string[] = []; + let released = false; + // A resource the sequence has no hook to release. If the sequence ever + // released a resource, it would have to reach one, and it cannot. + const borrowedResource = { + release: () => { + released = true; + }, + }; + void borrowedResource; + + const steps = baseSteps(order); + const completion = await runTurn(steps); + + expect(completion).toEqual({ kind: "finalized" }); + // The sequence ran exactly the five ordered steps, then finalized. + expect(order).toEqual([ + "promptBuild", + "preTurnUsage", + "turnStart", + "eventRelay", + "turnFinalize:terminal", + ]); + // It never released the borrowed resource. + expect(released).toBe(false); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/turn-sequence.ts b/packages/adapter-utils/src/acpx-engine/turn-sequence.ts new file mode 100644 index 0000000000..b1abf77539 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/turn-sequence.ts @@ -0,0 +1,100 @@ +// The turn sequence. +// +// `runTurn(steps)` is a plain sequence. It runs five steps in a fixed order: +// promptBuild, preTurnUsage, turnStart, eventRelay, turnFinalize. It never +// rejects: every step error becomes a typed `TurnCompletion` through the +// `turnFinalize` step. The sequence owns the wall-clock timer and the abort +// controller. It borrows the run resources; it finalizes no resource. The +// coordinator (Phase 20) settles the resources after the turn. +// +// The sequence is data, not a graph and not a state machine. The engine supplies +// each step; the sequence owns only the order, the timer, and the abort signal. +// This keeps the sequence free of the engine's private run state. + +import type { TurnCompletion } from "./run-contracts.js"; + +/** + * A turn the site started. The sequence borrows it to cancel it on a timeout; it + * releases nothing. The engine's `eventRelay` step reads the events and the + * terminal result. + */ +export interface StartedTurn { + cancel(reason: string): Promise; +} + +/** + * The input the sequence hands to the one `turnFinalize` step. A `terminal` + * input carries the turn's terminal result; an `error` input carries the step + * error and the phase it failed in. Both carry the wall-clock timeout flag. + */ +export type TurnFinalizeInput = + | { readonly kind: "terminal"; readonly terminal: TTerminal; readonly timedOut: boolean } + | { + readonly kind: "error"; + readonly error: unknown; + readonly phase: "prepare_turn" | "turn"; + readonly timedOut: boolean; + }; + +/** + * The five steps the sequence runs, plus the timer inputs. The engine implements + * each step over its own run state. `turnFinalize` maps a terminal result or a + * step error to a `TurnCompletion`; it must not reject, so the sequence never + * rejects. + */ +export interface TurnSteps { + /** The wall-clock timeout, in milliseconds, or undefined for no timeout. */ + readonly timeoutMs: number | undefined; + /** The message the timeout cancel carries. */ + readonly timeoutMessage: string; + /** Build the prompt and emit the run metadata. A failure here is `prepare_turn`. */ + promptBuild(signal: AbortSignal): Promise; + /** Snapshot the pre-turn usage, so this run's cost excludes earlier runs. */ + preTurnUsage(): Promise; + /** Start the turn with the run's six inputs. The sequence adds the signal and the timeout. */ + turnStart(signal: AbortSignal, timeoutMs: number | undefined): StartedTurn; + /** Relay the turn events and return the terminal result. */ + eventRelay(turn: StartedTurn): Promise; + /** Map a terminal result or a step error to a `TurnCompletion`. Must not reject. */ + turnFinalize(input: TurnFinalizeInput): Promise; +} + +/** + * Run the turn sequence. It returns a `TurnCompletion` on every path and never + * rejects. A failure before `turnStart` returns is a `prepare_turn` failure; a + * failure after it is a `turn` failure. Only the reported phase differs. + */ +export async function runTurn(steps: TurnSteps): Promise { + const controller = new AbortController(); + let timeout: ReturnType | null = null; + let timedOut = false; + let started: StartedTurn | null = null; + // `turnStarted` separates a pre-turn preparation failure from a running-turn + // failure, so `turnFinalize` reports the right phase. + let turnStarted = false; + try { + // Build the prompt and snapshot the pre-turn usage inside the failure + // boundary. A failure here is a `prepare_turn` failure. + await steps.promptBuild(controller.signal); + await steps.preTurnUsage(); + // The sequence owns the wall-clock timer. On a timeout it marks the run timed + // out, aborts the shared signal, and cancels the started turn. The cancel + // no-ops before `turnStart` returns, because `started` is still null. + if (steps.timeoutMs !== undefined) { + timeout = setTimeout(() => { + timedOut = true; + controller.abort(); + void started?.cancel(steps.timeoutMessage).catch(() => {}); + }, steps.timeoutMs); + } + started = steps.turnStart(controller.signal, steps.timeoutMs); + turnStarted = true; + const terminal = await steps.eventRelay(started); + if (timeout) clearTimeout(timeout); + return await steps.turnFinalize({ kind: "terminal", terminal, timedOut }); + } catch (error) { + if (timeout) clearTimeout(timeout); + const phase: "prepare_turn" | "turn" = turnStarted ? "turn" : "prepare_turn"; + return await steps.turnFinalize({ kind: "error", error, phase, timedOut }); + } +} diff --git a/packages/shared/src/telemetry/README.md b/packages/shared/src/telemetry/README.md index 329640cdf8..7eb7c40f9a 100644 --- a/packages/shared/src/telemetry/README.md +++ b/packages/shared/src/telemetry/README.md @@ -306,6 +306,45 @@ ensure-session sub-times. To read the detailed timing, use the startup spans. The spans need an OTLP endpoint. A run with no endpoint keeps only the three run-log fields above. +## Run Phase Timing Run-Log Event + +Paperclip writes one `run.phase.timing` event to the run log for each +run-lifecycle phase. This event is a run-log record, not a first-party telemetry +event. The generated telemetry contract does not cover it, so this section is its +canonical contract. The producer is `emitRunPhaseTiming` in +`packages/adapter-utils/src/acpx-engine/startup-timing.ts`. + +The event payload carries only three fields. + +| Field | Type | Meaning | +| --- | --- | --- | +| `phase` | string | The run-lifecycle phase name from the closed allowlist below. | +| `durationMs` | number | The wall time of the phase. A negative or a non-finite value clamps to `0`. | +| `outcome` | string | The phase outcome (`ok` or `failed`). | + +The `phase` field is one member of a closed, low-cardinality allowlist. The +producer drops any event whose phase name is outside this allowlist, so a +free-form label never reaches the run log. The allowlist has twelve phase names. + +| Phase | Meaning | +| --- | --- | +| `place_workspace` | Place the run workspace. | +| `start_transport` | Start the agent transport. | +| `create_runtime` | Create the agent runtime. | +| `ensure_session` | Ensure the agent session exists. | +| `configure_session` | Configure the agent session. | +| `prepare_turn` | Prepare the turn. | +| `turn` | Run the turn. | +| `end_session` | End the agent session. | +| `settle_reuse` | Settle the session for reuse. | +| `stop_transport` | Stop the agent transport. | +| `sync_back` | Sync the workspace back. | +| `release_staging_lease` | Release the staging lease. | + +The payload never carries a command, an argument, a path, an environment value, +or a raw identifier. The event rides the `ctx.onEvent` run-event bridge and is +observability-only. It needs no OTLP endpoint. + ## Dimension Values Telemetry dimension values must be primitives. Use only the value types allowed