feat(runtime): managed Tailscale HTTPS lifecycle, durable runtime leases, and bounded control recovery (#11525)
<!-- Simplified Technical English (ASD-STE100). --> > **Stacked pull request.** This targets #11524. Merge #11524 first. Review only the second commit, `feat(runtime): managed Tailscale HTTPS lifecycle...`. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip starts and supervises managed runtime services, so an agent's branch can be previewed while the agent works > - The previous pull request added the host broker, the shared contract, and the database columns, but no code used them > - A managed runtime can only be exposed over HTTPS if it holds a stable loopback port pair for the whole life of the service. The current control path cannot promise this: two controls can race the same execution workspace, a stranded control can stay `running` forever, and a start can adopt a port it does not own > - This pull request adds the HTTPS lifecycle and the control-path hardening that the lifecycle depends on > - The benefit is that a managed preview becomes reachable from another device, and a managed control now always reaches a terminal state ## Linked Issues or Issue Description No public GitHub issue exists. The change follows the feature request template. **Subsystem affected** Managed workspace runtime services, workspace operations, the execution workspace routes, and the workspace runtime UI. **Problem or motivation** A managed runtime service is reachable only on loopback, so a preview cannot be opened from a phone or a second computer. Exposing it safely needs an exclusively held port pair. Three existing gaps block that. Overlapping controls can race the same workspace. A control whose owner dies stays `running` and blocks the lane forever. Port allocation does not confirm that the process holding a port is the process Paperclip spawned. **Proposed solution** Add the exposure lifecycle on top of the broker from #11524: reserve before spawn, expose after readiness, validate the public URL, and remove on stop. In the same change, make managed controls mutually exclusive per workspace, give each control a durable issue-owned lease and a terminal state, and verify port ownership before use. **Alternatives considered** - Add HTTPS exposure without the control hardening. This was rejected because a raced or stranded control makes exposure point at the wrong process. - Guard the lane with an in-memory lock only. This was rejected because the lock does not survive a server restart, so the lane can be lost or double-claimed. - Trust the requested bind address. This was rejected because a checkout that predates managed HTTPS overwrites `PAPERCLIP_BIND` from its own `--bind` argument, and then binds the wildcard address. **Roadmap alignment** This completes the managed workspace runtime capability that already exists. It adds no new product surface beyond the HTTPS link. **Additional context** This is the second of three pull requests. The third adds central mediation of leased port pairs. ## What Changed Exposure lifecycle: - Add the server-side broker client and the exposure lifecycle manager. The manager reserves the mapping before spawn, exposes after backend readiness, validates the public URL, and removes the mapping on stop. - Default managed worktree runtimes to `tailscale_https`, read exposure intent from legacy `expose` blocks, and backfill runtimes that are still HTTP-only. - Verify listener ownership for the app port and its Vite HMR companion before the broker is asked to expose anything. An unrelated listener on either port fails the start closed. - Force the loopback bind through argv instead of environment hints. Leave a non-Paperclip service's `--bind` argument alone. - Probe loopback for readiness instead of the public URL, and give Vite HMR its own loopback-bound server in middleware mode. - Preserve operator-declared Serve mappings across the managed lifecycle, so cleanup never removes a mapping that Paperclip did not create. - Name which listener predicate denied an expose, so an operator can act on the message. Control-path hardening: - Make `start`, `stop`, `restart`, and job `run` mutually exclusive per execution workspace. An overlap gets `409 workspace_runtime_control_in_progress`, and authorization is still checked first. - Take a durable exclusivity lease on the execution workspace, owned by the controlling issue. A different issue gets `409 workspace_runtime_lease_conflict` before any operation is recorded. Board and operator actions bypass the lease. - Give every control a terminal state. Each control stamps its owning process and pid, heartbeats while it runs, and has a wall-clock ceiling. Recovery of a stranded control uses a compare-and-swap on `updated_at`, so a live owner is never stolen. - Bound readiness probes, verify allocated port ownership on POSIX and Windows, harden sibling port allocation, and reconcile desired runtimes on server startup. - Surface exposure state and bounded runtime errors in the workspace runtime UI. - Record the new behavior in `doc/DEVELOPING.md`. ## Verification Focused checks, all run on this branch: - `npx tsc --noEmit -p server/tsconfig.json` — 139 errors, exactly the count on `master`. All 139 come from the unbuilt `@paperclipai/plugin-sdk` package. - `pnpm --filter @paperclipai/ui typecheck` — clean. - Server suites, 177 tests pass across 9 files: `workspace-runtime.test.ts`, `workspace-runtime-leases.test.ts`, `workspace-runtime-control-recovery.test.ts`, `execution-workspace-runtime-control-conflict.test.ts`, `execution-workspace-runtime-lease-route.test.ts`, `workspace-operations-reconciliation.test.ts`, `workspace-runtime-start-terminality.test.ts`, `app-hmr-port.test.ts`, and `workspace-runtime-ready-comment.test.ts`. - Exposure unit suites, 77 tests pass: `src/services/runtime-exposure/` and `workspace-runtime-exposure-backfill.test.ts`. - UI: `WorkspaceRuntimeControls.test.tsx` and `WorkspaceServiceControlBar.test.tsx` — 34 tests pass. **One suite is red on the development host and is expected to be green in CI.** `server/src/services/workspace-runtime-exposure.test.ts` has 10 failures on the machine used to write this branch. The cause is host contamination, not the code. That machine already runs an HTTPS canary that holds ports 42000, 42001, 52000, and 52001 on a tailnet address. The suite allocates from the same range, so the new listener-ownership check correctly reports: ``` listener_ownership_mismatch — port 42000 is bound to 100.123.243.20, 127.0.0.1, fd7a:115c:a1e0:0:0:0:dd3a:f314 ... instead of loopback only ``` A CI runner has no listener on those ports, so the check sees loopback only and the suite passes. Please confirm this from the CI result on this pull request rather than from a local run on a host that already exposes a managed runtime. This is a real weakness of the current test fixture, and the third pull request in the series removes it by allocating the pair through a central mediator instead of a stubbed availability check. `workspace-runtime-https-live-exercise.test.ts` needs a live `tailscale` host and was not run locally. ## Risks - This is the behavior-bearing pull request of the three, so it carries the most risk. - Two new `409` responses appear on managed control routes. A caller that assumed a control always starts must handle a conflict. Board and operator actions are deliberately exempt, so an agent lease cannot lock an operator out. - Managed worktree runtimes now default to `tailscale_https`. If the host has no working broker, the start fails closed and reports the exposure failure instead of silently serving plain HTTP. This is intended, and it is the reason the failure message names the denying predicate. - Startup reconciliation touches persisted runtime rows. It is scoped to desired state and does not resurrect a service that never came up. - The lease has a 30-minute time to live and explicit release paths, so a crashed owner cannot hold a lane forever. - No migration runs in this pull request. The tables and columns land in #11524. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. ## Model Used Claude Opus 5 (`claude-opus-5`), 1M context window, extended thinking, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass, with the one host-contaminated suite explained above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
parent
bfc19e2ebd
commit
4c349fe6b7
|
|
@ -648,6 +648,22 @@ For project execution worktrees, Paperclip can also run a project-defined provis
|
|||
|
||||
Heavier setup that is only needed by a managed runtime service can use `workspaceStrategy.runtimeProvisionCommand`. Paperclip runs this command lazily before spawning the first service in a start batch, serializes concurrent provisioning for the same workspace, and records the attempt as `workspace_runtime_provision`. The command receives the same workspace environment as `provisionCommand` and should be idempotent because later service-start batches invoke it again.
|
||||
|
||||
Managed runtime control actions (`start`, `stop`, `restart`, and job `run`) are mutually exclusive per execution workspace. An overlapping control is rejected with `409 workspace_runtime_control_in_progress` instead of racing the active operation, and authorization is still checked first, so the conflict never widens who may control a workspace.
|
||||
|
||||
Every managed control reaches a terminal operation state. Each one stamps the owning server process and pid on its `workspace_operations` row and heartbeats while it runs, and each one carries a wall-clock ceiling (30 minutes for lifecycle controls, 4 hours for workspace jobs) so a hung provider or listener fails the operation rather than leaving it active. When a start fails part-way, Paperclip tears the workspace's runtime services down through the ordinary stop path and records a stopped desired state, so the lane is retryable and a startup reconcile will not resurrect a service that never came up.
|
||||
|
||||
Recovery of stranded controls is bounded and cannot steal a live operation. A `running` control is only terminalized when its owning process is gone, when the owning request in this process no longer exists, or after 60 seconds without a heartbeat; the terminalizing write is a compare-and-swap on `updated_at`, so an owner that heartbeats concurrently keeps its operation. Recovery runs on server startup and before each managed control, appends reconciliation evidence to the workspace-operation log, and stays inside the requested workspace's scope.
|
||||
|
||||
Readiness probes and port allocation are bounded for the same reason. Each HTTP readiness probe is aborted after at most 5 seconds (never past the service's readiness budget), so a foreign listener that accepts a connection but never answers cannot park a start forever. Auto-allocated loopback ports are reserved in-process for the duration of a start and re-checked for a live owner, and a configured port already claimed by another in-flight start fails that start terminally — so two isolated workspaces asking for the same app/HMR pair either get distinct healthy ports or one fails cleanly and retryably.
|
||||
|
||||
Beyond that in-flight guard, `start`, `stop`, and `restart` also take a durable exclusivity lease on the execution workspace (`execution_workspace_runtime_leases`). The lease is keyed by execution workspace and owned by the controlling issue (or, when a run has no issue in scope, by the run or agent), so it survives across calls, heartbeats, and server processes. The owning issue can keep operating; a different issue or run is rejected with `409 workspace_runtime_lease_conflict` before any workspace operation is recorded and before any runtime service is touched. Board/operator actions bypass the lease entirely and never take the lane away from an agent.
|
||||
|
||||
Lease recovery is bounded and explicit. Another issue may reclaim the lane once the owner becomes ineligible — the owning issue reaches a terminal status, is hidden, or is deleted; the owning run reaches a terminal status — or once the lease's 30-minute TTL elapses without the owner touching it. Archiving the execution workspace releases the lease outright. Conflict responses carry the owning issue/run ids and the lease expiry so an operator can tell who holds the lane.
|
||||
|
||||
For Tailscale HTTPS exposure, readiness includes stable listener-ownership checks for every requested loopback port (the app and, when configured, its Vite HMR companion). Each listener must belong to the spawned managed process group; an unrelated listener that races onto either reserved port fails the start closed before the broker is asked to expose it.
|
||||
|
||||
In Vite middleware mode, Paperclip gives HMR a dedicated HTTP server bound to the managed runtime's loopback host. The browser still derives the HMR hostname from the public HTTPS page, so listener containment does not break remote hot reload.
|
||||
|
||||
## App-Shipped Skills Catalog
|
||||
|
||||
The Paperclip app ships a curated catalog of company skills out of the box. The
|
||||
|
|
|
|||
|
|
@ -90,6 +90,7 @@ const tailscaleAuthFlagNames = new Set([
|
|||
let tailscaleAuth = false;
|
||||
let bindMode: BindMode | null = null;
|
||||
let bindHost: string | null = null;
|
||||
const managedRuntimeExposure = process.env.PAPERCLIP_MANAGED_RUNTIME_EXPOSURE === "tailscale_https";
|
||||
const forwardedArgs: string[] = [];
|
||||
|
||||
for (let index = 0; index < cliArgs.length; index += 1) {
|
||||
|
|
@ -133,6 +134,10 @@ if (!bindMode && process.env.npm_config_bind && BIND_MODES.includes(process.env.
|
|||
if (!bindHost && process.env.npm_config_bind_host) {
|
||||
bindHost = process.env.npm_config_bind_host;
|
||||
}
|
||||
if (managedRuntimeExposure) {
|
||||
bindMode = "custom";
|
||||
bindHost = "127.0.0.1";
|
||||
}
|
||||
if (bindMode === "custom" && !bindHost) {
|
||||
console.error("[paperclip] --bind custom requires --bind-host <host>");
|
||||
process.exit(1);
|
||||
|
|
@ -174,7 +179,7 @@ if (tailscaleAuth || bindMode) {
|
|||
} else {
|
||||
env.PAPERCLIP_DEPLOYMENT_MODE = "authenticated";
|
||||
env.PAPERCLIP_DEPLOYMENT_EXPOSURE = "private";
|
||||
env.PAPERCLIP_AUTH_BASE_URL_MODE = "auto";
|
||||
env.PAPERCLIP_AUTH_BASE_URL_MODE = managedRuntimeExposure ? "explicit" : "auto";
|
||||
console.log(
|
||||
`[paperclip] dev mode: authenticated/private (bind=${effectiveBind}${bindHost ? `:${bindHost}` : ""})`,
|
||||
);
|
||||
|
|
|
|||
|
|
@ -1,5 +1,11 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import { resolveViteHmrHost, resolveViteHmrPort } from "../app.ts";
|
||||
import { createServer } from "node:http";
|
||||
import {
|
||||
listenViteHmrServer,
|
||||
resolveViteHmrHost,
|
||||
resolveViteHmrPort,
|
||||
resolveViteHmrProtocol,
|
||||
} from "../app.ts";
|
||||
|
||||
describe("resolveViteHmrPort", () => {
|
||||
it("uses serverPort + 10000 when the result stays in range", () => {
|
||||
|
|
@ -24,8 +30,30 @@ describe("resolveViteHmrHost", () => {
|
|||
expect(resolveViteHmrHost("::")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps concrete bind hosts", () => {
|
||||
expect(resolveViteHmrHost("127.0.0.1")).toBe("127.0.0.1");
|
||||
it("uses the browser hostname for loopback while keeping non-loopback concrete hosts", () => {
|
||||
expect(resolveViteHmrHost("127.0.0.1")).toBeUndefined();
|
||||
expect(resolveViteHmrHost("localhost")).toBeUndefined();
|
||||
expect(resolveViteHmrHost("paperclip-dev")).toBe("paperclip-dev");
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveViteHmrProtocol", () => {
|
||||
it("selects secure WebSockets for HTTPS branch runtimes", () => {
|
||||
expect(resolveViteHmrProtocol("wss")).toBe("wss");
|
||||
expect(resolveViteHmrProtocol(undefined)).toBeUndefined();
|
||||
expect(() => resolveViteHmrProtocol("https")).toThrow(/ws or wss/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("listenViteHmrServer", () => {
|
||||
it("binds the dedicated middleware-mode HMR listener to loopback", async () => {
|
||||
const server = createServer();
|
||||
await listenViteHmrServer(server, 0, "127.0.0.1");
|
||||
|
||||
expect(server.address()).toMatchObject({ address: "127.0.0.1" });
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.close((error) => error ? reject(error) : resolve());
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -90,13 +90,14 @@ describe("boardMutationGuard", () => {
|
|||
expect([200, 204]).toContain(res.status);
|
||||
});
|
||||
|
||||
it("allows board mutations when x-forwarded-host matches origin", async () => {
|
||||
it("allows HTTPS branch-runtime mutations when forwarded MagicDNS host and non-standard port match", async () => {
|
||||
const app = createApp("board");
|
||||
const res = await request(app)
|
||||
.post("/mutate")
|
||||
.set("Host", "127.0.0.1")
|
||||
.set("X-Forwarded-Host", "10.90.10.20:3443")
|
||||
.set("Origin", "https://10.90.10.20:3443")
|
||||
.set("X-Forwarded-Host", "branch-runner.tail123.ts.net:42000")
|
||||
.set("X-Forwarded-Proto", "https")
|
||||
.set("Origin", "https://branch-runner.tail123.ts.net:42000")
|
||||
.send({ ok: true });
|
||||
expect([200, 204]).toContain(res.status);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -0,0 +1,286 @@
|
|||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const mockExecutionWorkspaceService = vi.hoisted(() => ({
|
||||
getById: vi.fn(),
|
||||
update: vi.fn(),
|
||||
}));
|
||||
const mockWorkspaceOperationService = vi.hoisted(() => ({
|
||||
assertRuntimeControlAvailable: vi.fn(),
|
||||
createRecorder: vi.fn(),
|
||||
reconcileStaleRuntimeControlOperations: vi.fn(),
|
||||
}));
|
||||
const mockAccessService = vi.hoisted(() => ({ decide: vi.fn() }));
|
||||
const mockEnvironmentService = vi.hoisted(() => ({ getById: vi.fn() }));
|
||||
const mockSecretService = vi.hoisted(() => ({ normalizeEnvBindingsForPersistence: vi.fn() }));
|
||||
const mockProjectService = vi.hoisted(() => ({ getById: vi.fn() }));
|
||||
const mockHeartbeatService = vi.hoisted(() => ({}));
|
||||
const mockLogActivity = vi.hoisted(() => vi.fn());
|
||||
const mockGetTelemetryClient = vi.hoisted(() => vi.fn());
|
||||
const mockAssertCanManageExecutionWorkspaceRuntimeServices = vi.hoisted(() => vi.fn());
|
||||
const mockAssertCanManageProjectWorkspaceRuntimeServices = vi.hoisted(() => vi.fn());
|
||||
const mockStartRuntimeServices = vi.hoisted(() => vi.fn());
|
||||
const mockStopRuntimeServicesForExecutionWorkspace = vi.hoisted(() => vi.fn());
|
||||
const mockEnsurePersistedExecutionWorkspaceAvailable = vi.hoisted(() => vi.fn());
|
||||
const mockBuildWorkspaceRuntimeDesiredStatePatch = vi.hoisted(() => vi.fn());
|
||||
// The integrated control path also takes the durable runtime-control lease (PAP-17205). This
|
||||
// suite covers the in-flight guard and failed-start reconciliation, so the lease always grants;
|
||||
// `execution-workspace-runtime-lease-route.test.ts` exercises the lease itself against a real db.
|
||||
const mockClaimRuntimeLease = vi.hoisted(() => vi.fn(async () => null));
|
||||
const mockReleaseRuntimeLease = vi.hoisted(() => vi.fn(async () => undefined));
|
||||
|
||||
vi.mock("../telemetry.js", () => ({ getTelemetryClient: mockGetTelemetryClient }));
|
||||
|
||||
vi.mock("../services/index.js", () => ({
|
||||
accessService: () => mockAccessService,
|
||||
environmentService: () => mockEnvironmentService,
|
||||
executionWorkspaceService: () => mockExecutionWorkspaceService,
|
||||
heartbeatService: () => mockHeartbeatService,
|
||||
logActivity: mockLogActivity,
|
||||
projectService: () => mockProjectService,
|
||||
secretService: () => mockSecretService,
|
||||
workspaceOperationService: () => mockWorkspaceOperationService,
|
||||
workspaceRuntimeLeaseService: () => ({
|
||||
claim: mockClaimRuntimeLease,
|
||||
release: mockReleaseRuntimeLease,
|
||||
}),
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS: ["start", "stop", "restart"],
|
||||
}));
|
||||
|
||||
vi.mock("../services/workspace-runtime.js", () => ({
|
||||
buildWorkspaceRuntimeDesiredStatePatch: mockBuildWorkspaceRuntimeDesiredStatePatch,
|
||||
cleanupExecutionWorkspaceArtifacts: vi.fn(),
|
||||
ensurePersistedExecutionWorkspaceAvailable: mockEnsurePersistedExecutionWorkspaceAvailable,
|
||||
listConfiguredRuntimeServiceEntries: vi.fn(() => []),
|
||||
runWorkspaceJobForControl: vi.fn(),
|
||||
startRuntimeServicesForWorkspaceControl: mockStartRuntimeServices,
|
||||
stopRuntimeServicesForExecutionWorkspace: mockStopRuntimeServicesForExecutionWorkspace,
|
||||
stopRuntimeServicesForProjectWorkspace: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("../routes/workspace-runtime-service-authz.js", () => ({
|
||||
assertCanManageExecutionWorkspaceRuntimeServices: mockAssertCanManageExecutionWorkspaceRuntimeServices,
|
||||
assertCanManageProjectWorkspaceRuntimeServices: mockAssertCanManageProjectWorkspaceRuntimeServices,
|
||||
}));
|
||||
|
||||
const executionWorkspaceId = "33333333-3333-4333-8333-333333333333";
|
||||
|
||||
function buildExecutionWorkspace(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id: executionWorkspaceId,
|
||||
companyId: "company-1",
|
||||
// Left unset so the handler does not need a real `db` for project/policy lookups; the
|
||||
// runtime config below is what the control path actually reads.
|
||||
projectId: null,
|
||||
projectWorkspaceId: null,
|
||||
sourceIssueId: null,
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Lane A",
|
||||
status: "active",
|
||||
cwd: "/tmp/lane-a",
|
||||
repoUrl: null,
|
||||
baseRef: "main",
|
||||
branchName: "canary/lane-a",
|
||||
providerType: "git_worktree",
|
||||
providerRef: null,
|
||||
derivedFromExecutionWorkspaceId: null,
|
||||
lastUsedAt: new Date(),
|
||||
openedAt: new Date(),
|
||||
closedAt: null,
|
||||
cleanupEligibleAt: null,
|
||||
cleanupReason: null,
|
||||
config: {
|
||||
workspaceRuntime: {
|
||||
services: [{ name: "app", command: "pnpm dev", port: { type: "fixed", value: 42003 } }],
|
||||
},
|
||||
desiredState: "running",
|
||||
},
|
||||
metadata: null,
|
||||
runtimeServices: [],
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
async function createApp() {
|
||||
const [{ executionWorkspaceRoutes }, { errorHandler }] = await Promise.all([
|
||||
import("../routes/execution-workspaces.js"),
|
||||
import("../middleware/index.js"),
|
||||
]);
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
(req as any).actor = {
|
||||
type: "agent",
|
||||
agentId: "agent-1",
|
||||
companyId: "company-1",
|
||||
source: "agent_key",
|
||||
runId: "run-1",
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", executionWorkspaceRoutes({} as any));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
/**
|
||||
* Route-level guarantees for PAP-17249: the competing lane in the PAP-17207 report must get a
|
||||
* stable 409 while a control is genuinely live, and a start that fails must leave the workspace
|
||||
* stopped and retryable instead of "desired running" with residue.
|
||||
*/
|
||||
describe.sequential("execution workspace runtime control conflict and failure reconciliation", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockAccessService.decide.mockResolvedValue({
|
||||
allowed: true,
|
||||
action: "runtime:manage",
|
||||
reason: "allow_test",
|
||||
explanation: "Allowed by test mock.",
|
||||
});
|
||||
mockExecutionWorkspaceService.getById.mockResolvedValue(buildExecutionWorkspace());
|
||||
mockExecutionWorkspaceService.update.mockResolvedValue(buildExecutionWorkspace());
|
||||
mockAssertCanManageExecutionWorkspaceRuntimeServices.mockResolvedValue(undefined);
|
||||
mockWorkspaceOperationService.assertRuntimeControlAvailable.mockResolvedValue(undefined);
|
||||
mockBuildWorkspaceRuntimeDesiredStatePatch.mockReturnValue({
|
||||
desiredState: "stopped",
|
||||
serviceStates: { app: "stopped" },
|
||||
});
|
||||
mockEnsurePersistedExecutionWorkspaceAvailable.mockResolvedValue({
|
||||
cwd: "/tmp/lane-a",
|
||||
projectId: "project-1",
|
||||
workspaceId: null,
|
||||
branchName: "canary/lane-a",
|
||||
worktreePath: "/tmp/lane-a",
|
||||
repoUrl: null,
|
||||
repoRef: "main",
|
||||
});
|
||||
mockStopRuntimeServicesForExecutionWorkspace.mockResolvedValue(undefined);
|
||||
mockWorkspaceOperationService.createRecorder.mockReturnValue({
|
||||
attachExecutionWorkspaceId: vi.fn(),
|
||||
recordOperation: async (input: any) => {
|
||||
// Mirror the real recorder: a throwing `run` becomes a terminal failed operation and
|
||||
// the error still propagates to the caller.
|
||||
try {
|
||||
await input.run();
|
||||
} catch (error) {
|
||||
(error as any).recordedOperationStatus = "failed";
|
||||
throw error;
|
||||
}
|
||||
return { id: "operation-1", status: "succeeded" };
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("returns a stable 409 while a managed control is genuinely live", async () => {
|
||||
const { conflict } = await import("../errors.js");
|
||||
mockWorkspaceOperationService.assertRuntimeControlAvailable.mockRejectedValue(
|
||||
conflict("A managed runtime control operation is already in progress for this execution workspace.", {
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
executionWorkspaceId,
|
||||
activeAction: "start",
|
||||
requestedAction: "stop",
|
||||
activeOperationId: "operation-live",
|
||||
remediation: "Wait for the active operation to reach a terminal state before retrying.",
|
||||
}),
|
||||
);
|
||||
const app = await createApp();
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/stop`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.details ?? res.body).toMatchObject({
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
activeAction: "start",
|
||||
requestedAction: "stop",
|
||||
});
|
||||
// The conflict is decided before any runtime mutation happens.
|
||||
expect(mockStopRuntimeServicesForExecutionWorkspace).not.toHaveBeenCalled();
|
||||
expect(mockStartRuntimeServices).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("checks authorization before recovering or claiming the workspace", async () => {
|
||||
const { forbidden } = await import("../errors.js");
|
||||
mockAssertCanManageExecutionWorkspaceRuntimeServices.mockRejectedValue(
|
||||
forbidden("Missing permission to manage workspace runtime services"),
|
||||
);
|
||||
const app = await createApp();
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(403);
|
||||
expect(mockWorkspaceOperationService.assertRuntimeControlAvailable).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("tears down residue and records a stopped desired state when a start fails", async () => {
|
||||
mockStartRuntimeServices.mockRejectedValue(
|
||||
new Error(
|
||||
'Runtime service "app" could not start because port 42003 is already in use by pid 4242 (cwd: /tmp/lane-b)',
|
||||
),
|
||||
);
|
||||
const app = await createApp();
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBeGreaterThanOrEqual(400);
|
||||
// No exposure/listener residue: the failed lane is stopped through the ordinary teardown.
|
||||
expect(mockStopRuntimeServicesForExecutionWorkspace).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ executionWorkspaceId, workspaceCwd: "/tmp/lane-a" }),
|
||||
);
|
||||
// ...and the workspace is no longer recorded as wanting to run, so it is retryable and a
|
||||
// startup reconcile will not resurrect a lane that never came up.
|
||||
expect(mockBuildWorkspaceRuntimeDesiredStatePatch).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ action: "stop" }),
|
||||
);
|
||||
expect(mockExecutionWorkspaceService.update).toHaveBeenCalledWith(
|
||||
executionWorkspaceId,
|
||||
expect.objectContaining({ metadata: expect.anything() }),
|
||||
);
|
||||
});
|
||||
|
||||
it("reconciles once when the operation's own time budget fails a start that never settles", async () => {
|
||||
const { WorkspaceOperationTimeoutError } = await import("../services/workspace-operations.js");
|
||||
// The recorder's ceiling fires outside `run`, so only the outer handler can reconcile.
|
||||
mockWorkspaceOperationService.createRecorder.mockReturnValue({
|
||||
attachExecutionWorkspaceId: vi.fn(),
|
||||
recordOperation: async () => {
|
||||
throw new WorkspaceOperationTimeoutError(1_000, "start");
|
||||
},
|
||||
});
|
||||
const app = await createApp();
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBeGreaterThanOrEqual(400);
|
||||
expect(mockStopRuntimeServicesForExecutionWorkspace).toHaveBeenCalledTimes(1);
|
||||
expect(mockBuildWorkspaceRuntimeDesiredStatePatch).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ action: "stop" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves a successful start's desired state untouched by the failure path", async () => {
|
||||
mockStartRuntimeServices.mockResolvedValue([{ id: "runtime-1" }]);
|
||||
const app = await createApp();
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBeLessThan(400);
|
||||
expect(mockStopRuntimeServicesForExecutionWorkspace).not.toHaveBeenCalled();
|
||||
expect(mockBuildWorkspaceRuntimeDesiredStatePatch).not.toHaveBeenCalledWith(
|
||||
expect.objectContaining({ action: "stop" }),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,351 @@
|
|||
import { randomUUID } from "node:crypto";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agents,
|
||||
companies,
|
||||
createDb,
|
||||
executionWorkspaceRuntimeLeases,
|
||||
executionWorkspaces,
|
||||
issues,
|
||||
projects,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
const mockExecutionWorkspaceService = vi.hoisted(() => ({
|
||||
getById: vi.fn(),
|
||||
update: vi.fn(),
|
||||
}));
|
||||
const mockAccessService = vi.hoisted(() => ({ decide: vi.fn(async () => ({ allowed: true })) }));
|
||||
const mockRecordOperation = vi.hoisted(() => vi.fn());
|
||||
const mockReconcileStaleRuntimeControlOperations = vi.hoisted(() => vi.fn(async () => ({ reconciled: 0, operationIds: [] })));
|
||||
// The control path recovers stranded operations and then refuses only a genuinely live one
|
||||
// (PAP-17249). This suite is about the durable lease, so availability always resolves.
|
||||
const mockAssertRuntimeControlAvailable = vi.hoisted(() => vi.fn(async () => undefined));
|
||||
const mockWorkspaceOperationService = vi.hoisted(() => ({
|
||||
createRecorder: vi.fn(() => ({
|
||||
attachExecutionWorkspaceId: vi.fn(),
|
||||
recordOperation: mockRecordOperation,
|
||||
})),
|
||||
assertRuntimeControlAvailable: mockAssertRuntimeControlAvailable,
|
||||
reconcileStaleRuntimeControlOperations: mockReconcileStaleRuntimeControlOperations,
|
||||
listForExecutionWorkspace: vi.fn(async () => []),
|
||||
}));
|
||||
const mockStartRuntimeServices = vi.hoisted(() => vi.fn(async () => []));
|
||||
const mockStopRuntimeServices = vi.hoisted(() => vi.fn(async () => undefined));
|
||||
const mockAssertCanManage = vi.hoisted(() => vi.fn());
|
||||
const mockLogActivity = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("../services/index.js", async () => {
|
||||
const leases = await import("../services/workspace-runtime-leases.js");
|
||||
return {
|
||||
accessService: () => mockAccessService,
|
||||
executionWorkspaceService: () => mockExecutionWorkspaceService,
|
||||
heartbeatService: () => ({}),
|
||||
logActivity: mockLogActivity,
|
||||
workspaceOperationService: () => mockWorkspaceOperationService,
|
||||
workspaceRuntimeLeaseService: leases.workspaceRuntimeLeaseService,
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS: leases.LEASED_WORKSPACE_RUNTIME_ACTIONS,
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock("../services/environment-runtime.js", () => ({
|
||||
environmentRuntimeService: () => ({ destroyReusableSandboxLeases: vi.fn() }),
|
||||
}));
|
||||
|
||||
vi.mock("../services/workspace-runtime.js", () => ({
|
||||
buildWorkspaceRuntimeDesiredStatePatch: () => ({ desiredState: "running", serviceStates: null }),
|
||||
cleanupExecutionWorkspaceArtifacts: vi.fn(),
|
||||
ensurePersistedExecutionWorkspaceAvailable: vi.fn(async () => ({ cwd: "/tmp/lease-route-workspace" })),
|
||||
listConfiguredRuntimeServiceEntries: () => [],
|
||||
runWorkspaceJobForControl: vi.fn(),
|
||||
startRuntimeServicesForWorkspaceControl: mockStartRuntimeServices,
|
||||
stopRuntimeServicesForExecutionWorkspace: mockStopRuntimeServices,
|
||||
}));
|
||||
|
||||
vi.mock("../routes/workspace-runtime-service-authz.js", () => ({
|
||||
assertCanManageExecutionWorkspaceRuntimeServices: mockAssertCanManage,
|
||||
}));
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping runtime lease route tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("execution workspace runtime control lease enforcement", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
let companyId = "";
|
||||
let projectId = "";
|
||||
let executionWorkspaceId = "";
|
||||
let agentId = "";
|
||||
let canaryIssueId = "";
|
||||
let competingIssueId = "";
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-runtime-lease-route-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 30_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.clearAllMocks();
|
||||
mockAccessService.decide.mockResolvedValue({ allowed: true } as never);
|
||||
mockReconcileStaleRuntimeControlOperations.mockResolvedValue({ reconciled: 0, operationIds: [] } as never);
|
||||
mockStartRuntimeServices.mockResolvedValue([] as never);
|
||||
// Owner identity is read per-request from a header so concurrent requests cannot
|
||||
// observe each other's authorization mock.
|
||||
mockAssertCanManage.mockImplementation(async (_db: unknown, req: { get: (name: string) => string | undefined }) => ({
|
||||
actorType: "agent",
|
||||
agentId,
|
||||
runId: null,
|
||||
issueId: req.get("x-test-owner-issue") ?? null,
|
||||
}));
|
||||
mockRecordOperation.mockImplementation(async (input: { run: () => Promise<unknown> }) => {
|
||||
await input.run();
|
||||
return { id: randomUUID(), status: "succeeded" };
|
||||
});
|
||||
|
||||
companyId = randomUUID();
|
||||
projectId = randomUUID();
|
||||
executionWorkspaceId = randomUUID();
|
||||
agentId = randomUUID();
|
||||
canaryIssueId = randomUUID();
|
||||
competingIssueId = randomUUID();
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Runtime lease route",
|
||||
issuePrefix: `R${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({ id: projectId, companyId, name: "Lane", status: "in_progress" });
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: executionWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
mode: "shared_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Shared canary workspace",
|
||||
status: "active",
|
||||
cwd: "/tmp/lease-route-workspace",
|
||||
});
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Engineer", role: "engineer" });
|
||||
await db.insert(issues).values([
|
||||
{
|
||||
id: canaryIssueId,
|
||||
companyId,
|
||||
projectId,
|
||||
executionWorkspaceId,
|
||||
title: "HTTPS canary",
|
||||
status: "in_progress",
|
||||
priority: "high",
|
||||
assigneeAgentId: agentId,
|
||||
},
|
||||
{
|
||||
id: competingIssueId,
|
||||
companyId,
|
||||
projectId,
|
||||
executionWorkspaceId,
|
||||
title: "Unrelated shared-workspace work",
|
||||
status: "in_progress",
|
||||
priority: "high",
|
||||
assigneeAgentId: agentId,
|
||||
},
|
||||
]);
|
||||
|
||||
mockExecutionWorkspaceService.getById.mockResolvedValue({
|
||||
id: executionWorkspaceId,
|
||||
companyId,
|
||||
projectId: null,
|
||||
projectWorkspaceId: null,
|
||||
sourceIssueId: null,
|
||||
mode: "shared_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Shared canary workspace",
|
||||
status: "active",
|
||||
cwd: "/tmp/lease-route-workspace",
|
||||
repoUrl: null,
|
||||
baseRef: "main",
|
||||
branchName: null,
|
||||
providerType: "git_worktree",
|
||||
providerRef: null,
|
||||
config: { workspaceRuntime: { command: "pnpm dev" } },
|
||||
metadata: null,
|
||||
runtimeServices: [],
|
||||
} as never);
|
||||
mockExecutionWorkspaceService.update.mockResolvedValue({ id: executionWorkspaceId } as never);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(executionWorkspaceRuntimeLeases);
|
||||
await db.delete(issues);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(projects);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
async function createApp() {
|
||||
const [{ executionWorkspaceRoutes }, { errorHandler }] = await Promise.all([
|
||||
import("../routes/execution-workspaces.js"),
|
||||
import("../middleware/index.js"),
|
||||
]);
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
(req as any).actor = {
|
||||
type: "agent",
|
||||
agentId,
|
||||
companyId,
|
||||
source: "agent_key",
|
||||
runId: null,
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", executionWorkspaceRoutes(db as never));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
it("lets the lease owner start and restart, and blocks a sequential competing issue without mutating anything", async () => {
|
||||
const app = await createApp();
|
||||
|
||||
const started = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.set("x-test-owner-issue", canaryIssueId)
|
||||
.send({});
|
||||
expect(started.status).toBe(200);
|
||||
expect(mockStartRuntimeServices).toHaveBeenCalledTimes(1);
|
||||
|
||||
const restarted = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/restart`)
|
||||
.set("x-test-owner-issue", canaryIssueId)
|
||||
.send({});
|
||||
expect(restarted.status).toBe(200);
|
||||
expect(mockStartRuntimeServices).toHaveBeenCalledTimes(2);
|
||||
|
||||
mockRecordOperation.mockClear();
|
||||
mockStartRuntimeServices.mockClear();
|
||||
mockStopRuntimeServices.mockClear();
|
||||
mockReconcileStaleRuntimeControlOperations.mockClear();
|
||||
|
||||
const blocked = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.set("x-test-owner-issue", competingIssueId)
|
||||
.send({});
|
||||
|
||||
expect(blocked.status).toBe(409);
|
||||
expect(blocked.body.code).toBe("workspace_runtime_lease_conflict");
|
||||
expect(blocked.body.details).toMatchObject({
|
||||
ownerIssueId: canaryIssueId,
|
||||
requestedAction: "start",
|
||||
});
|
||||
expect(typeof blocked.body.remediation).toBe("string");
|
||||
|
||||
// No workspace operation, no runtime-service mutation, no reconciliation side effect.
|
||||
expect(mockRecordOperation).not.toHaveBeenCalled();
|
||||
expect(mockStartRuntimeServices).not.toHaveBeenCalled();
|
||||
expect(mockStopRuntimeServices).not.toHaveBeenCalled();
|
||||
expect(mockReconcileStaleRuntimeControlOperations).not.toHaveBeenCalled();
|
||||
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.ownerIssueId).toBe(canaryIssueId);
|
||||
});
|
||||
|
||||
it("blocks a competing issue that races the owner concurrently", async () => {
|
||||
const app = await createApp();
|
||||
|
||||
const ownerStart = request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.set("x-test-owner-issue", canaryIssueId)
|
||||
.send({});
|
||||
const competingStart = request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/stop`)
|
||||
.set("x-test-owner-issue", competingIssueId)
|
||||
.send({});
|
||||
|
||||
const [ownerRes, competingRes] = await Promise.all([ownerStart, competingStart]);
|
||||
|
||||
expect(ownerRes.status).toBe(200);
|
||||
expect(competingRes.status).toBe(409);
|
||||
expect(competingRes.body.code).toMatch(
|
||||
/workspace_runtime_lease_conflict|workspace_runtime_control_in_progress/,
|
||||
);
|
||||
expect(mockStopRuntimeServices).not.toHaveBeenCalled();
|
||||
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.ownerIssueId).toBe(canaryIssueId);
|
||||
});
|
||||
|
||||
it("lets a teardown issue take the lane once the canary issue is terminal", async () => {
|
||||
const app = await createApp();
|
||||
|
||||
expect((await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.set("x-test-owner-issue", canaryIssueId)
|
||||
.send({})).status).toBe(200);
|
||||
|
||||
const { eq } = await import("drizzle-orm");
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, canaryIssueId));
|
||||
|
||||
const teardown = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/stop`)
|
||||
.set("x-test-owner-issue", competingIssueId)
|
||||
.send({});
|
||||
|
||||
expect(teardown.status).toBe(200);
|
||||
expect(mockStopRuntimeServices).toHaveBeenCalledTimes(1);
|
||||
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.ownerIssueId).toBe(competingIssueId);
|
||||
});
|
||||
|
||||
it("leaves board actors unleased and unblocked", async () => {
|
||||
const [{ executionWorkspaceRoutes }, { errorHandler }] = await Promise.all([
|
||||
import("../routes/execution-workspaces.js"),
|
||||
import("../middleware/index.js"),
|
||||
]);
|
||||
mockAssertCanManage.mockResolvedValue({
|
||||
actorType: "board",
|
||||
agentId: null,
|
||||
runId: null,
|
||||
issueId: null,
|
||||
} as never);
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
(req as any).actor = {
|
||||
type: "board",
|
||||
userId: "board-1",
|
||||
companyIds: [companyId],
|
||||
source: "session",
|
||||
isInstanceAdmin: false,
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", executionWorkspaceRoutes(db as never));
|
||||
app.use(errorHandler);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/execution-workspaces/${executionWorkspaceId}/runtime-services/start`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(await db.select().from(executionWorkspaceRuntimeLeases)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
|
@ -21,6 +21,12 @@ const mockWorkspaceOperationService = vi.hoisted(() => ({
|
|||
createRecorder: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockWorkspaceRuntimeLeaseService = vi.hoisted(() => ({
|
||||
claim: vi.fn(async () => ({ outcome: "created", ownerKey: "issue:issue-1", lease: null, reclaimedFrom: null })),
|
||||
release: vi.fn(async () => ({ released: false, ownerKey: null })),
|
||||
get: vi.fn(async () => null),
|
||||
}));
|
||||
|
||||
const mockHeartbeatService = vi.hoisted(() => ({
|
||||
wakeup: vi.fn(),
|
||||
}));
|
||||
|
|
@ -40,6 +46,8 @@ vi.mock("../services/index.js", () => ({
|
|||
heartbeatService: () => mockHeartbeatService,
|
||||
logActivity: mockLogActivity,
|
||||
workspaceOperationService: () => mockWorkspaceOperationService,
|
||||
workspaceRuntimeLeaseService: () => mockWorkspaceRuntimeLeaseService,
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS: ["start", "stop", "restart"],
|
||||
}));
|
||||
|
||||
vi.mock("../services/environment-runtime.js", () => ({
|
||||
|
|
|
|||
|
|
@ -3948,6 +3948,186 @@ describeEmbeddedPostgres("executionWorkspaceService.getCloseReadiness", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("serializes the verified HTTPS URL as canonical and never falls back to the HTTP backend", async () => {
|
||||
// PAP-17158: the UI's workspace/project/issue launch links read `url` off the
|
||||
// serialized runtime service. Two things have to hold for a managed HTTPS
|
||||
// runtime: once exposure is `ready` the canonical `url` is the HTTPS public
|
||||
// URL, and while exposure is *not* ready the canonical `url` stays null even
|
||||
// though the row still knows its loopback `backendUrl`. Serializing that
|
||||
// backend URL would put `http://…` back into a launch link, which is exactly
|
||||
// the fail-closed contract this feature exists to enforce.
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
const projectWorkspaceId = randomUUID();
|
||||
const readyWorkspaceId = randomUUID();
|
||||
const provisioningWorkspaceId = randomUUID();
|
||||
const readyServiceId = randomUUID();
|
||||
const provisioningServiceId = randomUUID();
|
||||
const hostname = "paperclip-dev.tail29c1aa.ts.net";
|
||||
const httpsUrl = `https://${hostname}:42010`;
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: "PAP",
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "HTTPS URL serialization",
|
||||
status: "in_progress",
|
||||
});
|
||||
await db.insert(projectWorkspaces).values({
|
||||
id: projectWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
name: "Primary",
|
||||
sourceType: "local_path",
|
||||
isPrimary: true,
|
||||
cwd: "/tmp/https-url-serialization",
|
||||
metadata: {
|
||||
runtimeConfig: {
|
||||
workspaceRuntime: { services: [{ name: "paperclip-dev", command: "pnpm dev" }] },
|
||||
desiredState: "running",
|
||||
},
|
||||
},
|
||||
});
|
||||
await db.insert(executionWorkspaces).values([
|
||||
{
|
||||
id: readyWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
projectWorkspaceId,
|
||||
mode: "dedicated_worktree",
|
||||
strategyType: "git_worktree",
|
||||
name: "Exposed workspace",
|
||||
status: "idle",
|
||||
providerType: "local_fs",
|
||||
cwd: "/tmp/https-url-serialization/ready",
|
||||
},
|
||||
{
|
||||
id: provisioningWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
projectWorkspaceId,
|
||||
mode: "dedicated_worktree",
|
||||
strategyType: "git_worktree",
|
||||
name: "Provisioning workspace",
|
||||
status: "idle",
|
||||
providerType: "local_fs",
|
||||
cwd: "/tmp/https-url-serialization/provisioning",
|
||||
},
|
||||
]);
|
||||
await db.insert(workspaceRuntimeServices).values([
|
||||
{
|
||||
id: readyServiceId,
|
||||
companyId,
|
||||
projectId,
|
||||
projectWorkspaceId,
|
||||
executionWorkspaceId: readyWorkspaceId,
|
||||
scopeType: "execution_workspace",
|
||||
scopeId: readyWorkspaceId,
|
||||
serviceName: "paperclip-dev",
|
||||
status: "running",
|
||||
lifecycle: "shared",
|
||||
reuseKey: "ready-dev",
|
||||
command: "pnpm dev",
|
||||
cwd: "/tmp/https-url-serialization/ready",
|
||||
port: 42_010,
|
||||
url: httpsUrl,
|
||||
// The loopback backend is still recorded; it must never be serialized.
|
||||
backendUrl: "http://127.0.0.1:42010",
|
||||
provider: "local_process",
|
||||
healthStatus: "healthy",
|
||||
exposure: {
|
||||
provider: "tailscale_https",
|
||||
state: "ready",
|
||||
publicUrl: httpsUrl,
|
||||
hostname,
|
||||
listeners: [
|
||||
{ purpose: "app", publicPort: 42_010, targetPort: 42_010 },
|
||||
{ purpose: "vite_hmr", publicPort: 52_010, targetPort: 52_010 },
|
||||
],
|
||||
brokerRef: "broker-ref-1",
|
||||
lastError: null,
|
||||
updatedAt: "2026-08-12T10:00:00.000Z",
|
||||
},
|
||||
updatedAt: new Date("2026-08-12T10:00:00.000Z"),
|
||||
},
|
||||
{
|
||||
id: provisioningServiceId,
|
||||
companyId,
|
||||
projectId,
|
||||
projectWorkspaceId,
|
||||
executionWorkspaceId: provisioningWorkspaceId,
|
||||
scopeType: "execution_workspace",
|
||||
scopeId: provisioningWorkspaceId,
|
||||
serviceName: "paperclip-dev",
|
||||
status: "running",
|
||||
lifecycle: "shared",
|
||||
reuseKey: "provisioning-dev",
|
||||
command: "pnpm dev",
|
||||
cwd: "/tmp/https-url-serialization/provisioning",
|
||||
port: 42_020,
|
||||
url: null,
|
||||
backendUrl: "http://127.0.0.1:42020",
|
||||
provider: "local_process",
|
||||
healthStatus: "healthy",
|
||||
exposure: {
|
||||
provider: "tailscale_https",
|
||||
state: "pending",
|
||||
publicUrl: null,
|
||||
hostname,
|
||||
listeners: [],
|
||||
brokerRef: null,
|
||||
lastError: null,
|
||||
updatedAt: "2026-08-12T10:00:00.000Z",
|
||||
},
|
||||
updatedAt: new Date("2026-08-12T10:00:00.000Z"),
|
||||
},
|
||||
]);
|
||||
|
||||
const workspaces = await svc.list(companyId);
|
||||
const readyService = workspaces
|
||||
.find((workspace) => workspace.id === readyWorkspaceId)
|
||||
?.runtimeServices.find((service) => service.id === readyServiceId);
|
||||
expect(readyService).toMatchObject({
|
||||
url: httpsUrl,
|
||||
port: 42_010,
|
||||
exposure: { provider: "tailscale_https", state: "ready", publicUrl: httpsUrl, hostname },
|
||||
});
|
||||
// Lease handles are server-private and must never reach a serialized DTO.
|
||||
expect(readyService).not.toHaveProperty("exposureHandle");
|
||||
|
||||
const provisioningService = workspaces
|
||||
.find((workspace) => workspace.id === provisioningWorkspaceId)
|
||||
?.runtimeServices.find((service) => service.id === provisioningServiceId);
|
||||
expect(provisioningService?.url).toBeNull();
|
||||
expect(provisioningService?.exposure).toMatchObject({ state: "pending", publicUrl: null });
|
||||
expect(JSON.stringify(provisioningService)).not.toContain("http://127.0.0.1:42020");
|
||||
|
||||
// The overview feeds the workspace list launch links.
|
||||
const overview = await svc.listOverview(companyId, { limit: 10, offset: 0 });
|
||||
const readyItem = overview.items.find((item) => item.workspaceId === readyWorkspaceId);
|
||||
expect(readyItem?.primaryService).toMatchObject({
|
||||
id: readyServiceId,
|
||||
status: "running",
|
||||
url: httpsUrl,
|
||||
exposure: { state: "ready", publicUrl: httpsUrl },
|
||||
});
|
||||
expect(new URL(readyItem!.primaryService!.url!).protocol).toBe("https:");
|
||||
|
||||
const provisioningItem = overview.items.find((item) => item.workspaceId === provisioningWorkspaceId);
|
||||
expect(provisioningItem?.primaryService).toMatchObject({
|
||||
id: provisioningServiceId,
|
||||
status: "running",
|
||||
url: null,
|
||||
exposure: { state: "pending" },
|
||||
});
|
||||
expect(JSON.stringify(provisioningItem)).not.toContain("http://127.0.0.1:42020");
|
||||
}, 30_000);
|
||||
|
||||
it("returns a bounded company-scoped workspace overview with service and linked issue summaries", async () => {
|
||||
const companyId = randomUUID();
|
||||
const otherCompanyId = randomUUID();
|
||||
|
|
|
|||
|
|
@ -30,6 +30,14 @@ if (!process.env.CODEX_HOME) {
|
|||
process.env.CODEX_HOME = codexHome;
|
||||
}
|
||||
|
||||
// The automatic Tailscale HTTPS default (PAP-17158) probes for a real host
|
||||
// broker socket, so leaving it enabled would make every test that starts a
|
||||
// service named `paperclip-dev` behave differently on a broker-capable host
|
||||
// than on CI. Tests that exercise the default opt in explicitly.
|
||||
if (!process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS) {
|
||||
process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS = "off";
|
||||
}
|
||||
|
||||
if (!SupertestTest.prototype.__paperclipLoopbackPatched) {
|
||||
SupertestTest.prototype.serverAddress = function serverAddress(app, path) {
|
||||
const addr = app.address();
|
||||
|
|
|
|||
|
|
@ -0,0 +1,207 @@
|
|||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
companies,
|
||||
createDb,
|
||||
executionWorkspaces,
|
||||
projects,
|
||||
workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { getWorkspaceOperationLogStore } from "../services/workspace-operation-log-store.js";
|
||||
import {
|
||||
RUNTIME_CONTROL_STALE_AFTER_MS,
|
||||
resetWorkspaceRuntimeControlLocksForTests,
|
||||
runExclusiveWorkspaceRuntimeControl,
|
||||
workspaceOperationService,
|
||||
} from "../services/workspace-operations.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
describe("workspace runtime control serialization", () => {
|
||||
afterEach(() => {
|
||||
resetWorkspaceRuntimeControlLocksForTests();
|
||||
});
|
||||
|
||||
it("deterministically rejects an overlapping control operation and releases the claim", async () => {
|
||||
let releaseFirst!: () => void;
|
||||
const firstGate = new Promise<void>((resolve) => {
|
||||
releaseFirst = resolve;
|
||||
});
|
||||
const first = runExclusiveWorkspaceRuntimeControl({
|
||||
executionWorkspaceId: "workspace-1",
|
||||
action: "start",
|
||||
run: async () => {
|
||||
await firstGate;
|
||||
return "started";
|
||||
},
|
||||
});
|
||||
|
||||
await expect(runExclusiveWorkspaceRuntimeControl({
|
||||
executionWorkspaceId: "workspace-1",
|
||||
action: "restart",
|
||||
run: async () => "restarted",
|
||||
})).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: {
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
activeAction: "start",
|
||||
requestedAction: "restart",
|
||||
},
|
||||
});
|
||||
|
||||
releaseFirst();
|
||||
await expect(first).resolves.toBe("started");
|
||||
await expect(runExclusiveWorkspaceRuntimeControl({
|
||||
executionWorkspaceId: "workspace-1",
|
||||
action: "stop",
|
||||
run: async () => "stopped",
|
||||
})).resolves.toBe("stopped");
|
||||
});
|
||||
});
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping workspace-operation reconciliation tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("workspace operation reconciliation", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
let logRoot = "";
|
||||
let previousLogRoot: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-operation-logs-"));
|
||||
previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot;
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(workspaceOperations);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(projects);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot;
|
||||
await fs.rm(logRoot, { recursive: true, force: true });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it("terminalizes an orphaned runtime start with an inspectable reconciliation log", async () => {
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
const executionWorkspaceId = randomUUID();
|
||||
const staleOperationId = randomUUID();
|
||||
const unrelatedOperationId = randomUUID();
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Workspace operation reconciliation",
|
||||
issuePrefix: `W${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "Runtime reconciliation",
|
||||
status: "in_progress",
|
||||
});
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: executionWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Reconciliation workspace",
|
||||
status: "active",
|
||||
cwd: "/tmp/reconciliation-workspace",
|
||||
});
|
||||
|
||||
const logHandle = await getWorkspaceOperationLogStore().begin({
|
||||
companyId,
|
||||
operationId: staleOperationId,
|
||||
});
|
||||
await db.insert(workspaceOperations).values([
|
||||
{
|
||||
id: staleOperationId,
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
cwd: "/tmp/reconciliation-workspace",
|
||||
status: "running",
|
||||
logStore: logHandle.store,
|
||||
logRef: logHandle.logRef,
|
||||
metadata: { action: "start", executionWorkspaceId },
|
||||
},
|
||||
{
|
||||
id: unrelatedOperationId,
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
phase: "workspace_provision",
|
||||
command: "bash ./scripts/provision-worktree.sh",
|
||||
cwd: "/tmp/reconciliation-workspace",
|
||||
status: "running",
|
||||
metadata: { created: true },
|
||||
},
|
||||
]);
|
||||
|
||||
const service = workspaceOperationService(db);
|
||||
|
||||
// Recovery is bounded and cannot steal a live operation (PAP-17249): a `running` control
|
||||
// that has signalled recently is left alone even when it carries no ownership stamp, which
|
||||
// is the shape a control recorded by a pre-stamp build has.
|
||||
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
||||
reconciled: 0,
|
||||
operationIds: [],
|
||||
});
|
||||
expect((await service.getById(staleOperationId))?.status).toBe("running");
|
||||
|
||||
// Once it has gone quiet for longer than the staleness window, it is terminalized.
|
||||
await expect(
|
||||
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId, {
|
||||
now: new Date(Date.now() + RUNTIME_CONTROL_STALE_AFTER_MS + 5_000),
|
||||
}),
|
||||
).resolves.toEqual({
|
||||
reconciled: 1,
|
||||
operationIds: [staleOperationId],
|
||||
});
|
||||
|
||||
const stale = await service.getById(staleOperationId);
|
||||
expect(stale).toMatchObject({
|
||||
status: "failed",
|
||||
metadata: {
|
||||
action: "start",
|
||||
reconciled: true,
|
||||
reconciliationReason: "orphaned_runtime_control",
|
||||
},
|
||||
});
|
||||
expect(stale?.finishedAt).toBeInstanceOf(Date);
|
||||
expect(stale?.stderrExcerpt).toContain("no live owner has reported progress");
|
||||
expect((await service.readLog(staleOperationId)).content).toContain(
|
||||
"no live owner has reported progress",
|
||||
);
|
||||
|
||||
const unrelated = await db
|
||||
.select({ status: workspaceOperations.status })
|
||||
.from(workspaceOperations)
|
||||
.where(eq(workspaceOperations.id, unrelatedOperationId))
|
||||
.then((rows) => rows[0]);
|
||||
expect(unrelated?.status).toBe("running");
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,519 @@
|
|||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
companies,
|
||||
createDb,
|
||||
executionWorkspaces,
|
||||
projects,
|
||||
workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import {
|
||||
RUNTIME_CONTROL_STALE_AFTER_MS,
|
||||
WorkspaceOperationTimeoutError,
|
||||
resetWorkspaceRuntimeControlStateForTests,
|
||||
workspaceOperationService,
|
||||
workspaceRuntimeControlOwnerIdForTests,
|
||||
} from "../services/workspace-operations.js";
|
||||
import { getWorkspaceOperationLogStore } from "../services/workspace-operation-log-store.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping runtime-control recovery tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Regression coverage for PAP-17249. The adverse report on PAP-17207 left an execution
|
||||
* workspace with a `running` managed start that no owner was driving, so every managed
|
||||
* cleanup was rejected with `409 workspace_runtime_control_in_progress` forever.
|
||||
*
|
||||
* These cases run against real Postgres because the recovery guarantee is a
|
||||
* compare-and-swap over `workspace_operations`, not an in-process lock.
|
||||
*/
|
||||
describeEmbeddedPostgres("managed runtime-control operation recovery", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let stopDb: (() => Promise<void>) | null = null;
|
||||
let logRoot = "";
|
||||
let previousLogRoot: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
const started = await startEmbeddedPostgresTestDatabase("paperclip-runtime-control-recovery-");
|
||||
stopDb = started.cleanup;
|
||||
db = createDb(started.connectionString);
|
||||
logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-control-logs-"));
|
||||
previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot;
|
||||
}, 60_000);
|
||||
|
||||
afterEach(async () => {
|
||||
resetWorkspaceRuntimeControlStateForTests();
|
||||
await db.delete(workspaceOperations);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(projects);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot;
|
||||
await fs.rm(logRoot, { recursive: true, force: true });
|
||||
if (stopDb) await stopDb();
|
||||
});
|
||||
|
||||
async function seedWorkspace() {
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
const executionWorkspaceId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Runtime control recovery",
|
||||
issuePrefix: `R${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "Runtime control recovery",
|
||||
status: "in_progress",
|
||||
});
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: executionWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Lane A",
|
||||
status: "active",
|
||||
cwd: "/tmp/runtime-control-lane-a",
|
||||
});
|
||||
return { companyId, projectId, executionWorkspaceId };
|
||||
}
|
||||
|
||||
/** Wait for a managed control to have actually persisted its claim row. */
|
||||
async function waitForRunningOperation(executionWorkspaceId: string) {
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
const row = await db
|
||||
.select({ id: workspaceOperations.id })
|
||||
.from(workspaceOperations)
|
||||
.where(
|
||||
and(
|
||||
eq(workspaceOperations.executionWorkspaceId, executionWorkspaceId),
|
||||
eq(workspaceOperations.status, "running"),
|
||||
),
|
||||
)
|
||||
.then((rows) => rows[0]);
|
||||
if (row) return row.id;
|
||||
await delay(20);
|
||||
}
|
||||
throw new Error("timed out waiting for the managed control operation row");
|
||||
}
|
||||
|
||||
/** Insert a `running` start the way an abandoned request or dead server process leaves one. */
|
||||
async function seedRunningStart(input: {
|
||||
companyId: string;
|
||||
executionWorkspaceId: string;
|
||||
owner?: { ownerId: string; pid: number; heartbeatAt: Date };
|
||||
withLog?: boolean;
|
||||
startedAt?: Date;
|
||||
/**
|
||||
* Let Postgres stamp `started_at`/`updated_at` from the column defaults, at the
|
||||
* microsecond precision the driver truncates on the way back into JS. This is how every
|
||||
* row written before managed controls stamped their own timestamps looks.
|
||||
*/
|
||||
columnDefaultTimestamps?: boolean;
|
||||
}) {
|
||||
const id = randomUUID();
|
||||
const handle = input.withLog
|
||||
? await getWorkspaceOperationLogStore().begin({ companyId: input.companyId, operationId: id })
|
||||
: null;
|
||||
const startedAt = input.startedAt ?? new Date();
|
||||
await db.insert(workspaceOperations).values({
|
||||
id,
|
||||
companyId: input.companyId,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
cwd: "/tmp/runtime-control-lane-a",
|
||||
status: "running",
|
||||
logStore: handle?.store ?? null,
|
||||
logRef: handle?.logRef ?? null,
|
||||
metadata: {
|
||||
action: "start",
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
...(input.owner
|
||||
? {
|
||||
runtimeControlOwner: {
|
||||
ownerId: input.owner.ownerId,
|
||||
pid: input.owner.pid,
|
||||
action: "start",
|
||||
heartbeatAt: input.owner.heartbeatAt.toISOString(),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
...(input.columnDefaultTimestamps ? {} : { startedAt, updatedAt: startedAt }),
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
it("recovers a start abandoned by a dead server process and frees managed control", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
// pid 2^22-1 is above every Linux/macOS pid ceiling, so it can never be alive.
|
||||
const deadPid = 4_194_303;
|
||||
const staleId = await seedRunningStart({
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
owner: { ownerId: randomUUID(), pid: deadPid, heartbeatAt: new Date() },
|
||||
withLog: true,
|
||||
});
|
||||
|
||||
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
||||
reconciled: 1,
|
||||
operationIds: [staleId],
|
||||
});
|
||||
|
||||
const recovered = await service.getById(staleId);
|
||||
expect(recovered).toMatchObject({
|
||||
status: "failed",
|
||||
metadata: {
|
||||
action: "start",
|
||||
reconciled: true,
|
||||
reconciliationReason: "orphaned_runtime_control",
|
||||
},
|
||||
});
|
||||
expect(recovered?.finishedAt).toBeInstanceOf(Date);
|
||||
expect(recovered?.stderrExcerpt).toContain(`the owning server process (pid ${deadPid}) is gone`);
|
||||
expect((await service.readLog(staleId)).content).toContain("is gone");
|
||||
|
||||
// The workspace is controllable again through the ordinary managed path.
|
||||
await expect(
|
||||
service.assertRuntimeControlAvailable({ executionWorkspaceId, action: "stop" }),
|
||||
).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("refuses to steal a start whose owning process is still alive inside the staleness window", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const liveId = await seedRunningStart({
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
// A different server instance, but its pid is this very test process: genuinely alive.
|
||||
owner: { ownerId: randomUUID(), pid: process.pid, heartbeatAt: new Date() },
|
||||
});
|
||||
|
||||
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
||||
reconciled: 0,
|
||||
operationIds: [],
|
||||
});
|
||||
expect((await service.getById(liveId))?.status).toBe("running");
|
||||
|
||||
// ...and a competing control gets the stable conflict, not a silent steal.
|
||||
await expect(
|
||||
service.assertRuntimeControlAvailable({ executionWorkspaceId, action: "start" }),
|
||||
).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: {
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
activeAction: "start",
|
||||
requestedAction: "start",
|
||||
activeOperationId: liveId,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("bounds recovery: the same live owner is reclaimed once its heartbeat lapses", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const staleId = await seedRunningStart({
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
owner: { ownerId: randomUUID(), pid: process.pid, heartbeatAt: new Date() },
|
||||
});
|
||||
|
||||
const afterWindow = new Date(Date.now() + RUNTIME_CONTROL_STALE_AFTER_MS + 1_000);
|
||||
await expect(
|
||||
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId, { now: afterWindow }),
|
||||
).resolves.toEqual({ reconciled: 1, operationIds: [staleId] });
|
||||
expect((await service.getById(staleId))?.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("recovers a legacy start with no ownership stamp only after the staleness window", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const legacyId = await seedRunningStart({ companyId, executionWorkspaceId });
|
||||
|
||||
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
||||
reconciled: 0,
|
||||
operationIds: [],
|
||||
});
|
||||
await expect(
|
||||
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId, {
|
||||
now: new Date(Date.now() + RUNTIME_CONTROL_STALE_AFTER_MS + 1_000),
|
||||
}),
|
||||
).resolves.toEqual({ reconciled: 1, operationIds: [legacyId] });
|
||||
});
|
||||
|
||||
it("recovers a start whose timestamps came from the column defaults", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
// Regression guard for the compare-and-swap: `updated_at` written by `defaultNow()` keeps
|
||||
// microsecond precision, while the driver hands JS a millisecond-truncated Date. An `=`
|
||||
// CAS against that read never matches, which would make recovery silently no-op on exactly
|
||||
// the rows it exists to clear — the ones a previous build left `running`.
|
||||
const legacyId = await seedRunningStart({
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
owner: { ownerId: randomUUID(), pid: 4_194_303, heartbeatAt: new Date() },
|
||||
columnDefaultTimestamps: true,
|
||||
withLog: true,
|
||||
});
|
||||
|
||||
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
||||
reconciled: 1,
|
||||
operationIds: [legacyId],
|
||||
});
|
||||
expect((await service.getById(legacyId))?.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("terminalizes exactly once when two recovery sweeps race the same operation", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const staleId = await seedRunningStart({
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
owner: { ownerId: randomUUID(), pid: 4_194_303, heartbeatAt: new Date() },
|
||||
});
|
||||
|
||||
const [a, b] = await Promise.all([
|
||||
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId),
|
||||
service.reconcileStaleRuntimeControlOperations(executionWorkspaceId),
|
||||
]);
|
||||
expect(a.reconciled + b.reconciled).toBe(1);
|
||||
expect((await service.getById(staleId))?.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("leaves non-runtime-control operations alone", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const provisionId = randomUUID();
|
||||
await db.insert(workspaceOperations).values({
|
||||
id: provisionId,
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
phase: "workspace_provision",
|
||||
command: "bash ./scripts/provision-worktree.sh",
|
||||
cwd: "/tmp/runtime-control-lane-a",
|
||||
status: "running",
|
||||
metadata: { created: true },
|
||||
startedAt: new Date(Date.now() - 60 * 60_000),
|
||||
updatedAt: new Date(Date.now() - 60 * 60_000),
|
||||
});
|
||||
|
||||
await expect(service.reconcileStaleRuntimeControlOperations()).resolves.toEqual({
|
||||
reconciled: 0,
|
||||
operationIds: [],
|
||||
});
|
||||
const row = await db
|
||||
.select({ status: workspaceOperations.status })
|
||||
.from(workspaceOperations)
|
||||
.where(eq(workspaceOperations.id, provisionId))
|
||||
.then((rows) => rows[0]);
|
||||
expect(row?.status).toBe("running");
|
||||
});
|
||||
|
||||
it("keeps a company's recovery inside its own scope", async () => {
|
||||
const laneA = await seedWorkspace();
|
||||
const laneB = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const staleA = await seedRunningStart({
|
||||
companyId: laneA.companyId,
|
||||
executionWorkspaceId: laneA.executionWorkspaceId,
|
||||
owner: { ownerId: randomUUID(), pid: 4_194_303, heartbeatAt: new Date() },
|
||||
});
|
||||
const staleB = await seedRunningStart({
|
||||
companyId: laneB.companyId,
|
||||
executionWorkspaceId: laneB.executionWorkspaceId,
|
||||
owner: { ownerId: randomUUID(), pid: 4_194_303, heartbeatAt: new Date() },
|
||||
});
|
||||
|
||||
await expect(
|
||||
service.reconcileStaleRuntimeControlOperations(laneA.executionWorkspaceId),
|
||||
).resolves.toEqual({ reconciled: 1, operationIds: [staleA] });
|
||||
expect((await service.getById(staleB))?.status).toBe("running");
|
||||
});
|
||||
|
||||
it("holds the workspace while a start is genuinely live in this process, then releases it", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const recorder = service.createRecorder({ companyId, executionWorkspaceId });
|
||||
|
||||
let releaseStart!: () => void;
|
||||
const startGate = new Promise<void>((resolve) => {
|
||||
releaseStart = resolve;
|
||||
});
|
||||
const inFlight = recorder.recordOperation({
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
metadata: { action: "start", executionWorkspaceId },
|
||||
run: async () => {
|
||||
await startGate;
|
||||
return { status: "succeeded" as const, exitCode: 0 };
|
||||
},
|
||||
});
|
||||
|
||||
// The claim exists once the operation row lands, which is the point at which a competing
|
||||
// control could observe it.
|
||||
const liveOperationId = await waitForRunningOperation(executionWorkspaceId);
|
||||
|
||||
// The live claim is visible to another service instance on the same database, and a
|
||||
// recovery sweep must not steal it even though this operation has not heartbeated yet.
|
||||
const competitor = workspaceOperationService(db);
|
||||
await expect(
|
||||
competitor.assertRuntimeControlAvailable({ executionWorkspaceId, action: "restart" }),
|
||||
).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: {
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
requestedAction: "restart",
|
||||
activeOperationId: liveOperationId,
|
||||
},
|
||||
});
|
||||
await expect(
|
||||
competitor.reconcileStaleRuntimeControlOperations(executionWorkspaceId),
|
||||
).resolves.toEqual({ reconciled: 0, operationIds: [] });
|
||||
|
||||
releaseStart();
|
||||
const completed = await inFlight;
|
||||
expect(completed.status).toBe("succeeded");
|
||||
expect(completed.metadata).toMatchObject({
|
||||
runtimeControlOwner: { ownerId: workspaceRuntimeControlOwnerIdForTests() },
|
||||
});
|
||||
|
||||
// Terminal operation, so the next managed control proceeds.
|
||||
await expect(
|
||||
competitor.assertRuntimeControlAvailable({ executionWorkspaceId, action: "stop" }),
|
||||
).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("protects an in-process start from recovery even with the staleness window collapsed", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const recorder = service.createRecorder({ companyId, executionWorkspaceId });
|
||||
const sweeper = workspaceOperationService(db);
|
||||
|
||||
let sweeping = true;
|
||||
let reconciled = 0;
|
||||
// `staleAfterMs: 0` removes the heartbeat grace period, so the only thing that can keep
|
||||
// this start alive is the in-process ownership claim. Sweeping continuously across the
|
||||
// whole recordOperation lifetime also covers the row-insert window.
|
||||
const sweeps = (async () => {
|
||||
while (sweeping) {
|
||||
const result = await sweeper.reconcileStaleRuntimeControlOperations(executionWorkspaceId, {
|
||||
staleAfterMs: 0,
|
||||
});
|
||||
reconciled += result.reconciled;
|
||||
await delay(2);
|
||||
}
|
||||
})();
|
||||
|
||||
const operation = await recorder.recordOperation({
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
metadata: { action: "start", executionWorkspaceId },
|
||||
run: async () => {
|
||||
await delay(120);
|
||||
return { status: "succeeded" as const, exitCode: 0 };
|
||||
},
|
||||
});
|
||||
sweeping = false;
|
||||
await sweeps;
|
||||
|
||||
expect(reconciled).toBe(0);
|
||||
expect(operation.status).toBe("succeeded");
|
||||
expect((await service.getById(operation.id))?.status).toBe("succeeded");
|
||||
});
|
||||
|
||||
it("fails a hung start on its own time budget so no active operation is left behind", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const recorder = service.createRecorder({ companyId, executionWorkspaceId });
|
||||
|
||||
await expect(
|
||||
recorder.recordOperation({
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
metadata: { action: "start", executionWorkspaceId },
|
||||
timeoutMs: 150,
|
||||
// A start that never settles: the readiness probe that parked on a foreign listener.
|
||||
run: () => new Promise(() => {}),
|
||||
}),
|
||||
).rejects.toBeInstanceOf(WorkspaceOperationTimeoutError);
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(workspaceOperations)
|
||||
.where(eq(workspaceOperations.executionWorkspaceId, executionWorkspaceId));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ status: "failed" });
|
||||
expect(rows[0]?.finishedAt).toBeInstanceOf(Date);
|
||||
expect(rows[0]?.metadata).toMatchObject({ failureReason: "runtime_control_timeout" });
|
||||
expect(rows[0]?.stderrExcerpt).toContain("time budget");
|
||||
|
||||
// No orphan active operation: a managed stop or retry goes straight through.
|
||||
await expect(
|
||||
service.assertRuntimeControlAvailable({ executionWorkspaceId, action: "stop" }),
|
||||
).resolves.toBeUndefined();
|
||||
await expect(service.reconcileStaleRuntimeControlOperations(executionWorkspaceId)).resolves.toEqual({
|
||||
reconciled: 0,
|
||||
operationIds: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("lets a managed retry succeed immediately after a failed start", async () => {
|
||||
const { companyId, executionWorkspaceId } = await seedWorkspace();
|
||||
const service = workspaceOperationService(db);
|
||||
const recorder = service.createRecorder({ companyId, executionWorkspaceId });
|
||||
|
||||
await expect(
|
||||
recorder.recordOperation({
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
metadata: { action: "start", executionWorkspaceId },
|
||||
run: async () => {
|
||||
throw new Error("Runtime service \"app\" could not start because port 42003 is already in use by pid 1234");
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(/already in use/);
|
||||
|
||||
await expect(
|
||||
service.assertRuntimeControlAvailable({ executionWorkspaceId, action: "start" }),
|
||||
).resolves.toBeUndefined();
|
||||
|
||||
const retry = await recorder.recordOperation({
|
||||
phase: "workspace_provision",
|
||||
command: "workspace command start",
|
||||
metadata: { action: "start", executionWorkspaceId },
|
||||
run: async () => ({ status: "succeeded" as const, exitCode: 0 }),
|
||||
});
|
||||
expect(retry.status).toBe("succeeded");
|
||||
|
||||
const statuses = await db
|
||||
.select({ status: workspaceOperations.status })
|
||||
.from(workspaceOperations)
|
||||
.where(eq(workspaceOperations.executionWorkspaceId, executionWorkspaceId));
|
||||
expect(statuses.map((row) => row.status).sort()).toEqual(["failed", "succeeded"]);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,259 @@
|
|||
/**
|
||||
* Opt-in LIVE exercise of the PAP-17158 in-place HTTPS backfill.
|
||||
*
|
||||
* Unlike `workspace-runtime.test.ts`, which drives the backfill through an
|
||||
* injected broker fake, this test uses the production exposure dependencies: the
|
||||
* real Unix-socket broker client, real Tailscale MagicDNS resolution, and a real
|
||||
* cert-validating HTTPS probe. It is the check that the fail-closed lifecycle
|
||||
* actually terminates in a browser-trusted `https://<node>.<tailnet>.ts.net:<port>`
|
||||
* URL rather than only in a mock's return value.
|
||||
*
|
||||
* It is skipped unless BOTH hold, because it mutates host-level Tailscale serve
|
||||
* state and must never run in CI:
|
||||
*
|
||||
* - `PAPERCLIP_LIVE_BROKER_EXERCISE=1`
|
||||
* - the broker socket exists (`PAPERCLIP_TAILSCALE_BROKER_SOCKET` or the default)
|
||||
*
|
||||
* Run it on a broker-provisioned host with:
|
||||
*
|
||||
* PAPERCLIP_LIVE_BROKER_EXERCISE=1 pnpm --filter @paperclipai/server exec \
|
||||
* vitest run src/__tests__/workspace-runtime-https-live-exercise.test.ts
|
||||
*
|
||||
* The caller must be the broker's configured service UID/GID and its listeners
|
||||
* must run as `BROKER_RUNTIME_UID`, or the broker correctly refuses to publish.
|
||||
*/
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
companies,
|
||||
createDb,
|
||||
projectWorkspaces,
|
||||
projects,
|
||||
workspaceRuntimeServices,
|
||||
type Db,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
reconcilePersistedRuntimeServicesOnStartup,
|
||||
resetRuntimeServicesForTests,
|
||||
startRuntimeServicesForWorkspaceControl,
|
||||
stopRuntimeServicesForProjectWorkspace,
|
||||
type RealizedExecutionWorkspace,
|
||||
} from "../services/workspace-runtime.ts";
|
||||
|
||||
const DEFAULT_BROKER_SOCKET = "/run/paperclip-tailscale-broker/broker.sock";
|
||||
const brokerSocketPath = process.env.PAPERCLIP_TAILSCALE_BROKER_SOCKET ?? DEFAULT_BROKER_SOCKET;
|
||||
|
||||
async function brokerSocketPresent() {
|
||||
try {
|
||||
return (await fs.stat(brokerSocketPath)).isSocket();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
const optedIn = process.env.PAPERCLIP_LIVE_BROKER_EXERCISE === "1";
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const live = optedIn && embeddedPostgresSupport.supported && (await brokerSocketPresent());
|
||||
|
||||
if (optedIn && !live) {
|
||||
console.warn(
|
||||
`[PAP-17158] live exercise opted in but skipped: broker socket at ${brokerSocketPath} `
|
||||
+ `present=${await brokerSocketPresent()}, embeddedPostgres=${embeddedPostgresSupport.supported}`,
|
||||
);
|
||||
}
|
||||
|
||||
(live ? describe : describe.skip)("PAP-17158 live HTTPS backfill exercise", () => {
|
||||
let db: Db;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("pap17158-live");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await resetRuntimeServicesForTests();
|
||||
await tempDb?.stop?.();
|
||||
}, 60_000);
|
||||
|
||||
it("upgrades a pre-existing HTTP workspace in place to a browser-trusted HTTPS URL", async () => {
|
||||
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "pap17158-live-"));
|
||||
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "pap17158-live-home-"));
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
const previousMode = process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS;
|
||||
process.env.PAPERCLIP_HOME = paperclipHome;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = `pap17158-live-${randomUUID()}`;
|
||||
|
||||
// An ephemeral legacy port rather than the real template's 45439, so this
|
||||
// never contends with the live workspace runtime on the same host.
|
||||
const reservePort = async () => {
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
const probe = net.createServer();
|
||||
await new Promise<void>((resolve) => probe.listen(0, "127.0.0.1", resolve));
|
||||
const address = probe.address();
|
||||
const port = typeof address === "object" && address ? address.port : null;
|
||||
await new Promise<void>((resolve, reject) => probe.close((e) => (e ? reject(e) : resolve())));
|
||||
if (port && port <= 55_535 && (port < 42_000 || port > 42_999)) return port;
|
||||
}
|
||||
throw new Error("failed to reserve a legacy port outside the broker range");
|
||||
};
|
||||
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
const projectWorkspaceId = randomUUID();
|
||||
const legacyPort = await reservePort();
|
||||
// Serves 200 at /api/health on the app port and its HMR companion, loopback
|
||||
// only — the shape the broker's /proc ownership proof requires.
|
||||
const command =
|
||||
"node -e \"const http=require('node:http');const p=Number(process.env.PORT);"
|
||||
+ "for(const q of [p,p+10000])http.createServer((req,res)=>{res.writeHead(200,{'content-type':'application/json'});"
|
||||
+ "res.end(JSON.stringify({status:'ok',port:q}))}).listen(q,'127.0.0.1');setInterval(()=>{},1000)\"";
|
||||
const workspaceRuntime = {
|
||||
services: [
|
||||
{
|
||||
name: "paperclip-dev",
|
||||
command,
|
||||
port: legacyPort,
|
||||
// Pre-feature block: backend URL only, no exposure declaration.
|
||||
expose: { type: "url", urlTemplate: "http://127.0.0.1:{{port}}" },
|
||||
readiness: {
|
||||
type: "http",
|
||||
urlTemplate: "http://127.0.0.1:{{port}}/api/health",
|
||||
timeoutSec: 20,
|
||||
intervalMs: 100,
|
||||
},
|
||||
lifecycle: "shared",
|
||||
reuseScope: "project_workspace",
|
||||
stopPolicy: { type: "manual" },
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `L${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "PAP-17158 live exercise",
|
||||
status: "in_progress",
|
||||
});
|
||||
await db.insert(projectWorkspaces).values({
|
||||
id: projectWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
name: "Primary",
|
||||
sourceType: "local_path",
|
||||
cwd: workspaceRoot,
|
||||
isPrimary: true,
|
||||
metadata: {
|
||||
runtimeConfig: { workspaceRuntime, desiredState: "running", serviceStates: { "0": "running" } },
|
||||
},
|
||||
});
|
||||
|
||||
const evidence: Record<string, unknown> = {};
|
||||
try {
|
||||
// ---- Before: the workspace as it exists today, plain HTTP. ----
|
||||
process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS = "off";
|
||||
const before = await startRuntimeServicesForWorkspaceControl({
|
||||
db,
|
||||
actor: { id: null, name: "Paperclip", companyId },
|
||||
issue: null,
|
||||
workspace: {
|
||||
baseCwd: workspaceRoot,
|
||||
source: "project_primary",
|
||||
projectId,
|
||||
workspaceId: projectWorkspaceId,
|
||||
repoUrl: null,
|
||||
repoRef: "HEAD",
|
||||
strategy: "project_primary",
|
||||
cwd: workspaceRoot,
|
||||
branchName: null,
|
||||
worktreePath: null,
|
||||
warnings: [],
|
||||
created: false,
|
||||
} satisfies RealizedExecutionWorkspace,
|
||||
config: { workspaceRuntime, desiredState: "running", serviceStates: { "0": "running" } },
|
||||
adapterEnv: {},
|
||||
});
|
||||
const runtimeServiceId = before[0]?.id;
|
||||
expect(runtimeServiceId).toBeTruthy();
|
||||
const [httpRow] = await db.select().from(workspaceRuntimeServices);
|
||||
expect(httpRow.exposure).toBeNull();
|
||||
evidence.beforeUrl = httpRow.url;
|
||||
evidence.beforePort = httpRow.port;
|
||||
expect(String(httpRow.url)).toMatch(/^http:\/\//);
|
||||
await expect(fetch(`http://127.0.0.1:${legacyPort}/api/health`)).resolves.toMatchObject({ ok: true });
|
||||
|
||||
// ---- Deploy: production exposure deps, automatic default on. ----
|
||||
await resetRuntimeServicesForTests(); // restores the real broker client
|
||||
delete process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS;
|
||||
|
||||
const result = await reconcilePersistedRuntimeServicesOnStartup(db);
|
||||
evidence.reconcile = result;
|
||||
expect(result.backfilled).toBe(1);
|
||||
expect(result.restartFailed).toBe(0);
|
||||
|
||||
// ---- After: same row, real cert-validated HTTPS URL. ----
|
||||
const afterRows = await db.select().from(workspaceRuntimeServices);
|
||||
expect(afterRows).toHaveLength(1);
|
||||
const httpsRow = afterRows[0]!;
|
||||
expect(httpsRow.id).toBe(runtimeServiceId);
|
||||
evidence.afterUrl = httpsRow.url;
|
||||
evidence.afterPort = httpsRow.port;
|
||||
evidence.afterExposureState = httpsRow.exposure?.state;
|
||||
expect(httpsRow.status).toBe("running");
|
||||
expect(httpsRow.exposure?.state).toBe("ready");
|
||||
expect(String(httpsRow.url)).toMatch(/^https:\/\/.+\.ts\.net:\d+$/);
|
||||
expect(httpsRow.port).toBeGreaterThanOrEqual(42_000);
|
||||
expect(httpsRow.port).toBeLessThanOrEqual(42_999);
|
||||
|
||||
// Strict TLS, no relaxed verification: this is the whole point.
|
||||
const probe = await fetch(`${httpsRow.url}/api/health`, { redirect: "error" });
|
||||
expect(probe.ok).toBe(true);
|
||||
evidence.liveProbeStatus = probe.status;
|
||||
|
||||
// The old HTTP backend is gone, not merely shadowed.
|
||||
await expect(fetch(`http://127.0.0.1:${legacyPort}/api/health`)).rejects.toThrow();
|
||||
|
||||
// ---- Repeat: idempotent, no churn. ----
|
||||
await resetRuntimeServicesForTests();
|
||||
const second = await reconcilePersistedRuntimeServicesOnStartup(db);
|
||||
expect(second.backfilled).toBe(0);
|
||||
const [repeatRow] = await db.select().from(workspaceRuntimeServices);
|
||||
expect(repeatRow.port).toBe(httpsRow.port);
|
||||
expect(repeatRow.url).toBe(httpsRow.url);
|
||||
evidence.repeatUrl = repeatRow.url;
|
||||
} finally {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log("[PAP-17158 live exercise]", JSON.stringify(evidence, null, 2));
|
||||
// Deprovisions the broker lease as well as the backend process.
|
||||
await stopRuntimeServicesForProjectWorkspace({
|
||||
db,
|
||||
projectWorkspaceId,
|
||||
workspaceCwd: workspaceRoot,
|
||||
}).catch((error) => console.error("[PAP-17158] teardown failed", error));
|
||||
await resetRuntimeServicesForTests();
|
||||
await fs.rm(paperclipHome, { recursive: true, force: true });
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = previousHome;
|
||||
if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
|
||||
else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;
|
||||
if (previousMode === undefined) delete process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS;
|
||||
else process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS = previousMode;
|
||||
}
|
||||
}, 180_000);
|
||||
});
|
||||
|
|
@ -0,0 +1,475 @@
|
|||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
agents,
|
||||
companies,
|
||||
createDb,
|
||||
executionWorkspaceRuntimeLeases,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
projects,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
WORKSPACE_RUNTIME_LEASE_TTL_MS,
|
||||
workspaceRuntimeLeaseService,
|
||||
} from "../services/workspace-runtime-leases.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping execution workspace runtime lease tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("execution workspace runtime leases", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
// A second client with its own connection pool stands in for a second server
|
||||
// process: nothing is shared in memory, so exclusivity has to come from Postgres.
|
||||
let otherProcessDb!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-runtime-lease-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
otherProcessDb = createDb(tempDb.connectionString);
|
||||
}, 30_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(executionWorkspaceRuntimeLeases);
|
||||
await db.delete(issues);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(projects);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
async function seedLane() {
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
const executionWorkspaceId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Runtime lease",
|
||||
issuePrefix: `L${companyId.replace(/-/g, "").slice(0, 7).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "Canary lane",
|
||||
status: "in_progress",
|
||||
});
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: executionWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
mode: "shared_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Shared canary workspace",
|
||||
status: "active",
|
||||
cwd: "/tmp/runtime-lease-workspace",
|
||||
});
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "Security Engineer",
|
||||
role: "engineer",
|
||||
});
|
||||
|
||||
return { companyId, projectId, executionWorkspaceId, agentId };
|
||||
}
|
||||
|
||||
async function seedIssue(
|
||||
lane: { companyId: string; projectId: string; executionWorkspaceId: string; agentId: string },
|
||||
input: { title: string; status?: string },
|
||||
) {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(issues).values({
|
||||
id: issueId,
|
||||
companyId: lane.companyId,
|
||||
projectId: lane.projectId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
title: input.title,
|
||||
status: input.status ?? "in_progress",
|
||||
priority: "high",
|
||||
assigneeAgentId: lane.agentId,
|
||||
});
|
||||
return issueId;
|
||||
}
|
||||
|
||||
async function seedRun(
|
||||
lane: { companyId: string; agentId: string },
|
||||
input: { status?: string } = {},
|
||||
) {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId: lane.companyId,
|
||||
agentId: lane.agentId,
|
||||
status: input.status ?? "running",
|
||||
});
|
||||
return runId;
|
||||
}
|
||||
|
||||
function ownerFor(input: { agentId: string; runId?: string | null; issueId?: string | null }) {
|
||||
return {
|
||||
actorType: "agent",
|
||||
agentId: input.agentId,
|
||||
runId: input.runId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
it("grants the first claim and lets the same owner keep operating across runs", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const firstRunId = await seedRun(lane);
|
||||
const secondRunId = await seedRun(lane);
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
const first = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, runId: firstRunId, issueId: canaryIssueId }),
|
||||
});
|
||||
expect(first.outcome).toBe("created");
|
||||
expect(first.ownerKey).toBe(`issue:${canaryIssueId}`);
|
||||
expect(first.lease?.lastAction).toBe("start");
|
||||
|
||||
// A later heartbeat of the same issue is still the same owner, even though the
|
||||
// run identity changed.
|
||||
const second = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "restart",
|
||||
owner: ownerFor({ agentId: lane.agentId, runId: secondRunId, issueId: canaryIssueId }),
|
||||
});
|
||||
expect(second.outcome).toBe("renewed");
|
||||
expect(second.lease?.id).toBe(first.lease?.id);
|
||||
expect(second.lease?.lastAction).toBe("restart");
|
||||
expect(second.lease?.ownerRunId).toBe(secondRunId);
|
||||
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("rejects a sequential competing issue with 409 and leaves the lease untouched", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const competingIssueId = await seedIssue(lane, { title: "Unrelated shared-workspace work" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
const held = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
});
|
||||
|
||||
await expect(leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: competingIssueId }),
|
||||
})).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: {
|
||||
code: "workspace_runtime_lease_conflict",
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
requestedAction: "start",
|
||||
ownerKey: `issue:${canaryIssueId}`,
|
||||
ownerIssueId: canaryIssueId,
|
||||
},
|
||||
});
|
||||
|
||||
const after = await leases.get(lane.executionWorkspaceId);
|
||||
expect(after?.id).toBe(held.lease?.id);
|
||||
expect(after?.ownerKey).toBe(`issue:${canaryIssueId}`);
|
||||
expect(after?.renewedAt.getTime()).toBe(held.lease?.renewedAt.getTime());
|
||||
});
|
||||
|
||||
it("serializes concurrent claims from separate database clients so exactly one owner wins", async () => {
|
||||
const lane = await seedLane();
|
||||
const contenderIssueIds = await Promise.all(
|
||||
Array.from({ length: 6 }, (_, index) => seedIssue(lane, { title: `Contender ${index}` })),
|
||||
);
|
||||
const leaseClients = [workspaceRuntimeLeaseService(db), workspaceRuntimeLeaseService(otherProcessDb)];
|
||||
|
||||
const results = await Promise.allSettled(
|
||||
contenderIssueIds.map((issueId, index) =>
|
||||
leaseClients[index % leaseClients.length]!.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId }),
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const granted = results.filter((result) => result.status === "fulfilled");
|
||||
const rejected = results.filter((result) => result.status === "rejected");
|
||||
expect(granted).toHaveLength(1);
|
||||
expect(rejected).toHaveLength(contenderIssueIds.length - 1);
|
||||
for (const failure of rejected) {
|
||||
expect((failure as PromiseRejectedResult).reason).toMatchObject({
|
||||
status: 409,
|
||||
details: { code: "workspace_runtime_lease_conflict" },
|
||||
});
|
||||
}
|
||||
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.ownerKey).toBe(
|
||||
(granted[0] as PromiseFulfilledResult<{ ownerKey: string | null }>).value.ownerKey,
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps a competing claim out from another database client after the lease is established", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const competingIssueId = await seedIssue(lane, { title: "Competing lane" });
|
||||
|
||||
await workspaceRuntimeLeaseService(db).claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
});
|
||||
|
||||
await expect(workspaceRuntimeLeaseService(otherProcessDb).claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "stop",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: competingIssueId }),
|
||||
})).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: { code: "workspace_runtime_lease_conflict", requestedAction: "stop" },
|
||||
});
|
||||
});
|
||||
|
||||
it("lets the teardown issue reclaim once the canary issue reaches a terminal status", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const teardownIssueId = await seedIssue(lane, { title: "Authorized teardown" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
});
|
||||
|
||||
await expect(leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "stop",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: teardownIssueId }),
|
||||
})).rejects.toMatchObject({ status: 409 });
|
||||
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, canaryIssueId));
|
||||
|
||||
const reclaimed = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "stop",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: teardownIssueId }),
|
||||
});
|
||||
expect(reclaimed.outcome).toBe("reclaimed");
|
||||
expect(reclaimed.reclaimedFrom).toEqual({
|
||||
ownerKey: `issue:${canaryIssueId}`,
|
||||
reason: "owner_issue_terminal",
|
||||
});
|
||||
expect(reclaimed.lease?.ownerIssueId).toBe(teardownIssueId);
|
||||
});
|
||||
|
||||
it("reclaims from a hidden owner issue and from a terminal owner run", async () => {
|
||||
const lane = await seedLane();
|
||||
const hiddenIssueId = await seedIssue(lane, { title: "Hidden owner" });
|
||||
const nextIssueId = await seedIssue(lane, { title: "Next owner" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: hiddenIssueId }),
|
||||
});
|
||||
await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, hiddenIssueId));
|
||||
|
||||
const afterHidden = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: nextIssueId }),
|
||||
});
|
||||
expect(afterHidden.reclaimedFrom?.reason).toBe("owner_issue_hidden");
|
||||
|
||||
// Run-scoped owners (no issue in run context) recover once the run is terminal.
|
||||
await leases.release({ executionWorkspaceId: lane.executionWorkspaceId, force: true });
|
||||
const staleRunId = await seedRun(lane, { status: "running" });
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, runId: staleRunId }),
|
||||
});
|
||||
|
||||
await expect(leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: nextIssueId }),
|
||||
})).rejects.toMatchObject({ status: 409 });
|
||||
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, staleRunId));
|
||||
const afterRunTerminal = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: nextIssueId }),
|
||||
});
|
||||
expect(afterRunTerminal.reclaimedFrom).toEqual({
|
||||
ownerKey: `run:${staleRunId}`,
|
||||
reason: "owner_run_terminal",
|
||||
});
|
||||
});
|
||||
|
||||
it("bounds recovery with a lease TTL even when the owner still looks eligible", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const competingIssueId = await seedIssue(lane, { title: "Competing lane" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
const claimedAt = new Date(Date.now() - WORKSPACE_RUNTIME_LEASE_TTL_MS - 60_000);
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
now: claimedAt,
|
||||
});
|
||||
|
||||
const reclaimed = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: competingIssueId }),
|
||||
});
|
||||
expect(reclaimed.reclaimedFrom).toEqual({
|
||||
ownerKey: `issue:${canaryIssueId}`,
|
||||
reason: "lease_expired",
|
||||
});
|
||||
});
|
||||
|
||||
it("lets a teardown issue claim immediately after the owner releases the lane", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const teardownIssueId = await seedIssue(lane, { title: "Authorized teardown" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
});
|
||||
|
||||
// A non-owner cannot release the lane out from under the owner.
|
||||
await expect(leases.release({
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: teardownIssueId }),
|
||||
})).resolves.toEqual({ released: false, ownerKey: null });
|
||||
|
||||
await expect(leases.release({
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
})).resolves.toEqual({ released: true, ownerKey: `issue:${canaryIssueId}` });
|
||||
|
||||
const claimed = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "stop",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: teardownIssueId }),
|
||||
});
|
||||
expect(claimed.outcome).toBe("created");
|
||||
expect(claimed.lease?.ownerIssueId).toBe(teardownIssueId);
|
||||
});
|
||||
|
||||
it("leaves board/operator control unleased", async () => {
|
||||
const lane = await seedLane();
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
});
|
||||
|
||||
const boardClaim = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "restart",
|
||||
owner: { actorType: "board", agentId: null, runId: null, issueId: null },
|
||||
});
|
||||
expect(boardClaim).toEqual({ outcome: "bypassed", ownerKey: null, lease: null, reclaimedFrom: null });
|
||||
|
||||
// The agent's lease survives the operator action.
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.ownerKey).toBe(`issue:${canaryIssueId}`);
|
||||
});
|
||||
|
||||
it("scopes leases per execution workspace", async () => {
|
||||
const lane = await seedLane();
|
||||
const otherWorkspaceId = randomUUID();
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: otherWorkspaceId,
|
||||
companyId: lane.companyId,
|
||||
projectId: lane.projectId,
|
||||
mode: "shared_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Second lane",
|
||||
status: "active",
|
||||
cwd: "/tmp/runtime-lease-workspace-2",
|
||||
});
|
||||
const canaryIssueId = await seedIssue(lane, { title: "HTTPS canary" });
|
||||
const competingIssueId = await seedIssue(lane, { title: "Competing lane" });
|
||||
const leases = workspaceRuntimeLeaseService(db);
|
||||
|
||||
await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: lane.executionWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: canaryIssueId }),
|
||||
});
|
||||
const other = await leases.claim({
|
||||
companyId: lane.companyId,
|
||||
executionWorkspaceId: otherWorkspaceId,
|
||||
action: "start",
|
||||
owner: ownerFor({ agentId: lane.agentId, issueId: competingIssueId }),
|
||||
});
|
||||
expect(other.outcome).toBe("created");
|
||||
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases);
|
||||
expect(rows).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
|
@ -26,6 +26,11 @@ const mockEnvironmentService = vi.hoisted(() => ({
|
|||
}));
|
||||
|
||||
const mockWorkspaceOperationService = vi.hoisted(() => ({}));
|
||||
const mockWorkspaceRuntimeLeaseService = vi.hoisted(() => ({
|
||||
claim: vi.fn(async () => ({ outcome: "created", ownerKey: "issue:issue-1", lease: null, reclaimedFrom: null })),
|
||||
release: vi.fn(async () => ({ released: false, ownerKey: null })),
|
||||
get: vi.fn(async () => null),
|
||||
}));
|
||||
const mockHeartbeatService = vi.hoisted(() => ({}));
|
||||
const mockLogActivity = vi.hoisted(() => vi.fn());
|
||||
const mockGetTelemetryClient = vi.hoisted(() => vi.fn());
|
||||
|
|
@ -48,6 +53,8 @@ vi.mock("../services/index.js", () => ({
|
|||
projectService: () => mockProjectService,
|
||||
secretService: () => mockSecretService,
|
||||
workspaceOperationService: () => mockWorkspaceOperationService,
|
||||
workspaceRuntimeLeaseService: () => mockWorkspaceRuntimeLeaseService,
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS: ["start", "stop", "restart"],
|
||||
}));
|
||||
|
||||
vi.mock("../services/workspace-runtime.js", () => ({
|
||||
|
|
@ -76,6 +83,8 @@ function registerWorkspaceRouteMocks() {
|
|||
projectService: () => mockProjectService,
|
||||
secretService: () => mockSecretService,
|
||||
workspaceOperationService: () => mockWorkspaceOperationService,
|
||||
workspaceRuntimeLeaseService: () => mockWorkspaceRuntimeLeaseService,
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS: ["start", "stop", "restart"],
|
||||
}));
|
||||
|
||||
vi.doMock("../services/workspace-runtime.js", () => ({
|
||||
|
|
|
|||
|
|
@ -349,7 +349,11 @@ describeEmbeddedPostgres("workspace runtime service authz helper", () => {
|
|||
} as any, {
|
||||
companyId,
|
||||
executionWorkspaceId,
|
||||
})).resolves.toBeUndefined();
|
||||
})).resolves.toMatchObject({
|
||||
actorType: "agent",
|
||||
agentId: managerId,
|
||||
issueId: null,
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects unrelated same-company agents without matching workspace assignments", async () => {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,161 @@
|
|||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
RUNTIME_SERVICE_READINESS_PROBE_TIMEOUT_MS,
|
||||
allocateRuntimeServicePort,
|
||||
claimRuntimeServiceBindPort,
|
||||
resetRuntimeServicePortReservationsForTests,
|
||||
waitForRuntimeServiceReadiness,
|
||||
} from "../services/workspace-runtime.js";
|
||||
|
||||
/**
|
||||
* Regression fixture for the adverse report on PAP-17207: two exclusive isolated workspaces
|
||||
* asked for the same configured app/HMR pair at the same moment. One lane came up healthy,
|
||||
* the loser was reallocated onto a port owned by an unrelated listener, and its managed start
|
||||
* never reached a terminal state — leaving an active operation that rejected managed cleanup.
|
||||
*
|
||||
* The two mechanisms that produced the non-terminality are covered here:
|
||||
* 1. a readiness probe against a socket that accepts but never answers, and
|
||||
* 2. concurrent allocation handing the same port to two starts.
|
||||
*/
|
||||
describe("managed runtime start terminality", () => {
|
||||
afterEach(() => {
|
||||
resetRuntimeServicePortReservationsForTests();
|
||||
});
|
||||
|
||||
const hangingService = {
|
||||
readiness: { type: "http", timeoutSec: 1, intervalMs: 50 },
|
||||
} as Record<string, unknown>;
|
||||
|
||||
it("fails a readiness check whose probes never answer instead of hanging forever", async () => {
|
||||
let probes = 0;
|
||||
const abortedProbes: string[] = [];
|
||||
/** Accepts the request and only ever settles when the caller aborts, like a foreign listener. */
|
||||
const neverAnsweringFetch = (async (_url: string | URL | Request, init?: RequestInit) => {
|
||||
probes += 1;
|
||||
return await new Promise<Response>((_resolve, reject) => {
|
||||
init?.signal?.addEventListener("abort", () => {
|
||||
abortedProbes.push(String(init?.signal?.reason ?? "aborted"));
|
||||
reject(new Error("The operation was aborted due to timeout"));
|
||||
});
|
||||
});
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
const startedAt = Date.now();
|
||||
await expect(
|
||||
waitForRuntimeServiceReadiness({
|
||||
service: hangingService,
|
||||
serviceName: "app",
|
||||
command: "pnpm dev",
|
||||
url: "http://127.0.0.1:45439/",
|
||||
readinessUrl: null,
|
||||
fetchImpl: neverAnsweringFetch,
|
||||
}),
|
||||
).rejects.toThrow(/Readiness check failed for http:\/\/127\.0\.0\.1:45439\//);
|
||||
const elapsedMs = Date.now() - startedAt;
|
||||
|
||||
expect(probes).toBeGreaterThan(0);
|
||||
expect(abortedProbes.length).toBeGreaterThan(0);
|
||||
// Terminal well inside the 1s readiness budget plus one probe budget.
|
||||
expect(elapsedMs).toBeLessThan(1_000 + RUNTIME_SERVICE_READINESS_PROBE_TIMEOUT_MS);
|
||||
});
|
||||
|
||||
it("negative control: an unbounded probe loop never terminalizes the same start", async () => {
|
||||
// This is the shape of the code that shipped: `fetch` with no abort signal, so the
|
||||
// deadline at the top of the loop is never re-evaluated once a probe parks.
|
||||
const legacyWait = async () => {
|
||||
const deadline = Date.now() + 1_000;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise<never>(() => {});
|
||||
await delay(50);
|
||||
}
|
||||
throw new Error("Readiness check failed");
|
||||
};
|
||||
|
||||
const settled = await Promise.race([
|
||||
legacyWait().then(() => "settled").catch(() => "settled"),
|
||||
delay(1_500).then(() => "still-hanging"),
|
||||
]);
|
||||
expect(settled).toBe("still-hanging");
|
||||
});
|
||||
|
||||
it("stops probing as soon as the readiness budget is spent", async () => {
|
||||
let probes = 0;
|
||||
const refusingFetch = (async () => {
|
||||
probes += 1;
|
||||
throw new Error("connect ECONNREFUSED 127.0.0.1:45439");
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
await expect(
|
||||
waitForRuntimeServiceReadiness({
|
||||
service: { readiness: { type: "http", timeoutSec: 1, intervalMs: 200 } },
|
||||
url: "http://127.0.0.1:45439/",
|
||||
readinessUrl: null,
|
||||
fetchImpl: refusingFetch,
|
||||
}),
|
||||
).rejects.toThrow(/ECONNREFUSED/);
|
||||
// ~1s budget at a 200ms interval: a bounded handful of probes, never an unbounded spin.
|
||||
expect(probes).toBeGreaterThan(1);
|
||||
expect(probes).toBeLessThan(12);
|
||||
});
|
||||
|
||||
it("hands two concurrent starts distinct ports even when the kernel repeats a candidate", async () => {
|
||||
// The kernel really does hand the same ephemeral port to two `listen(0)` probes once the
|
||||
// first probe socket has closed, which is exactly the reallocation collision in the report.
|
||||
const candidates = [49871, 49871, 49871, 49872];
|
||||
let index = 0;
|
||||
const probe = async () => candidates[Math.min(index++, candidates.length - 1)]!;
|
||||
const portOwnerLookup = async () => null;
|
||||
|
||||
const [first, second] = await Promise.all([
|
||||
allocateRuntimeServicePort({ probe, portOwnerLookup }),
|
||||
allocateRuntimeServicePort({ probe, portOwnerLookup }),
|
||||
]);
|
||||
|
||||
expect(first).not.toBe(second);
|
||||
expect(new Set([first, second])).toEqual(new Set([49871, 49872]));
|
||||
});
|
||||
|
||||
it("skips a candidate port that a live process already owns", async () => {
|
||||
const candidates = [49881, 49882];
|
||||
let index = 0;
|
||||
const probe = async () => candidates[Math.min(index++, candidates.length - 1)]!;
|
||||
const portOwnerLookup = async (port: number) => (port === 49881 ? 4242 : null);
|
||||
|
||||
await expect(allocateRuntimeServicePort({ probe, portOwnerLookup })).resolves.toBe(49882);
|
||||
// The rejected candidate must not stay reserved, or later starts would leak allocations.
|
||||
await expect(
|
||||
allocateRuntimeServicePort({ probe: async () => 49881, portOwnerLookup: async () => null }),
|
||||
).resolves.toBe(49881);
|
||||
});
|
||||
|
||||
it("refuses a configured port a sibling start is already claiming", () => {
|
||||
// The reported collision was on a *configured* app/HMR pair, not an auto-allocated one:
|
||||
// both lanes read the pair as free because neither had bound or persisted a row yet.
|
||||
expect(claimRuntimeServiceBindPort(42003, null)).toBe(true);
|
||||
expect(claimRuntimeServiceBindPort(42003, null)).toBe(false);
|
||||
// The HMR companion is claimed independently, so a lane can lose on either port.
|
||||
expect(claimRuntimeServiceBindPort(42004, null)).toBe(true);
|
||||
expect(claimRuntimeServiceBindPort(42004, null)).toBe(false);
|
||||
});
|
||||
|
||||
it("never lets a start be refused by its own allocation", async () => {
|
||||
// An auto-allocated port arrives already reserved by this start, and its identity port can
|
||||
// resolve to that same value. Treating that as a sibling's claim would fail the start on its
|
||||
// own reservation.
|
||||
const held = await allocateRuntimeServicePort({
|
||||
probe: async () => 49901,
|
||||
portOwnerLookup: async () => null,
|
||||
});
|
||||
expect(held).toBe(49901);
|
||||
expect(claimRuntimeServiceBindPort(49901, held)).toBe(true);
|
||||
// A different start still loses against that held reservation.
|
||||
expect(claimRuntimeServiceBindPort(49901, null)).toBe(false);
|
||||
});
|
||||
|
||||
it("fails terminally rather than looping when no free port can be found", async () => {
|
||||
await expect(
|
||||
allocateRuntimeServicePort({ probe: async () => 49891, portOwnerLookup: async () => 99 }),
|
||||
).rejects.toThrow(/Could not allocate a free loopback port/);
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,9 +1,10 @@
|
|||
import express, { Router, type Request as ExpressRequest } from "express";
|
||||
import { createServer as createHttpServer, type Server as HttpServer } from "node:http";
|
||||
import path from "node:path";
|
||||
import fs from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { DeploymentExposure, DeploymentMode } from "@paperclipai/shared";
|
||||
import { derivePaperclipViteHmrPort, type DeploymentExposure, type DeploymentMode } from "@paperclipai/shared";
|
||||
import type { InspectDatabaseBackupHealthOptions } from "./services/database-backup-health.js";
|
||||
import type { StorageService } from "./storage/types.js";
|
||||
import { httpLogger, errorHandler } from "./middleware/index.js";
|
||||
|
|
@ -131,18 +132,43 @@ export function isDatabaseConnectionUnavailableError(err: unknown): boolean {
|
|||
}
|
||||
|
||||
export function resolveViteHmrPort(serverPort: number): number {
|
||||
if (serverPort <= 55_535) {
|
||||
return serverPort + 10_000;
|
||||
}
|
||||
return Math.max(1_024, serverPort - 10_000);
|
||||
return derivePaperclipViteHmrPort(serverPort);
|
||||
}
|
||||
|
||||
export function resolveViteHmrHost(bindHost: string): string | undefined {
|
||||
const normalized = bindHost.trim().toLowerCase();
|
||||
if (normalized === "0.0.0.0" || normalized === "::") return undefined;
|
||||
if (
|
||||
normalized === "0.0.0.0"
|
||||
|| normalized === "::"
|
||||
|| normalized === "127.0.0.1"
|
||||
|| normalized === "::1"
|
||||
|| normalized === "localhost"
|
||||
) return undefined;
|
||||
return bindHost;
|
||||
}
|
||||
|
||||
export function resolveViteHmrProtocol(value: string | undefined): "ws" | "wss" | undefined {
|
||||
if (!value) return undefined;
|
||||
if (value === "ws" || value === "wss") return value;
|
||||
throw new Error("PAPERCLIP_VITE_HMR_PROTOCOL must be ws or wss");
|
||||
}
|
||||
|
||||
export function listenViteHmrServer(server: HttpServer, port: number, bindHost: string): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const onError = (error: Error) => {
|
||||
server.off("listening", onListening);
|
||||
reject(error);
|
||||
};
|
||||
const onListening = () => {
|
||||
server.off("error", onError);
|
||||
resolve();
|
||||
};
|
||||
server.once("error", onError);
|
||||
server.once("listening", onListening);
|
||||
server.listen(port, bindHost);
|
||||
});
|
||||
}
|
||||
|
||||
export function shouldServeViteDevHtml(req: ExpressRequest): boolean {
|
||||
const pathname = req.path;
|
||||
if (VITE_DEV_STATIC_PATHS.has(pathname)) return false;
|
||||
|
|
@ -512,6 +538,8 @@ export async function createApp(
|
|||
});
|
||||
const hostServiceCleanup = createPluginHostServiceCleanup(lifecycle, hostServicesDisposers);
|
||||
let viteHtmlRenderer: ReturnType<typeof createCachedViteHtmlRenderer> | null = null;
|
||||
let viteDevServer: { close(): Promise<void> } | null = null;
|
||||
let viteHmrServer: HttpServer | null = null;
|
||||
const loader = pluginLoader(
|
||||
db,
|
||||
{
|
||||
|
|
@ -642,20 +670,39 @@ export async function createApp(
|
|||
const publicUiRoot = path.resolve(uiRoot, "public");
|
||||
const hmrPort = resolveViteHmrPort(opts.serverPort);
|
||||
const hmrHost = resolveViteHmrHost(opts.bindHost);
|
||||
const hmrProtocol = resolveViteHmrProtocol(process.env.PAPERCLIP_VITE_HMR_PROTOCOL);
|
||||
const hmrServer = createHttpServer((_req, res) => {
|
||||
res.writeHead(426, { "Content-Type": "text/plain" });
|
||||
res.end("Upgrade Required");
|
||||
});
|
||||
const { createServer: createViteServer } = await import("vite");
|
||||
const vite = await createViteServer({
|
||||
root: uiRoot,
|
||||
appType: "custom",
|
||||
server: {
|
||||
// Listener binding and browser HMR hostname are deliberately separate:
|
||||
// exposed branch runtimes stay loopback-only while the browser uses the
|
||||
// current MagicDNS hostname through the broker's HTTPS listener.
|
||||
host: opts.bindHost,
|
||||
middlewareMode: true,
|
||||
hmr: {
|
||||
server: hmrServer,
|
||||
...(hmrHost ? { host: hmrHost } : {}),
|
||||
...(hmrProtocol ? { protocol: hmrProtocol } : {}),
|
||||
port: hmrPort,
|
||||
clientPort: hmrPort,
|
||||
},
|
||||
allowedHosts: privateHostnameGateEnabled ? Array.from(privateHostnameAllowSet) : undefined,
|
||||
},
|
||||
});
|
||||
try {
|
||||
await listenViteHmrServer(hmrServer, hmrPort, opts.bindHost);
|
||||
} catch (error) {
|
||||
await vite.close();
|
||||
throw error;
|
||||
}
|
||||
viteDevServer = vite;
|
||||
viteHmrServer = hmrServer;
|
||||
viteHtmlRenderer = createCachedViteHtmlRenderer({
|
||||
vite,
|
||||
uiRoot,
|
||||
|
|
@ -829,6 +876,8 @@ export async function createApp(
|
|||
}
|
||||
devWatcher?.close();
|
||||
viteHtmlRenderer?.dispose();
|
||||
void viteDevServer?.close().catch(() => undefined);
|
||||
viteHmrServer?.close();
|
||||
hostServiceCleanup.disposeAll();
|
||||
hostServiceCleanup.teardown();
|
||||
// Cancel every live setup-token login session, so each direct child stops
|
||||
|
|
|
|||
|
|
@ -62,6 +62,7 @@ import {
|
|||
routineService,
|
||||
statusCardService,
|
||||
toolAccessService,
|
||||
workspaceOperationService,
|
||||
} from "./services/index.js";
|
||||
import { queueIssueAssignmentWakeup } from "./services/issue-assignment-wakeup.js";
|
||||
import { createSecretProposalsService } from "./services/secret-proposals.js";
|
||||
|
|
@ -816,11 +817,38 @@ export async function startServer(): Promise<StartedServer> {
|
|||
},
|
||||
});
|
||||
|
||||
try {
|
||||
const result = await workspaceOperationService(db as any)
|
||||
.reconcileStaleRuntimeControlOperations();
|
||||
if (result.reconciled > 0) {
|
||||
logger.warn(
|
||||
{ reconciled: result.reconciled, operationIds: result.operationIds },
|
||||
"reconciled stale managed runtime control operations from a previous server process",
|
||||
);
|
||||
}
|
||||
} catch (err) {
|
||||
logger.error({ err }, "startup reconciliation of managed runtime control operations failed");
|
||||
}
|
||||
|
||||
void reconcilePersistedRuntimeServicesOnStartup(db as any)
|
||||
.then((result) => {
|
||||
if (result.reconciled > 0) {
|
||||
if (
|
||||
result.reconciled > 0
|
||||
|| result.restarted > 0
|
||||
|| result.restartFailed > 0
|
||||
|| result.backfilled > 0
|
||||
) {
|
||||
logger.warn(
|
||||
{ reconciled: result.reconciled },
|
||||
{
|
||||
reconciled: result.reconciled,
|
||||
adopted: result.adopted,
|
||||
stopped: result.stopped,
|
||||
// Managed HTTP-only services taken down so they come back on a
|
||||
// verified HTTPS origin (PAP-17158).
|
||||
httpsBackfilled: result.backfilled,
|
||||
restarted: result.restarted,
|
||||
restartFailed: result.restartFailed,
|
||||
},
|
||||
"reconciled persisted runtime services from a previous server process",
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,11 +12,17 @@ import {
|
|||
} from "@paperclipai/shared";
|
||||
import type { WorkspaceRuntimeDesiredState, WorkspaceRuntimeServiceStateMap } from "@paperclipai/shared";
|
||||
import { validate } from "../middleware/validate.js";
|
||||
import { accessService, executionWorkspaceService, heartbeatService, logActivity, workspaceOperationService } from "../services/index.js";
|
||||
import {
|
||||
mergeExecutionWorkspaceConfig,
|
||||
readExecutionWorkspaceConfig,
|
||||
} from "../services/execution-workspaces.js";
|
||||
accessService,
|
||||
executionWorkspaceService,
|
||||
heartbeatService,
|
||||
logActivity,
|
||||
workspaceOperationService,
|
||||
workspaceRuntimeLeaseService,
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS,
|
||||
type WorkspaceRuntimeLeaseClaim,
|
||||
} from "../services/index.js";
|
||||
import { mergeExecutionWorkspaceConfig, readExecutionWorkspaceConfig } from "../services/execution-workspaces.js";
|
||||
import { parseProjectExecutionWorkspacePolicy } from "../services/execution-workspace-policy.js";
|
||||
import { readProjectWorkspaceRuntimeConfig } from "../services/project-workspace-runtime-config.js";
|
||||
import {
|
||||
|
|
@ -38,6 +44,7 @@ import { assertCanManageExecutionWorkspaceRuntimeServices } from "./workspace-ru
|
|||
import { appendWithCap } from "../adapters/utils.js";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { runExclusiveWorkspaceRuntimeControl } from "../services/workspace-operations.js";
|
||||
|
||||
const WORKSPACE_CONTROL_OUTPUT_MAX_CHARS = 256 * 1024;
|
||||
|
||||
|
|
@ -46,6 +53,7 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
const svc = executionWorkspaceService(db);
|
||||
const access = accessService(db);
|
||||
const workspaceOperationsSvc = workspaceOperationService(db);
|
||||
const runtimeLeases = workspaceRuntimeLeaseService(db);
|
||||
const heartbeat = heartbeatService(db, {
|
||||
pluginWorkerManager: opts.pluginWorkerManager,
|
||||
});
|
||||
|
|
@ -152,12 +160,20 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
if (!existing) return;
|
||||
if (!(await assertRuntimeManageAllowed(req, res, existing.companyId))) return;
|
||||
|
||||
await assertCanManageExecutionWorkspaceRuntimeServices(db, req, {
|
||||
const authorization = await assertCanManageExecutionWorkspaceRuntimeServices(db, req, {
|
||||
companyId: existing.companyId,
|
||||
executionWorkspaceId: existing.id,
|
||||
sourceIssueId: existing.sourceIssueId,
|
||||
});
|
||||
|
||||
// Recover any managed runtime-control operation this workspace was stranded with, then
|
||||
// refuse only if one is genuinely still live. Authorization above still gates the caller,
|
||||
// so recovery never widens who may control the workspace.
|
||||
await workspaceOperationsSvc.assertRuntimeControlAvailable({
|
||||
executionWorkspaceId: existing.id,
|
||||
action,
|
||||
});
|
||||
|
||||
const workspaceCwd = existing.cwd;
|
||||
if (!workspaceCwd) {
|
||||
res.status(422).json({ error: "Execution workspace needs a local path before Paperclip can run workspace commands" });
|
||||
|
|
@ -260,7 +276,57 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
let stdout = "";
|
||||
let stderr = "";
|
||||
|
||||
const operation = await recorder.recordOperation({
|
||||
/**
|
||||
* Bring runtime rows, local listeners and the recorded desired state back to a consistent
|
||||
* stopped shape after a start failed part-way. Best-effort by design: the operation is
|
||||
* still failed (terminal) even if teardown itself cannot complete, and every step is
|
||||
* something a normal managed `stop` would do, so authorization is unchanged.
|
||||
*/
|
||||
let failedStartReconciled = false;
|
||||
async function reconcileFailedRuntimeStart(cause: unknown) {
|
||||
if (failedStartReconciled) return;
|
||||
failedStartReconciled = true;
|
||||
try {
|
||||
await stopRuntimeServicesForExecutionWorkspace({
|
||||
db,
|
||||
executionWorkspaceId: existing!.id,
|
||||
workspaceCwd: workspaceCwd!,
|
||||
runtimeServiceId: selectedRuntimeServiceId,
|
||||
});
|
||||
} catch (teardownError) {
|
||||
logger.warn(
|
||||
{
|
||||
executionWorkspaceId: existing!.id,
|
||||
err: teardownError,
|
||||
cause: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
"failed to tear down runtime services after a failed managed start",
|
||||
);
|
||||
}
|
||||
|
||||
try {
|
||||
const failedDesiredState = buildWorkspaceRuntimeDesiredStatePatch({
|
||||
config: { workspaceRuntime: effectiveRuntimeConfig },
|
||||
currentDesiredState: existing!.config?.desiredState ?? null,
|
||||
currentServiceStates: existing!.config?.serviceStates ?? null,
|
||||
action: "stop",
|
||||
serviceIndex: selectedServiceIndex,
|
||||
});
|
||||
await svc.update(existing!.id, {
|
||||
metadata: mergeExecutionWorkspaceConfig(existing!.metadata as Record<string, unknown> | null, {
|
||||
desiredState: failedDesiredState.desiredState,
|
||||
serviceStates: failedDesiredState.serviceStates,
|
||||
}),
|
||||
});
|
||||
} catch (patchError) {
|
||||
logger.warn(
|
||||
{ executionWorkspaceId: existing!.id, err: patchError },
|
||||
"failed to record the stopped desired state after a failed managed start",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const recordRuntimeControlOperation = () => recorder.recordOperation({
|
||||
phase: action === "stop" ? "workspace_teardown" : "workspace_provision",
|
||||
command: workspaceCommand?.command ?? `workspace command ${action}`,
|
||||
cwd: existing.cwd,
|
||||
|
|
@ -377,34 +443,43 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
if (!availableWorkspace) {
|
||||
throw new Error("Execution workspace needs a local path before Paperclip can manage local runtime services");
|
||||
}
|
||||
const startedServices = await startRuntimeServicesForWorkspaceControl({
|
||||
db,
|
||||
actor: {
|
||||
id: actor.agentId ?? null,
|
||||
name: actor.actorType === "user" ? "Board" : "Agent",
|
||||
companyId: existing.companyId,
|
||||
},
|
||||
issue: existing.sourceIssueId
|
||||
? {
|
||||
id: existing.sourceIssueId,
|
||||
identifier: null,
|
||||
title: existing.name,
|
||||
}
|
||||
: null,
|
||||
workspace: availableWorkspace,
|
||||
executionWorkspaceId: existing.id,
|
||||
config: {
|
||||
workspaceRuntime: effectiveRuntimeConfig,
|
||||
runtimeProvisionCommand:
|
||||
existing.config?.runtimeProvisionCommand
|
||||
?? projectPolicy?.workspaceStrategy?.runtimeProvisionCommand
|
||||
?? null,
|
||||
},
|
||||
adapterEnv: {},
|
||||
onLog,
|
||||
recorder,
|
||||
serviceIndex: selectedServiceIndex,
|
||||
});
|
||||
let startedServices;
|
||||
try {
|
||||
startedServices = await startRuntimeServicesForWorkspaceControl({
|
||||
db,
|
||||
actor: {
|
||||
id: actor.agentId ?? null,
|
||||
name: actor.actorType === "user" ? "Board" : "Agent",
|
||||
companyId: existing.companyId,
|
||||
},
|
||||
issue: existing.sourceIssueId
|
||||
? {
|
||||
id: existing.sourceIssueId,
|
||||
identifier: null,
|
||||
title: existing.name,
|
||||
}
|
||||
: null,
|
||||
workspace: availableWorkspace,
|
||||
executionWorkspaceId: existing.id,
|
||||
config: {
|
||||
workspaceRuntime: effectiveRuntimeConfig,
|
||||
runtimeProvisionCommand:
|
||||
existing.config?.runtimeProvisionCommand
|
||||
?? projectPolicy?.workspaceStrategy?.runtimeProvisionCommand
|
||||
?? null,
|
||||
},
|
||||
adapterEnv: {},
|
||||
onLog,
|
||||
recorder,
|
||||
serviceIndex: selectedServiceIndex,
|
||||
});
|
||||
} catch (error) {
|
||||
// A failed start must leave the workspace stopped and retryable rather than
|
||||
// "desired running" with a half-started listener: tear down residue and record
|
||||
// the stopped desired state before the operation is marked failed.
|
||||
await reconcileFailedRuntimeStart(error);
|
||||
throw error;
|
||||
}
|
||||
runtimeServiceCount = startedServices.length;
|
||||
} else {
|
||||
runtimeServiceCount = selectedRuntimeServiceId ? Math.max(0, (existing.runtimeServices?.length ?? 1) - 1) : 0;
|
||||
|
|
@ -458,6 +533,44 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
},
|
||||
});
|
||||
|
||||
const { operation, leaseClaim } = await runExclusiveWorkspaceRuntimeControl({
|
||||
executionWorkspaceId: existing.id,
|
||||
action,
|
||||
run: async () => {
|
||||
// Claim the durable exclusivity lease before anything mutates the workspace or
|
||||
// its runtime services. A competing issue/run throws 409 here, leaving no
|
||||
// workspace operation row and no runtime-service change behind.
|
||||
let leaseClaimResult: WorkspaceRuntimeLeaseClaim | null = null;
|
||||
if (LEASED_WORKSPACE_RUNTIME_ACTIONS.includes(action)) {
|
||||
leaseClaimResult = await runtimeLeases.claim({
|
||||
companyId: existing.companyId,
|
||||
executionWorkspaceId: existing.id,
|
||||
action,
|
||||
owner: authorization,
|
||||
});
|
||||
}
|
||||
// Re-check inside the claim: recovery of stranded operations plus a refusal if one
|
||||
// is genuinely still live. The same assertion ran before the lease was taken, but a
|
||||
// row can be stranded in that window, and only the recovering path may proceed.
|
||||
await workspaceOperationsSvc.assertRuntimeControlAvailable({
|
||||
executionWorkspaceId: existing.id,
|
||||
action,
|
||||
});
|
||||
try {
|
||||
return {
|
||||
operation: await recordRuntimeControlOperation(),
|
||||
leaseClaim: leaseClaimResult,
|
||||
};
|
||||
} catch (error) {
|
||||
// The operation is already terminal here. This also catches the recorder's own time
|
||||
// budget expiring on a start that never settles, which is the one failure the inner
|
||||
// handler cannot see — reconcile there too so no listener or desired state is left over.
|
||||
if (action === "start" || action === "restart") await reconcileFailedRuntimeStart(error);
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
const workspace = await svc.getById(id);
|
||||
if (!workspace) {
|
||||
res.status(404).json({ error: "Execution workspace not found" });
|
||||
|
|
@ -481,6 +594,13 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
workspaceCommandName: workspaceCommand?.name ?? null,
|
||||
runtimeServiceId: selectedRuntimeServiceId,
|
||||
serviceIndex: selectedServiceIndex,
|
||||
runtimeLease: leaseClaim
|
||||
? {
|
||||
outcome: leaseClaim.outcome,
|
||||
ownerKey: leaseClaim.ownerKey,
|
||||
reclaimedFrom: leaseClaim.reclaimedFrom,
|
||||
}
|
||||
: null,
|
||||
},
|
||||
});
|
||||
|
||||
|
|
@ -665,6 +785,10 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P
|
|||
workspace = archiveResult.workspace;
|
||||
const capturedGeneration = archiveResult.capturedGeneration;
|
||||
|
||||
// Closing the workspace ends the lane, so the runtime-control lease is released
|
||||
// outright rather than waiting for owner-eligibility or TTL recovery.
|
||||
await runtimeLeases.release({ executionWorkspaceId: existing.id, force: true });
|
||||
|
||||
if (existing.mode === "shared_workspace") {
|
||||
await db
|
||||
.update(issues)
|
||||
|
|
|
|||
|
|
@ -5543,6 +5543,7 @@ export function issueRoutes(
|
|||
startedAt: service.startedAt,
|
||||
stoppedAt: service.stoppedAt,
|
||||
healthStatus: service.healthStatus,
|
||||
exposure: service.exposure ?? null,
|
||||
configIndex: service.configIndex ?? null,
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,15 +8,21 @@ import { assertCompanyAccess, hasCompanyAccess } from "./authz.js";
|
|||
import { parseProjectExecutionWorkspacePolicy } from "../services/execution-workspace-policy.js";
|
||||
import { isLowTrustRuntimeManagementAllowed } from "../services/low-trust-runtime-containment.js";
|
||||
import { resolveCoreTrustPreset, type TrustPresetResolution } from "../services/trust-preset-resolver.js";
|
||||
import { WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES } from "../services/workspace-runtime-leases.js";
|
||||
import { readObject } from "../lib/objects.js";
|
||||
|
||||
const WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES: string[] = [
|
||||
"backlog",
|
||||
"todo",
|
||||
"in_progress",
|
||||
"in_review",
|
||||
"blocked",
|
||||
];
|
||||
const WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUS_LIST: string[] = [...WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES];
|
||||
|
||||
/**
|
||||
* Identity of the actor authorized to drive a workspace runtime control, resolved once
|
||||
* so the durable runtime lease and the authorization decision agree on who is calling.
|
||||
*/
|
||||
export type WorkspaceRuntimeControlAuthorization = {
|
||||
actorType: string;
|
||||
agentId: string | null;
|
||||
runId: string | null;
|
||||
issueId: string | null;
|
||||
};
|
||||
|
||||
function readRunIssueId(context: Record<string, unknown> | null) {
|
||||
const directIssueId = context?.issueId;
|
||||
|
|
@ -68,7 +74,7 @@ async function assertAgentCanManageRuntimeServicesForWorkspace(
|
|||
executionWorkspaceId?: string | null;
|
||||
sourceIssueId?: string | null;
|
||||
},
|
||||
) {
|
||||
): Promise<{ runIssueId: string | null }> {
|
||||
if (req.actor.type !== "agent" || !req.actor.agentId) {
|
||||
throw forbidden("Agent authentication required");
|
||||
}
|
||||
|
|
@ -112,11 +118,12 @@ async function assertAgentCanManageRuntimeServicesForWorkspace(
|
|||
runExecutionPolicy,
|
||||
});
|
||||
|
||||
const runIssueId = readRunIssueId(runContext);
|
||||
|
||||
if (actorAgent.role === "ceo" && actorRuntimeTrust.kind === "standard") {
|
||||
return;
|
||||
return { runIssueId };
|
||||
}
|
||||
|
||||
const runIssueId = readRunIssueId(runContext);
|
||||
const runScopedIssue = runIssueId
|
||||
? await db
|
||||
.select({
|
||||
|
|
@ -171,7 +178,7 @@ async function assertAgentCanManageRuntimeServicesForWorkspace(
|
|||
.where(and(
|
||||
eq(issues.companyId, input.companyId),
|
||||
isNull(issues.hiddenAt),
|
||||
inArray(issues.status, WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES),
|
||||
inArray(issues.status, WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUS_LIST),
|
||||
workspaceScopeCondition,
|
||||
));
|
||||
|
||||
|
|
@ -185,7 +192,7 @@ async function assertAgentCanManageRuntimeServicesForWorkspace(
|
|||
}
|
||||
|
||||
if (actorAgent.role === "ceo") {
|
||||
return;
|
||||
return { runIssueId };
|
||||
}
|
||||
|
||||
const eligibleAgentIds = await listReportingSubtreeAgentIds(db, input.companyId, actorAgent.id);
|
||||
|
|
@ -202,7 +209,7 @@ async function assertAgentCanManageRuntimeServicesForWorkspace(
|
|||
.where(and(
|
||||
eq(issues.companyId, input.companyId),
|
||||
isNull(issues.hiddenAt),
|
||||
inArray(issues.status, WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES),
|
||||
inArray(issues.status, WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUS_LIST),
|
||||
inArray(issues.assigneeAgentId, eligibleAgentIds),
|
||||
workspaceScopeCondition,
|
||||
))
|
||||
|
|
@ -215,7 +222,7 @@ async function assertAgentCanManageRuntimeServicesForWorkspace(
|
|||
projectExecutionWorkspacePolicy: linkedIssue.projectExecutionWorkspacePolicy,
|
||||
runExecutionPolicy,
|
||||
});
|
||||
return;
|
||||
return { runIssueId };
|
||||
}
|
||||
|
||||
throw forbidden("Missing permission to manage workspace runtime services");
|
||||
|
|
@ -321,11 +328,19 @@ export async function assertCanManageExecutionWorkspaceRuntimeServices(
|
|||
executionWorkspaceId: string;
|
||||
sourceIssueId?: string | null;
|
||||
},
|
||||
) {
|
||||
): Promise<WorkspaceRuntimeControlAuthorization> {
|
||||
if (!hasCompanyAccess(req, input.companyId)) {
|
||||
throw notFound("Execution workspace not found");
|
||||
}
|
||||
assertCompanyAccess(req, input.companyId);
|
||||
if (req.actor.type === "board") return;
|
||||
await assertAgentCanManageRuntimeServicesForWorkspace(db, req, input);
|
||||
if (req.actor.type === "board") {
|
||||
return { actorType: "board", agentId: null, runId: null, issueId: null };
|
||||
}
|
||||
const { runIssueId } = await assertAgentCanManageRuntimeServicesForWorkspace(db, req, input);
|
||||
return {
|
||||
actorType: req.actor.type,
|
||||
agentId: req.actor.agentId ?? null,
|
||||
runId: req.actor.runId ?? null,
|
||||
issueId: runIssueId,
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -981,6 +981,7 @@ function toRuntimeService(
|
|||
stoppedAt: row.stoppedAt ?? null,
|
||||
stopPolicy: (row.stopPolicy as Record<string, unknown> | null) ?? null,
|
||||
healthStatus: row.healthStatus as WorkspaceRuntimeService["healthStatus"],
|
||||
exposure: (row.exposure as WorkspaceRuntimeService["exposure"]) ?? null,
|
||||
configIndex: row.configIndex ?? null,
|
||||
createdAt: row.createdAt,
|
||||
updatedAt: row.updatedAt,
|
||||
|
|
@ -1059,6 +1060,7 @@ function toWorkspaceOverviewPrimaryService(
|
|||
url: service.url,
|
||||
port: service.port,
|
||||
healthStatus: service.healthStatus,
|
||||
exposure: service.exposure ?? null,
|
||||
updatedAt: service.updatedAt,
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -158,6 +158,16 @@ export {
|
|||
} from "./environment-custom-image-terminal-sessions.js";
|
||||
export { executionWorkspaceService } from "./execution-workspaces.js";
|
||||
export { workspaceOperationService } from "./workspace-operations.js";
|
||||
export {
|
||||
workspaceRuntimeLeaseService,
|
||||
buildWorkspaceRuntimeLeaseOwnerKey,
|
||||
LEASED_WORKSPACE_RUNTIME_ACTIONS,
|
||||
WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES,
|
||||
WORKSPACE_RUNTIME_LEASE_TTL_MS,
|
||||
type WorkspaceRuntimeLeaseClaim,
|
||||
type WorkspaceRuntimeLeaseOwner,
|
||||
type WorkspaceRuntimeLeaseService,
|
||||
} from "./workspace-runtime-leases.js";
|
||||
export { workspaceFileResourceService } from "./workspace-file-resources.js";
|
||||
export { workProductService } from "./work-products.js";
|
||||
export {
|
||||
|
|
|
|||
|
|
@ -285,7 +285,7 @@ export async function findAdoptableLocalService(input: {
|
|||
return record;
|
||||
}
|
||||
|
||||
async function readProcessGroupId(pid: number) {
|
||||
export async function readLocalServiceProcessGroupId(pid: number) {
|
||||
if (process.platform === "win32") return null;
|
||||
try {
|
||||
const { stdout } = await execFileAsync("ps", ["-o", "pgid=", "-p", String(pid)]);
|
||||
|
|
@ -296,6 +296,39 @@ async function readProcessGroupId(pid: number) {
|
|||
}
|
||||
}
|
||||
|
||||
export async function isLocalServiceProcessOwnedBy(pid: number, ownerProcessId: number) {
|
||||
if (pid === ownerProcessId) return true;
|
||||
if (process.platform !== "win32") {
|
||||
return (await readLocalServiceProcessGroupId(pid)) === ownerProcessId;
|
||||
}
|
||||
|
||||
try {
|
||||
const script = [
|
||||
`$currentProcessId = ${pid}`,
|
||||
"while ($currentProcessId -gt 0) {",
|
||||
" $process = Get-CimInstance Win32_Process -Filter \"ProcessId = $currentProcessId\" -ErrorAction SilentlyContinue",
|
||||
" if ($null -eq $process) { break }",
|
||||
" $parentProcessId = [int]$process.ParentProcessId",
|
||||
" Write-Output $parentProcessId",
|
||||
" if ($parentProcessId -eq $currentProcessId) { break }",
|
||||
" $currentProcessId = $parentProcessId",
|
||||
"}",
|
||||
].join("\n");
|
||||
const { stdout } = await execFileAsync("powershell.exe", [
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
script,
|
||||
]);
|
||||
return stdout
|
||||
.split(/\r?\n/)
|
||||
.map((line) => Number.parseInt(line.trim(), 10))
|
||||
.some((ancestorPid) => ancestorPid === ownerProcessId);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function adoptLocalServiceFromPortOwner(input: {
|
||||
serviceKey: string;
|
||||
profileKind?: string | null;
|
||||
|
|
@ -317,7 +350,7 @@ async function adoptLocalServiceFromPortOwner(input: {
|
|||
}
|
||||
}
|
||||
|
||||
const processGroupId = await readProcessGroupId(ownerPid);
|
||||
const processGroupId = await readLocalServiceProcessGroupId(ownerPid);
|
||||
const pid = processGroupId && isPidAlive(processGroupId) ? processGroupId : ownerPid;
|
||||
const now = new Date().toISOString();
|
||||
const record: LocalServiceRegistryRecord = {
|
||||
|
|
@ -405,8 +438,24 @@ export async function terminateLocalService(
|
|||
}
|
||||
|
||||
export async function readLocalServicePortOwner(port: number) {
|
||||
if (!Number.isInteger(port) || port <= 0 || process.platform === "win32") return null;
|
||||
if (!Number.isInteger(port) || port <= 0) return null;
|
||||
try {
|
||||
if (process.platform === "win32") {
|
||||
const { stdout } = await execFileAsync("netstat", ["-ano", "-p", "tcp"]);
|
||||
for (const line of stdout.split(/\r?\n/)) {
|
||||
const columns = line.trim().split(/\s+/);
|
||||
if (columns.length < 5 || columns[0]?.toUpperCase() !== "TCP") continue;
|
||||
const localAddress = columns[1] ?? "";
|
||||
const separatorIndex = localAddress.lastIndexOf(":");
|
||||
const localPort = Number.parseInt(localAddress.slice(separatorIndex + 1), 10);
|
||||
const state = columns.at(-2)?.toUpperCase();
|
||||
const pid = Number.parseInt(columns.at(-1) ?? "", 10);
|
||||
if (localPort === port && state === "LISTENING" && Number.isInteger(pid) && pid > 0) {
|
||||
return pid;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
const { stdout } = await execFileAsync("lsof", ["-nP", `-iTCP:${port}`, "-sTCP:LISTEN", "-t"]);
|
||||
const firstPid = stdout
|
||||
.split("\n")
|
||||
|
|
|
|||
|
|
@ -0,0 +1,196 @@
|
|||
import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { existsSync, unlinkSync } from "node:fs";
|
||||
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { BrokerClientError, UnixBrokerClient } from "./broker-client.js";
|
||||
|
||||
const LENGTH_PREFIX_BYTES = 4;
|
||||
|
||||
/**
|
||||
* Minimal in-process broker that speaks the exact `[4-byte BE length][json]`
|
||||
* framing of the real socket server, so these tests exercise the client's
|
||||
* framing + parsing without linking the host-privileged broker package.
|
||||
*/
|
||||
function startFakeBroker(
|
||||
handler: (request: Record<string, unknown>) => unknown,
|
||||
): { socketPath: string; server: net.Server } {
|
||||
const socketPath = path.join(os.tmpdir(), `tsh-broker-${randomUUID()}.sock`);
|
||||
const server = net.createServer((socket) => {
|
||||
const chunks: Buffer[] = [];
|
||||
let expected = -1;
|
||||
socket.on("data", (chunk) => {
|
||||
chunks.push(chunk);
|
||||
const buf = Buffer.concat(chunks);
|
||||
if (expected < 0) {
|
||||
if (buf.byteLength < LENGTH_PREFIX_BYTES) return;
|
||||
expected = buf.readUInt32BE(0);
|
||||
}
|
||||
if (buf.byteLength < LENGTH_PREFIX_BYTES + expected) return;
|
||||
const body = buf.subarray(LENGTH_PREFIX_BYTES, LENGTH_PREFIX_BYTES + expected).toString("utf8");
|
||||
const request = JSON.parse(body) as Record<string, unknown>;
|
||||
const response = handler(request);
|
||||
const respBody = Buffer.from(JSON.stringify(response), "utf8");
|
||||
const frame = Buffer.allocUnsafe(LENGTH_PREFIX_BYTES + respBody.byteLength);
|
||||
frame.writeUInt32BE(respBody.byteLength, 0);
|
||||
respBody.copy(frame, LENGTH_PREFIX_BYTES);
|
||||
socket.end(frame);
|
||||
});
|
||||
});
|
||||
server.listen(socketPath);
|
||||
return { socketPath, server };
|
||||
}
|
||||
|
||||
const servers: net.Server[] = [];
|
||||
const sockets: string[] = [];
|
||||
|
||||
function fake(handler: (request: Record<string, unknown>) => unknown): string {
|
||||
const { socketPath, server } = startFakeBroker(handler);
|
||||
servers.push(server);
|
||||
sockets.push(socketPath);
|
||||
return socketPath;
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
for (const server of servers.splice(0)) {
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
}
|
||||
for (const socketPath of sockets.splice(0)) {
|
||||
if (existsSync(socketPath)) {
|
||||
try {
|
||||
unlinkSync(socketPath);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
const RUNTIME_ID = "11111111-2222-4333-8444-555566667777";
|
||||
|
||||
describe("UnixBrokerClient", () => {
|
||||
it("round-trips reservation and expose requests", async () => {
|
||||
let seen: Record<string, unknown> | undefined;
|
||||
const socketPath = fake((request) => {
|
||||
seen = request;
|
||||
if (request.op === "reserve") {
|
||||
return { ok: true, op: "reserve", requestId: request.requestId, handle: "h".repeat(20), reservedPorts: [42000, 52000] };
|
||||
}
|
||||
return { ok: true, op: "expose", requestId: request.requestId, handle: "h".repeat(20), publicPorts: [42000, 52000] };
|
||||
});
|
||||
const client = new UnixBrokerClient({ socketPath });
|
||||
const reserved = await client.reserve(RUNTIME_ID, [
|
||||
{ purpose: "app", port: 42000 },
|
||||
{ purpose: "vite_hmr", port: 52000 },
|
||||
]);
|
||||
expect(reserved).toEqual({ handle: "h".repeat(20), reservedPorts: [42000, 52000] });
|
||||
const result = await client.expose(RUNTIME_ID, reserved.handle);
|
||||
expect(result).toEqual({ handle: "h".repeat(20), publicPorts: [42000, 52000] });
|
||||
expect(seen).toMatchObject({
|
||||
v: 1,
|
||||
op: "expose",
|
||||
runtimeId: RUNTIME_ID,
|
||||
handle: "h".repeat(20),
|
||||
});
|
||||
expect(typeof seen?.requestId).toBe("string");
|
||||
});
|
||||
|
||||
it("parses a remove response", async () => {
|
||||
const socketPath = fake((request) => ({
|
||||
ok: true,
|
||||
op: "remove",
|
||||
requestId: request.requestId,
|
||||
removedPorts: [42000],
|
||||
}));
|
||||
const client = new UnixBrokerClient({ socketPath });
|
||||
const result = await client.remove(RUNTIME_ID, "h".repeat(20));
|
||||
expect(result.removedPorts).toEqual([42000]);
|
||||
});
|
||||
|
||||
it("parses a list response into owned listeners", async () => {
|
||||
const socketPath = fake((request) => ({
|
||||
ok: true,
|
||||
op: "list",
|
||||
requestId: request.requestId,
|
||||
listeners: [{ runtimeId: RUNTIME_ID, port: 42000, purpose: "app" }],
|
||||
}));
|
||||
const client = new UnixBrokerClient({ socketPath });
|
||||
const listeners = await client.list();
|
||||
expect(listeners).toEqual([{ runtimeId: RUNTIME_ID, port: 42000, purpose: "app" }]);
|
||||
});
|
||||
|
||||
it("surfaces a broker error response as a coded BrokerClientError", async () => {
|
||||
const socketPath = fake((request) => ({
|
||||
ok: false,
|
||||
requestId: request.requestId,
|
||||
code: "port_not_allowlisted",
|
||||
message: "denied",
|
||||
}));
|
||||
const client = new UnixBrokerClient({ socketPath });
|
||||
await expect(client.reserve(RUNTIME_ID, [{ purpose: "app", port: 42000 }])).rejects.toMatchObject({
|
||||
name: "BrokerClientError",
|
||||
code: "port_not_allowlisted",
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects a malformed (non-ok) response shape", async () => {
|
||||
const socketPath = fake(() => ({ surprise: true }));
|
||||
const client = new UnixBrokerClient({ socketPath });
|
||||
await expect(client.list()).rejects.toBeInstanceOf(BrokerClientError);
|
||||
});
|
||||
|
||||
it("rejects an expose response missing publicPorts", async () => {
|
||||
const socketPath = fake((request) => ({ ok: true, op: "expose", requestId: request.requestId, handle: "h".repeat(20) }));
|
||||
const client = new UnixBrokerClient({ socketPath });
|
||||
await expect(client.expose(RUNTIME_ID, "h".repeat(20))).rejects.toMatchObject({
|
||||
code: "malformed_response",
|
||||
});
|
||||
});
|
||||
|
||||
it("times out when the broker accepts but never responds", async () => {
|
||||
await expect(silentTimeout()).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it("errors when the socket path does not exist (transport error)", async () => {
|
||||
const client = new UnixBrokerClient({
|
||||
socketPath: path.join(os.tmpdir(), `missing-${randomUUID()}.sock`),
|
||||
timeoutMs: 500,
|
||||
});
|
||||
await expect(client.list()).rejects.toMatchObject({ code: "transport_error" });
|
||||
});
|
||||
});
|
||||
|
||||
/** Build a truly-silent broker and assert the client times out against it. */
|
||||
async function silentTimeout(): Promise<boolean> {
|
||||
const socketPath = path.join(os.tmpdir(), `silent-${randomUUID()}.sock`);
|
||||
const connections: net.Socket[] = [];
|
||||
const server = net.createServer((socket) => {
|
||||
// Accept, never respond. Swallow the abortive close the client sends when
|
||||
// it times out so the connection socket does not emit an unhandled error.
|
||||
socket.on("error", () => {});
|
||||
connections.push(socket);
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(socketPath, () => resolve()));
|
||||
try {
|
||||
const client = new UnixBrokerClient({ socketPath, timeoutMs: 150 });
|
||||
try {
|
||||
await client.list();
|
||||
return false;
|
||||
} catch (error) {
|
||||
return error instanceof BrokerClientError && error.code === "cli_timeout";
|
||||
}
|
||||
} finally {
|
||||
for (const socket of connections) socket.destroy();
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
if (existsSync(socketPath)) {
|
||||
try {
|
||||
unlinkSync(socketPath);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,292 @@
|
|||
/**
|
||||
* Server-side client for the least-privilege Tailscale HTTPS host broker.
|
||||
*
|
||||
* The Paperclip server process holds NO Tailscale operator authority. Its only
|
||||
* way to publish a same-number HTTPS-to-loopback listener is to ask the broker
|
||||
* over its Unix domain socket. This client speaks the exact versioned,
|
||||
* length-prefixed, byte-bounded wire protocol implemented by the broker's
|
||||
* socket server (PAP-17049 plan, PAP-17050 verdict requirement #5).
|
||||
*
|
||||
* Framing (must match `packages/tailscale-https-broker/src/socket-server.ts`):
|
||||
* [4-byte big-endian body length][utf-8 JSON body]
|
||||
* One request per connection; the broker writes a single framed response and
|
||||
* closes the socket.
|
||||
*
|
||||
* Everything the broker returns is untrusted: the client validates the response
|
||||
* shape before handing it back and never logs lease handles.
|
||||
*/
|
||||
import net from "node:net";
|
||||
|
||||
/**
|
||||
* Must match the broker's `BROKER_PROTOCOL_VERSION`. Kept as a local constant
|
||||
* (rather than a dependency on the host-side broker package, which is installed
|
||||
* and versioned separately under its own operator account) so the server does
|
||||
* not import host-privileged code just to speak the wire protocol.
|
||||
*/
|
||||
const BROKER_PROTOCOL_VERSION = 1;
|
||||
|
||||
/** Must match the broker's `MAX_REQUEST_BYTES` (8 KiB). */
|
||||
const MAX_REQUEST_BYTES = 8 * 1024;
|
||||
/** Must match the broker's `MAX_RESPONSE_BYTES` (16 KiB). */
|
||||
const MAX_RESPONSE_BYTES = 16 * 1024;
|
||||
const LENGTH_PREFIX_BYTES = 4;
|
||||
|
||||
/** Listener purposes the broker understands. */
|
||||
export type BrokerListenerPurpose = "app" | "vite_hmr";
|
||||
|
||||
export interface BrokerListenerRequest {
|
||||
purpose: BrokerListenerPurpose;
|
||||
/** Public HTTPS port == loopback target port (same-number invariant). */
|
||||
port: number;
|
||||
}
|
||||
|
||||
export interface BrokerExposeResult {
|
||||
handle: string;
|
||||
publicPorts: number[];
|
||||
}
|
||||
|
||||
export interface BrokerReserveResult {
|
||||
handle: string;
|
||||
reservedPorts: number[];
|
||||
}
|
||||
|
||||
export interface BrokerRemoveResult {
|
||||
removedPorts: number[];
|
||||
}
|
||||
|
||||
export interface BrokerOwnedListener {
|
||||
runtimeId: string;
|
||||
port: number;
|
||||
purpose: BrokerListenerPurpose;
|
||||
}
|
||||
|
||||
/**
|
||||
* Error surfaced to the exposure manager. `code` is the broker's stable machine
|
||||
* code (or a transport-level code) so lifecycle logic can branch without
|
||||
* string-matching human messages.
|
||||
*/
|
||||
export class BrokerClientError extends Error {
|
||||
constructor(
|
||||
readonly code: string,
|
||||
message: string,
|
||||
) {
|
||||
super(message);
|
||||
this.name = "BrokerClientError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimal broker capability surface the exposure manager depends on. Defining
|
||||
* it as an interface lets tests inject a fake broker with zero sockets.
|
||||
*/
|
||||
export interface BrokerClient {
|
||||
reserve(runtimeId: string, listeners: BrokerListenerRequest[]): Promise<BrokerReserveResult>;
|
||||
expose(runtimeId: string, handle: string): Promise<BrokerExposeResult>;
|
||||
remove(runtimeId: string, handle: string): Promise<BrokerRemoveResult>;
|
||||
list(): Promise<BrokerOwnedListener[]>;
|
||||
}
|
||||
|
||||
export interface UnixBrokerClientOptions {
|
||||
socketPath: string;
|
||||
/** Per-request connect+round-trip deadline. Defaults to 5s. */
|
||||
timeoutMs?: number;
|
||||
/**
|
||||
* Seam for tests: open a duplex stream to the broker. Defaults to a real Unix
|
||||
* socket connection. Any injected connector must behave like a `net.Socket`.
|
||||
*/
|
||||
connect?: (socketPath: string) => net.Socket;
|
||||
}
|
||||
|
||||
let requestCounter = 0;
|
||||
|
||||
function nextRequestId(): string {
|
||||
requestCounter = (requestCounter + 1) % 1_000_000;
|
||||
// Matches the broker's REQUEST_ID_RE (`[A-Za-z0-9_-]{1,64}`). No time/random
|
||||
// needed for correctness; the broker keys responses by requestId echo only.
|
||||
return `req-${requestCounter.toString(36)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Unix-domain-socket implementation of {@link BrokerClient}. Each call opens a
|
||||
* fresh connection, sends one length-prefixed frame, reads one length-prefixed
|
||||
* response frame, and closes.
|
||||
*/
|
||||
export class UnixBrokerClient implements BrokerClient {
|
||||
private readonly socketPath: string;
|
||||
private readonly timeoutMs: number;
|
||||
private readonly connect: (socketPath: string) => net.Socket;
|
||||
|
||||
constructor(options: UnixBrokerClientOptions) {
|
||||
this.socketPath = options.socketPath;
|
||||
this.timeoutMs = options.timeoutMs ?? 5_000;
|
||||
this.connect = options.connect ?? ((p) => net.createConnection(p));
|
||||
}
|
||||
|
||||
async reserve(
|
||||
runtimeId: string,
|
||||
listeners: BrokerListenerRequest[],
|
||||
): Promise<BrokerReserveResult> {
|
||||
const response = await this.roundTrip({
|
||||
v: BROKER_PROTOCOL_VERSION,
|
||||
op: "reserve",
|
||||
requestId: nextRequestId(),
|
||||
runtimeId,
|
||||
listeners: listeners.map((l) => ({ purpose: l.purpose, port: l.port })),
|
||||
});
|
||||
if (
|
||||
typeof response.handle !== "string" ||
|
||||
!Array.isArray(response.reservedPorts) ||
|
||||
!response.reservedPorts.every((p) => Number.isInteger(p))
|
||||
) {
|
||||
throw new BrokerClientError("malformed_response", "reserve response missing handle/reservedPorts");
|
||||
}
|
||||
return { handle: response.handle, reservedPorts: response.reservedPorts as number[] };
|
||||
}
|
||||
|
||||
async expose(runtimeId: string, handle: string): Promise<BrokerExposeResult> {
|
||||
const response = await this.roundTrip({
|
||||
v: BROKER_PROTOCOL_VERSION,
|
||||
op: "expose",
|
||||
requestId: nextRequestId(),
|
||||
runtimeId,
|
||||
handle,
|
||||
});
|
||||
if (
|
||||
typeof response.handle !== "string" ||
|
||||
!Array.isArray(response.publicPorts) ||
|
||||
!response.publicPorts.every((p) => Number.isInteger(p))
|
||||
) {
|
||||
throw new BrokerClientError("malformed_response", "expose response missing handle/publicPorts");
|
||||
}
|
||||
return { handle: response.handle, publicPorts: response.publicPorts as number[] };
|
||||
}
|
||||
|
||||
async remove(runtimeId: string, handle: string): Promise<BrokerRemoveResult> {
|
||||
const response = await this.roundTrip({
|
||||
v: BROKER_PROTOCOL_VERSION,
|
||||
op: "remove",
|
||||
requestId: nextRequestId(),
|
||||
runtimeId,
|
||||
handle,
|
||||
});
|
||||
if (!Array.isArray(response.removedPorts) || !response.removedPorts.every((p) => Number.isInteger(p))) {
|
||||
throw new BrokerClientError("malformed_response", "remove response missing removedPorts");
|
||||
}
|
||||
return { removedPorts: response.removedPorts as number[] };
|
||||
}
|
||||
|
||||
async list(): Promise<BrokerOwnedListener[]> {
|
||||
const response = await this.roundTrip({
|
||||
v: BROKER_PROTOCOL_VERSION,
|
||||
op: "list",
|
||||
requestId: nextRequestId(),
|
||||
});
|
||||
if (!Array.isArray(response.listeners)) {
|
||||
throw new BrokerClientError("malformed_response", "list response missing listeners");
|
||||
}
|
||||
return response.listeners.map((entry): BrokerOwnedListener => {
|
||||
const listener = entry as Record<string, unknown>;
|
||||
if (
|
||||
typeof listener.runtimeId !== "string" ||
|
||||
!Number.isInteger(listener.port) ||
|
||||
(listener.purpose !== "app" && listener.purpose !== "vite_hmr")
|
||||
) {
|
||||
throw new BrokerClientError("malformed_response", "list returned a malformed listener");
|
||||
}
|
||||
return {
|
||||
runtimeId: listener.runtimeId,
|
||||
port: listener.port as number,
|
||||
purpose: listener.purpose,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Send one request frame, resolve the single framed response as an object. */
|
||||
private roundTrip(request: unknown): Promise<Record<string, unknown>> {
|
||||
const body = Buffer.from(JSON.stringify(request), "utf8");
|
||||
if (body.byteLength > MAX_REQUEST_BYTES) {
|
||||
return Promise.reject(new BrokerClientError("request_too_large", "broker request exceeds size limit"));
|
||||
}
|
||||
const frame = Buffer.allocUnsafe(LENGTH_PREFIX_BYTES + body.byteLength);
|
||||
frame.writeUInt32BE(body.byteLength, 0);
|
||||
body.copy(frame, LENGTH_PREFIX_BYTES);
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const socket = this.connect(this.socketPath);
|
||||
const chunks: Buffer[] = [];
|
||||
let received = 0;
|
||||
let expected = -1;
|
||||
let settled = false;
|
||||
|
||||
const done = (fn: () => void): void => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
socket.removeAllListeners();
|
||||
socket.destroy();
|
||||
fn();
|
||||
};
|
||||
|
||||
const timer = setTimeout(() => {
|
||||
done(() => reject(new BrokerClientError("cli_timeout", "broker request timed out")));
|
||||
}, this.timeoutMs);
|
||||
timer.unref?.();
|
||||
|
||||
socket.on("error", (err: Error) => {
|
||||
done(() => reject(new BrokerClientError("transport_error", err.message)));
|
||||
});
|
||||
|
||||
socket.on("close", () => {
|
||||
// Closed before a full response was parsed.
|
||||
done(() => reject(new BrokerClientError("transport_error", "broker closed connection early")));
|
||||
});
|
||||
|
||||
socket.on("data", (chunk: Buffer) => {
|
||||
received += chunk.byteLength;
|
||||
if (received > LENGTH_PREFIX_BYTES + MAX_RESPONSE_BYTES) {
|
||||
done(() => reject(new BrokerClientError("malformed_response", "broker response exceeds size limit")));
|
||||
return;
|
||||
}
|
||||
chunks.push(chunk);
|
||||
const buf = Buffer.concat(chunks);
|
||||
if (expected < 0) {
|
||||
if (buf.byteLength < LENGTH_PREFIX_BYTES) return;
|
||||
expected = buf.readUInt32BE(0);
|
||||
if (expected > MAX_RESPONSE_BYTES) {
|
||||
done(() => reject(new BrokerClientError("malformed_response", "broker response length invalid")));
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (buf.byteLength < LENGTH_PREFIX_BYTES + expected) return;
|
||||
const payload = buf.subarray(LENGTH_PREFIX_BYTES, LENGTH_PREFIX_BYTES + expected).toString("utf8");
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(payload);
|
||||
} catch {
|
||||
done(() => reject(new BrokerClientError("malformed_response", "broker response is not valid JSON")));
|
||||
return;
|
||||
}
|
||||
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
|
||||
done(() => reject(new BrokerClientError("malformed_response", "broker response is not an object")));
|
||||
return;
|
||||
}
|
||||
const obj = parsed as Record<string, unknown>;
|
||||
if (obj.ok === false) {
|
||||
const code = typeof obj.code === "string" ? obj.code : "internal_error";
|
||||
const message = typeof obj.message === "string" ? obj.message : code;
|
||||
done(() => reject(new BrokerClientError(code, message)));
|
||||
return;
|
||||
}
|
||||
if (obj.ok !== true) {
|
||||
done(() => reject(new BrokerClientError("malformed_response", "broker response missing ok flag")));
|
||||
return;
|
||||
}
|
||||
done(() => resolve(obj));
|
||||
});
|
||||
|
||||
socket.on("connect", () => {
|
||||
socket.write(frame);
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,449 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { RUNTIME_EXPOSURE_APP_PORT_MIN, deriveViteHmrPort } from "@paperclipai/shared";
|
||||
|
||||
import {
|
||||
BrokerClientError,
|
||||
type BrokerClient,
|
||||
type BrokerExposeResult,
|
||||
type BrokerListenerRequest,
|
||||
type BrokerOwnedListener,
|
||||
type BrokerReserveResult,
|
||||
type BrokerRemoveResult,
|
||||
} from "./broker-client.js";
|
||||
import {
|
||||
deprovisionExposure,
|
||||
provisionExposure,
|
||||
reconcileExposures,
|
||||
reserveExposure,
|
||||
type ExposureManagerDeps,
|
||||
} from "./exposure-manager.js";
|
||||
|
||||
const RUNTIME_ID = "11111111-2222-4333-8444-555566667777";
|
||||
const APP_PORT = RUNTIME_EXPOSURE_APP_PORT_MIN;
|
||||
const HMR_PORT = deriveViteHmrPort(APP_PORT);
|
||||
const HOSTNAME = "runner-abc.tail-scale.ts.net";
|
||||
|
||||
const CONFIG = {
|
||||
type: "tailscale_https" as const,
|
||||
hostname: "auto" as const,
|
||||
publicPort: "same" as const,
|
||||
includePaperclipViteHmr: true,
|
||||
failurePolicy: "fail_closed" as const,
|
||||
};
|
||||
|
||||
interface FakeBrokerScript {
|
||||
reserve?: (runtimeId: string, listeners: BrokerListenerRequest[]) => BrokerReserveResult;
|
||||
expose?: (runtimeId: string, handle: string) => BrokerExposeResult;
|
||||
remove?: (runtimeId: string, handle: string) => BrokerRemoveResult;
|
||||
list?: () => BrokerOwnedListener[];
|
||||
}
|
||||
|
||||
interface FakeBrokerCalls {
|
||||
reserve: Array<{ runtimeId: string; listeners: BrokerListenerRequest[] }>;
|
||||
expose: Array<{ runtimeId: string; handle: string }>;
|
||||
remove: Array<{ runtimeId: string; handle: string }>;
|
||||
list: number;
|
||||
}
|
||||
|
||||
function fakeBroker(script: FakeBrokerScript): { broker: BrokerClient; calls: FakeBrokerCalls } {
|
||||
const calls: FakeBrokerCalls = { reserve: [], expose: [], remove: [], list: 0 };
|
||||
const broker: BrokerClient = {
|
||||
async reserve(runtimeId, listeners) {
|
||||
calls.reserve.push({ runtimeId, listeners });
|
||||
if (!script.reserve) throw new BrokerClientError("internal_error", "no reserve script");
|
||||
return script.reserve(runtimeId, listeners);
|
||||
},
|
||||
async expose(runtimeId, handle) {
|
||||
calls.expose.push({ runtimeId, handle });
|
||||
if (!script.expose) throw new BrokerClientError("internal_error", "no expose script");
|
||||
return script.expose(runtimeId, handle);
|
||||
},
|
||||
async remove(runtimeId, handle) {
|
||||
calls.remove.push({ runtimeId, handle });
|
||||
if (!script.remove) throw new BrokerClientError("internal_error", "no remove script");
|
||||
return script.remove(runtimeId, handle);
|
||||
},
|
||||
async list() {
|
||||
calls.list += 1;
|
||||
return script.list ? script.list() : [];
|
||||
},
|
||||
};
|
||||
return { broker, calls };
|
||||
}
|
||||
|
||||
const HANDLE = "handle-abcdef1234567890";
|
||||
|
||||
describe("reserveExposure", () => {
|
||||
it("reserves the exact app + HMR pair before startup", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
reserve: () => ({ handle: HANDLE, reservedPorts: [APP_PORT, HMR_PORT] }),
|
||||
});
|
||||
const result = await reserveExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(calls.reserve).toEqual([{
|
||||
runtimeId: RUNTIME_ID,
|
||||
listeners: [
|
||||
{ purpose: "app", port: APP_PORT },
|
||||
{ purpose: "vite_hmr", port: HMR_PORT },
|
||||
],
|
||||
}]);
|
||||
expect(result.handle).toBe(HANDLE);
|
||||
expect(result.status.state).toBe("pending");
|
||||
});
|
||||
|
||||
it("releases a malformed reservation response and fails closed", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
reserve: () => ({ handle: HANDLE, reservedPorts: [APP_PORT] }),
|
||||
remove: () => ({ removedPorts: [] }),
|
||||
});
|
||||
const result = await reserveExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(result.status.state).toBe("failed");
|
||||
expect(result.handle).toBeNull();
|
||||
expect(calls.remove).toEqual([{ runtimeId: RUNTIME_ID, handle: HANDLE }]);
|
||||
});
|
||||
});
|
||||
|
||||
function deps(broker: BrokerClient, probeHealth: () => Promise<boolean>): ExposureManagerDeps {
|
||||
return { broker, probeHealth, now: () => "2026-08-11T00:00:00.000Z" };
|
||||
}
|
||||
|
||||
describe("provisionExposure loopback diagnosis (PAP-17256)", () => {
|
||||
it("fails before touching the broker when a listener is provably off-loopback", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
expose: () => ({ handle: HANDLE, publicPorts: [APP_PORT, HMR_PORT] }),
|
||||
});
|
||||
const seen: number[][] = [];
|
||||
const result = await provisionExposure(
|
||||
{
|
||||
...deps(broker, async () => true),
|
||||
diagnoseListenerBinds: async (ports) => {
|
||||
seen.push(ports);
|
||||
return `port ${APP_PORT} is bound to 0.0.0.0 instead of loopback only`;
|
||||
},
|
||||
},
|
||||
{ runtimeId: RUNTIME_ID, config: CONFIG, handle: HANDLE, hostname: HOSTNAME, appPort: APP_PORT },
|
||||
);
|
||||
|
||||
// Both the app port and its HMR companion are diagnosed, because the broker
|
||||
// gates on both and either one can be the wildcard bind.
|
||||
expect(seen).toEqual([[APP_PORT, HMR_PORT]]);
|
||||
expect(calls.expose).toHaveLength(0);
|
||||
expect(result.status.state).toBe("failed");
|
||||
// The persisted code stays the broker's own vocabulary so retry/cleanup
|
||||
// matching is unchanged; the reason rides alongside.
|
||||
expect(result.status.lastError).toBe("listener_ownership_mismatch");
|
||||
expect(result.errorDetail).toContain("0.0.0.0");
|
||||
// The reservation handle survives so the caller can still tear the lease down.
|
||||
expect(result.handle).toBe(HANDLE);
|
||||
});
|
||||
|
||||
it("exposes normally when the diagnosis finds nothing wrong", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
expose: () => ({ handle: HANDLE, publicPorts: [APP_PORT, HMR_PORT] }),
|
||||
});
|
||||
const result = await provisionExposure(
|
||||
{ ...deps(broker, async () => true), diagnoseListenerBinds: async () => null },
|
||||
{ runtimeId: RUNTIME_ID, config: CONFIG, handle: HANDLE, hostname: HOSTNAME, appPort: APP_PORT },
|
||||
);
|
||||
expect(calls.expose).toHaveLength(1);
|
||||
expect(result.status.state).toBe("ready");
|
||||
expect(result.errorDetail ?? null).toBeNull();
|
||||
});
|
||||
|
||||
it("surfaces the broker's own reason alongside its code", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
expose: () => {
|
||||
throw new BrokerClientError(
|
||||
"listener_ownership_mismatch",
|
||||
`listener on ${APP_PORT} is not loopback-only`,
|
||||
);
|
||||
},
|
||||
});
|
||||
const result = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(result.status.lastError).toBe("listener_ownership_mismatch");
|
||||
expect(result.errorDetail).toBe(`listener on ${APP_PORT} is not loopback-only`);
|
||||
});
|
||||
|
||||
it("does not repeat the code as a reason when the broker adds nothing", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
expose: () => {
|
||||
throw new BrokerClientError("cli_timeout", "cli_timeout");
|
||||
},
|
||||
});
|
||||
const result = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(result.status.lastError).toBe("cli_timeout");
|
||||
expect(result.errorDetail).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("provisionExposure", () => {
|
||||
it("exposes app + HMR and reports ready once the HTTPS probe validates", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
expose: () => ({ handle: "handle-abcdef1234567890", publicPorts: [APP_PORT, HMR_PORT] }),
|
||||
});
|
||||
const { status, handle } = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
|
||||
expect(calls.expose).toHaveLength(1);
|
||||
expect(calls.expose).toEqual([{ runtimeId: RUNTIME_ID, handle: HANDLE }]);
|
||||
expect(status.state).toBe("ready");
|
||||
expect(status.publicUrl).toBe(`https://${HOSTNAME}:${APP_PORT}`);
|
||||
expect(status.hostname).toBe(HOSTNAME);
|
||||
expect(status.listeners).toHaveLength(2);
|
||||
expect(status.brokerRef).toBe(RUNTIME_ID);
|
||||
expect(status.lastError).toBeNull();
|
||||
expect(handle).toBe("handle-abcdef1234567890");
|
||||
});
|
||||
|
||||
it("omits the HMR listener when HMR is not requested", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
expose: () => ({ handle: "handle-abcdef1234567890", publicPorts: [APP_PORT] }),
|
||||
});
|
||||
const { status } = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: { ...CONFIG, includePaperclipViteHmr: false },
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(calls.expose).toEqual([{ runtimeId: RUNTIME_ID, handle: HANDLE }]);
|
||||
expect(status.listeners).toEqual([{ purpose: "app", publicPort: APP_PORT, targetPort: APP_PORT }]);
|
||||
expect(status.state).toBe("ready");
|
||||
});
|
||||
|
||||
it("fails closed without calling the broker when the app port is out of range", async () => {
|
||||
const { broker, calls } = fakeBroker({});
|
||||
const { status, handle } = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: 3100, // primary app port — never eligible
|
||||
});
|
||||
expect(calls.expose).toHaveLength(0);
|
||||
expect(status.state).toBe("failed");
|
||||
expect(handle).toBe(HANDLE);
|
||||
});
|
||||
|
||||
it("fails when the broker rejects the expose request", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
expose: () => {
|
||||
throw new BrokerClientError("port_not_allowlisted", "denied");
|
||||
},
|
||||
});
|
||||
const { status, handle } = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(status.state).toBe("failed");
|
||||
expect(status.lastError).toBe("port_not_allowlisted");
|
||||
expect(status.listeners).toHaveLength(2);
|
||||
expect(handle).toBe(HANDLE);
|
||||
});
|
||||
|
||||
it("tears the mapping back down and fails when returned ports do not match", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
expose: () => ({ handle: "handle-abcdef1234567890", publicPorts: [APP_PORT] }), // missing HMR
|
||||
remove: () => ({ removedPorts: [APP_PORT] }),
|
||||
});
|
||||
const { status, handle } = await provisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(status.state).toBe("failed");
|
||||
expect(status.lastError).toMatch(/unexpected public ports/);
|
||||
expect(calls.remove).toHaveLength(1); // best-effort rollback
|
||||
expect(handle).toBe(HANDLE);
|
||||
});
|
||||
|
||||
it("fail-closed: keeps the handle but reports failed when the HTTPS probe does not validate", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
expose: () => ({ handle: "handle-abcdef1234567890", publicPorts: [APP_PORT, HMR_PORT] }),
|
||||
});
|
||||
const { status, handle } = await provisionExposure(deps(broker, async () => false), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
config: CONFIG,
|
||||
handle: HANDLE,
|
||||
hostname: HOSTNAME,
|
||||
appPort: APP_PORT,
|
||||
});
|
||||
expect(status.state).toBe("failed");
|
||||
expect(status.publicUrl).toBeNull();
|
||||
expect(status.listeners).toHaveLength(2); // attempted mapping still surfaced
|
||||
expect(status.lastError).toMatch(/health probe/);
|
||||
// Handle retained so the caller can retry or clean up the dangling mapping.
|
||||
expect(handle).toBe("handle-abcdef1234567890");
|
||||
});
|
||||
|
||||
it("fail-closed: treats a probe that throws as unhealthy", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
expose: () => ({ handle: "handle-abcdef1234567890", publicPorts: [APP_PORT, HMR_PORT] }),
|
||||
});
|
||||
const { status } = await provisionExposure(
|
||||
deps(broker, async () => {
|
||||
throw new Error("cert error");
|
||||
}),
|
||||
{ runtimeId: RUNTIME_ID, config: CONFIG, handle: HANDLE, hostname: HOSTNAME, appPort: APP_PORT },
|
||||
);
|
||||
expect(status.state).toBe("failed");
|
||||
});
|
||||
});
|
||||
|
||||
describe("deprovisionExposure", () => {
|
||||
it("is a no-op removal when no handle was ever recorded", async () => {
|
||||
const { broker, calls } = fakeBroker({});
|
||||
const { status, quarantinedPorts } = await deprovisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
handle: null,
|
||||
ports: [APP_PORT, HMR_PORT],
|
||||
});
|
||||
expect(calls.remove).toHaveLength(0);
|
||||
expect(status.state).toBe("removed");
|
||||
expect(quarantinedPorts).toEqual([]);
|
||||
});
|
||||
|
||||
it("removes cleanly with a handle", async () => {
|
||||
const { broker, calls } = fakeBroker({ remove: () => ({ removedPorts: [APP_PORT, HMR_PORT] }) });
|
||||
const { status, quarantinedPorts } = await deprovisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
handle: "handle-abcdef1234567890",
|
||||
ports: [APP_PORT, HMR_PORT],
|
||||
});
|
||||
expect(calls.remove).toHaveLength(1);
|
||||
expect(status.state).toBe("removed");
|
||||
expect(quarantinedPorts).toEqual([]);
|
||||
});
|
||||
|
||||
it("treats an already-gone mapping as removed (idempotent)", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
remove: () => {
|
||||
throw new BrokerClientError("invalid_handle", "gone");
|
||||
},
|
||||
});
|
||||
const { status, quarantinedPorts } = await deprovisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
handle: "handle-abcdef1234567890",
|
||||
ports: [APP_PORT, HMR_PORT],
|
||||
});
|
||||
expect(status.state).toBe("removed");
|
||||
expect(quarantinedPorts).toEqual([]);
|
||||
});
|
||||
|
||||
it("quarantines ports and reports cleanup_pending on an ambiguous cleanup failure", async () => {
|
||||
const { broker } = fakeBroker({
|
||||
remove: () => {
|
||||
throw new BrokerClientError("cli_error", "tailscale serve failed");
|
||||
},
|
||||
});
|
||||
const { status, quarantinedPorts } = await deprovisionExposure(deps(broker, async () => true), {
|
||||
runtimeId: RUNTIME_ID,
|
||||
handle: "handle-abcdef1234567890",
|
||||
ports: [APP_PORT, HMR_PORT, APP_PORT],
|
||||
});
|
||||
expect(status.state).toBe("cleanup_pending");
|
||||
expect(status.lastError).toBe("cli_error");
|
||||
expect(quarantinedPorts).toEqual([APP_PORT, HMR_PORT]); // deduped
|
||||
});
|
||||
});
|
||||
|
||||
describe("reconcileExposures", () => {
|
||||
it("adopts owned mappings that are still desired", async () => {
|
||||
const { broker, calls } = fakeBroker({
|
||||
list: () => [
|
||||
{ runtimeId: RUNTIME_ID, port: APP_PORT, purpose: "app" },
|
||||
{ runtimeId: RUNTIME_ID, port: HMR_PORT, purpose: "vite_hmr" },
|
||||
],
|
||||
});
|
||||
const result = await reconcileExposures(deps(broker, async () => true), {
|
||||
desiredRuntimeIds: new Set([RUNTIME_ID]),
|
||||
handlesByRuntimeId: new Map(),
|
||||
});
|
||||
expect(result.adopted).toEqual([RUNTIME_ID]);
|
||||
expect(result.removedOrphanPorts).toEqual([]);
|
||||
expect(calls.remove).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("removes an orphaned owned mapping when a handle is available", async () => {
|
||||
const orphan = "99999999-2222-4333-8444-555566667777";
|
||||
const { broker, calls } = fakeBroker({
|
||||
list: () => [{ runtimeId: orphan, port: APP_PORT, purpose: "app" }],
|
||||
remove: () => ({ removedPorts: [APP_PORT] }),
|
||||
});
|
||||
const result = await reconcileExposures(deps(broker, async () => true), {
|
||||
desiredRuntimeIds: new Set([RUNTIME_ID]),
|
||||
handlesByRuntimeId: new Map([[orphan, "handle-abcdef1234567890"]]),
|
||||
});
|
||||
expect(result.removedOrphanPorts).toEqual([APP_PORT]);
|
||||
expect(calls.remove).toEqual([{ runtimeId: orphan, handle: "handle-abcdef1234567890" }]);
|
||||
expect(result.unremovableOrphans).toEqual([]);
|
||||
});
|
||||
|
||||
it("never mutates an orphan it has no handle for (no remove call)", async () => {
|
||||
const orphan = "99999999-2222-4333-8444-555566667777";
|
||||
const { broker, calls } = fakeBroker({
|
||||
list: () => [{ runtimeId: orphan, port: APP_PORT, purpose: "app" }],
|
||||
});
|
||||
const result = await reconcileExposures(deps(broker, async () => true), {
|
||||
desiredRuntimeIds: new Set([RUNTIME_ID]),
|
||||
handlesByRuntimeId: new Map(),
|
||||
});
|
||||
expect(calls.remove).toHaveLength(0);
|
||||
expect(result.unremovableOrphans).toEqual([orphan]);
|
||||
expect(result.removedOrphanPorts).toEqual([]);
|
||||
});
|
||||
|
||||
it("records non-fatal errors without aborting reconciliation", async () => {
|
||||
const orphanA = "aaaaaaaa-2222-4333-8444-555566667777";
|
||||
const orphanB = "bbbbbbbb-2222-4333-8444-555566667777";
|
||||
const { broker } = fakeBroker({
|
||||
list: () => [
|
||||
{ runtimeId: orphanA, port: APP_PORT, purpose: "app" },
|
||||
{ runtimeId: orphanB, port: APP_PORT + 1, purpose: "app" },
|
||||
],
|
||||
remove: (runtimeId) => {
|
||||
if (runtimeId === orphanA) throw new BrokerClientError("cli_error", "boom");
|
||||
return { removedPorts: [APP_PORT + 1] };
|
||||
},
|
||||
});
|
||||
const result = await reconcileExposures(deps(broker, async () => true), {
|
||||
desiredRuntimeIds: new Set(),
|
||||
handlesByRuntimeId: new Map([
|
||||
[orphanA, "handle-aaaaaaaaaaaaaaaa"],
|
||||
[orphanB, "handle-bbbbbbbbbbbbbbbb"],
|
||||
]),
|
||||
});
|
||||
expect(result.errors).toEqual([{ runtimeId: orphanA, code: "cli_error" }]);
|
||||
expect(result.removedOrphanPorts).toEqual([APP_PORT + 1]);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,399 @@
|
|||
/**
|
||||
* Runtime exposure lifecycle orchestrator for the `tailscale_https` mode.
|
||||
*
|
||||
* Owns the exposure state machine independently of the backend process
|
||||
* lifecycle (PAP-17049 plan, PAP-17050 verdict). A backend can be running and
|
||||
* healthy while its HTTPS exposure is still `pending`, `failed`, or
|
||||
* `cleanup_pending`. This module never touches Tailscale directly — it drives
|
||||
* the least-privilege broker through an injected {@link BrokerClient} and an
|
||||
* injected external HTTPS health probe, so every branch is unit-testable
|
||||
* against a fake broker with no sockets, no CLI, and no certificates.
|
||||
*
|
||||
* Invariants enforced here (in addition to the broker's own controls):
|
||||
* - fail-closed: a configured HTTPS preview is never reported `ready` unless an
|
||||
* external, normally-validating HTTPS probe succeeds.
|
||||
* - same-number: only the app port and its derived HMR companion are exposed.
|
||||
* - idempotent stop/cleanup: a missing/already-removed lease is success, not an
|
||||
* error; genuine cleanup failures quarantine the ports and never reuse them.
|
||||
* - reconcile touches only Paperclip-owned mappings the broker reports; it can
|
||||
* never mutate an unknown/manual mapping or the primary `:443` route (the
|
||||
* broker's `list` never returns those, and remove requires an owned handle).
|
||||
*/
|
||||
import type { RuntimeExposureConfigInput, RuntimeExposureStatus } from "@paperclipai/shared";
|
||||
import {
|
||||
buildRuntimeExposureHealthUrl,
|
||||
buildRuntimeExposureUrl,
|
||||
deriveViteHmrPort,
|
||||
isRuntimeExposureAppPort,
|
||||
} from "@paperclipai/shared";
|
||||
|
||||
import {
|
||||
BrokerClientError,
|
||||
type BrokerClient,
|
||||
type BrokerListenerRequest,
|
||||
} from "./broker-client.js";
|
||||
|
||||
/** Injected seams — all side effects live behind these for testability. */
|
||||
export interface ExposureManagerDeps {
|
||||
broker: BrokerClient;
|
||||
/** External HTTPS probe; must reject/return false on cert or connection error. */
|
||||
probeHealth: (healthUrl: string) => Promise<boolean>;
|
||||
/** ISO timestamp source. Injected so tests are deterministic. */
|
||||
now: () => string;
|
||||
/**
|
||||
* Optional operator-facing diagnosis of the about-to-be-exposed listeners,
|
||||
* returning null when nothing is provably wrong (PAP-17256).
|
||||
*
|
||||
* This does not replace the broker's own loopback gate and cannot loosen it —
|
||||
* it runs first only so a wildcard bind fails with the port and address named
|
||||
* instead of a bare `listener_ownership_mismatch`.
|
||||
*/
|
||||
diagnoseListenerBinds?: (ports: number[]) => Promise<string | null>;
|
||||
}
|
||||
|
||||
export interface ProvisionInput {
|
||||
runtimeId: string;
|
||||
config: RuntimeExposureConfigInput;
|
||||
/** Broker-issued reservation handle persisted before the backend starts. */
|
||||
handle: string;
|
||||
/** Resolved Tailscale node DNS hostname (the manager does not resolve it). */
|
||||
hostname: string;
|
||||
/** Loopback app port already allocated in the dedicated range. */
|
||||
appPort: number;
|
||||
}
|
||||
|
||||
export interface ProvisionResult {
|
||||
status: RuntimeExposureStatus;
|
||||
/**
|
||||
* Server-only lease handle required to later remove the mapping. NEVER
|
||||
* serialized to the UI or embedded in {@link RuntimeExposureStatus}; persist
|
||||
* it in a server-private store. Null when nothing was exposed.
|
||||
*/
|
||||
handle: string | null;
|
||||
/**
|
||||
* Human-readable elaboration of {@link RuntimeExposureStatus.lastError}, when
|
||||
* one is available (PAP-17256).
|
||||
*
|
||||
* Deliberately NOT folded into the persisted status: `lastError` stays the
|
||||
* broker's stable machine code so the retry/cleanup vocabularies keep matching
|
||||
* on it. This rides alongside so the caller can put the reason in the failure
|
||||
* the operator actually reads.
|
||||
*/
|
||||
errorDetail?: string | null;
|
||||
}
|
||||
|
||||
export interface ReserveInput {
|
||||
runtimeId: string;
|
||||
config: RuntimeExposureConfigInput;
|
||||
appPort: number;
|
||||
}
|
||||
|
||||
/** Reserve the app/HMR pair before the backend binds either listener. */
|
||||
export async function reserveExposure(
|
||||
deps: ExposureManagerDeps,
|
||||
input: ReserveInput,
|
||||
): Promise<ProvisionResult> {
|
||||
const status = baseStatus(deps.now());
|
||||
if (!isRuntimeExposureAppPort(input.appPort)) {
|
||||
status.state = "failed";
|
||||
status.lastError = "app port outside dedicated runtime exposure range";
|
||||
return { status, handle: null };
|
||||
}
|
||||
const requested = buildListenerRequests(input.config, input.appPort);
|
||||
status.listeners = requested.map((listener) => ({
|
||||
purpose: listener.purpose,
|
||||
publicPort: listener.port,
|
||||
targetPort: listener.port,
|
||||
}));
|
||||
status.brokerRef = input.runtimeId;
|
||||
try {
|
||||
const result = await deps.broker.reserve(input.runtimeId, requested);
|
||||
const requestedPorts = new Set(requested.map((listener) => listener.port));
|
||||
const returnedPorts = new Set(result.reservedPorts);
|
||||
if (requestedPorts.size !== returnedPorts.size || ![...requestedPorts].every((port) => returnedPorts.has(port))) {
|
||||
await safeRemove(deps.broker, input.runtimeId, result.handle);
|
||||
status.state = "failed";
|
||||
status.lastError = "broker returned unexpected reserved ports";
|
||||
return { status, handle: null };
|
||||
}
|
||||
return { status, handle: result.handle };
|
||||
} catch (error) {
|
||||
status.state = "failed";
|
||||
status.lastError = brokerErrorCode(error);
|
||||
return { status, handle: null };
|
||||
}
|
||||
}
|
||||
|
||||
export interface DeprovisionInput {
|
||||
runtimeId: string;
|
||||
/** Server-persisted lease handle, or null if none was ever recorded. */
|
||||
handle: string | null;
|
||||
/** Ports previously exposed for this runtime (for quarantine on failure). */
|
||||
ports: number[];
|
||||
}
|
||||
|
||||
export interface DeprovisionResult {
|
||||
status: RuntimeExposureStatus;
|
||||
/** Ports that must be quarantined (never reused) after an ambiguous cleanup. */
|
||||
quarantinedPorts: number[];
|
||||
}
|
||||
|
||||
/** Broker error codes that mean "the mapping is already gone" — cleanup is a no-op. */
|
||||
const ALREADY_GONE_CODES: ReadonlySet<string> = new Set([
|
||||
"invalid_handle",
|
||||
"listener_not_owned",
|
||||
"listener_ownership_mismatch",
|
||||
]);
|
||||
|
||||
function baseStatus(now: string): RuntimeExposureStatus {
|
||||
return {
|
||||
provider: "tailscale_https",
|
||||
state: "pending",
|
||||
publicUrl: null,
|
||||
hostname: null,
|
||||
listeners: [],
|
||||
brokerRef: null,
|
||||
lastError: null,
|
||||
updatedAt: now,
|
||||
};
|
||||
}
|
||||
|
||||
/** Build the broker listener request list from a validated exposure config. */
|
||||
export function buildListenerRequests(
|
||||
config: RuntimeExposureConfigInput,
|
||||
appPort: number,
|
||||
): BrokerListenerRequest[] {
|
||||
const listeners: BrokerListenerRequest[] = [{ purpose: "app", port: appPort }];
|
||||
if (config.includePaperclipViteHmr) {
|
||||
listeners.push({ purpose: "vite_hmr", port: deriveViteHmrPort(appPort) });
|
||||
}
|
||||
return listeners;
|
||||
}
|
||||
|
||||
/**
|
||||
* Provision (or re-provision) the HTTPS exposure for one runtime service.
|
||||
* Idempotent at the broker level: a repeated call for a still-valid mapping
|
||||
* simply re-exposes and re-probes. Always returns a fully-formed status; the
|
||||
* caller persists both the status and the returned server-only handle.
|
||||
*/
|
||||
export async function provisionExposure(
|
||||
deps: ExposureManagerDeps,
|
||||
input: ProvisionInput,
|
||||
): Promise<ProvisionResult> {
|
||||
const now = deps.now();
|
||||
const status = baseStatus(now);
|
||||
status.hostname = input.hostname;
|
||||
|
||||
if (!isRuntimeExposureAppPort(input.appPort)) {
|
||||
status.state = "failed";
|
||||
status.lastError = "app port outside dedicated runtime exposure range";
|
||||
return { status, handle: input.handle };
|
||||
}
|
||||
|
||||
const requested = buildListenerRequests(input.config, input.appPort);
|
||||
status.listeners = requested.map((listener) => ({
|
||||
purpose: listener.purpose,
|
||||
publicPort: listener.port,
|
||||
targetPort: listener.port,
|
||||
}));
|
||||
status.brokerRef = input.runtimeId;
|
||||
|
||||
// Explain a provable off-loopback bind before the broker denies it, so the
|
||||
// failure names the port and the address instead of only the code.
|
||||
if (deps.diagnoseListenerBinds) {
|
||||
const violation = await deps.diagnoseListenerBinds(requested.map((listener) => listener.port));
|
||||
if (violation) {
|
||||
status.state = "failed";
|
||||
status.lastError = "listener_ownership_mismatch";
|
||||
return { status, handle: input.handle, errorDetail: violation };
|
||||
}
|
||||
}
|
||||
|
||||
let handle: string;
|
||||
let publicPorts: number[];
|
||||
try {
|
||||
const result = await deps.broker.expose(input.runtimeId, input.handle);
|
||||
handle = result.handle;
|
||||
publicPorts = result.publicPorts;
|
||||
} catch (error) {
|
||||
status.state = "failed";
|
||||
status.lastError = brokerErrorCode(error);
|
||||
return { status, handle: input.handle, errorDetail: brokerErrorDetail(error) };
|
||||
}
|
||||
|
||||
// The broker must return exactly the ports we asked for (same-number). A
|
||||
// mismatch is a contract violation: tear the mapping back down and fail.
|
||||
const requestedPorts = new Set(requested.map((l) => l.port));
|
||||
const returnedPorts = new Set(publicPorts);
|
||||
if (requestedPorts.size !== returnedPorts.size || ![...requestedPorts].every((p) => returnedPorts.has(p))) {
|
||||
await safeRemove(deps.broker, input.runtimeId, handle);
|
||||
status.state = "failed";
|
||||
status.lastError = "broker returned unexpected public ports";
|
||||
return { status, handle };
|
||||
}
|
||||
|
||||
// Reflect the attempted mapping regardless of health so the UI can show what
|
||||
// was provisioned even while probing.
|
||||
// fail-closed: never report ready until an external, cert-validating probe
|
||||
// succeeds. Keep the handle so the caller can retry or clean up.
|
||||
const healthUrl = buildRuntimeExposureHealthUrl(input.hostname, input.appPort);
|
||||
let healthy = false;
|
||||
try {
|
||||
healthy = await deps.probeHealth(healthUrl);
|
||||
} catch {
|
||||
healthy = false;
|
||||
}
|
||||
if (!healthy) {
|
||||
status.state = "failed";
|
||||
status.publicUrl = null;
|
||||
status.lastError = "external HTTPS health probe did not validate";
|
||||
return { status, handle };
|
||||
}
|
||||
|
||||
status.state = "ready";
|
||||
status.publicUrl = buildRuntimeExposureUrl(input.hostname, input.appPort);
|
||||
status.lastError = null;
|
||||
return { status, handle };
|
||||
}
|
||||
|
||||
/**
|
||||
* Tear down the HTTPS exposure for one runtime service. Idempotent: a null or
|
||||
* already-gone handle resolves to `removed` with no quarantine. A genuine
|
||||
* cleanup failure resolves to `cleanup_pending` and quarantines the ports.
|
||||
*/
|
||||
export async function deprovisionExposure(
|
||||
deps: ExposureManagerDeps,
|
||||
input: DeprovisionInput,
|
||||
): Promise<DeprovisionResult> {
|
||||
const now = deps.now();
|
||||
const status = baseStatus(now);
|
||||
status.state = "removed";
|
||||
|
||||
if (input.handle === null) {
|
||||
// Nothing the server knows to remove — treat as already cleaned up.
|
||||
return { status, quarantinedPorts: [] };
|
||||
}
|
||||
|
||||
try {
|
||||
await deps.broker.remove(input.runtimeId, input.handle);
|
||||
return { status, quarantinedPorts: [] };
|
||||
} catch (error) {
|
||||
const code = brokerErrorCode(error);
|
||||
if (ALREADY_GONE_CODES.has(code)) {
|
||||
// The mapping is already gone; cleanup is complete and safe.
|
||||
return { status, quarantinedPorts: [] };
|
||||
}
|
||||
// Ambiguous failure: we cannot prove the mapping is gone. Quarantine the
|
||||
// ports so they are never reused, and surface cleanup_pending.
|
||||
status.state = "cleanup_pending";
|
||||
status.lastError = code;
|
||||
return { status, quarantinedPorts: [...new Set(input.ports)] };
|
||||
}
|
||||
}
|
||||
|
||||
export interface ReconcileInput {
|
||||
/** Runtimes whose exposure SHOULD remain. Everything else owned is an orphan. */
|
||||
desiredRuntimeIds: ReadonlySet<string>;
|
||||
/** Every server-persisted lease handle, keyed by runtimeId (desired + orphaned). */
|
||||
handlesByRuntimeId: ReadonlyMap<string, string>;
|
||||
}
|
||||
|
||||
export interface ReconcileResult {
|
||||
/** Owned+desired runtimes confirmed present in the broker's mapping. */
|
||||
adopted: string[];
|
||||
/** Ports removed because their runtime is no longer desired. */
|
||||
removedOrphanPorts: number[];
|
||||
/**
|
||||
* Orphaned owned runtimes we could NOT remove because no handle was persisted.
|
||||
* Reported for operator/broker-side quarantine; the manager never fabricates a
|
||||
* handle and never touches a mapping it cannot prove ownership of.
|
||||
*/
|
||||
unremovableOrphans: string[];
|
||||
/** Non-fatal per-runtime removal errors during reconciliation. */
|
||||
errors: Array<{ runtimeId: string; code: string }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Startup / periodic reconciliation. Adopts owned mappings that are still
|
||||
* desired and removes owned mappings that are not. The broker's `list` only
|
||||
* ever returns Paperclip-owned listeners, so unknown/manual mappings and the
|
||||
* primary `:443` route are structurally out of scope here — they can never be
|
||||
* enumerated, adopted, or removed by this routine.
|
||||
*/
|
||||
export async function reconcileExposures(
|
||||
deps: ExposureManagerDeps,
|
||||
input: ReconcileInput,
|
||||
): Promise<ReconcileResult> {
|
||||
const result: ReconcileResult = {
|
||||
adopted: [],
|
||||
removedOrphanPorts: [],
|
||||
unremovableOrphans: [],
|
||||
errors: [],
|
||||
};
|
||||
|
||||
const owned = await deps.broker.list();
|
||||
|
||||
// Group owned ports by runtimeId.
|
||||
const portsByRuntime = new Map<string, number[]>();
|
||||
for (const listener of owned) {
|
||||
const ports = portsByRuntime.get(listener.runtimeId) ?? [];
|
||||
ports.push(listener.port);
|
||||
portsByRuntime.set(listener.runtimeId, ports);
|
||||
}
|
||||
|
||||
for (const [runtimeId, ports] of portsByRuntime) {
|
||||
if (input.desiredRuntimeIds.has(runtimeId)) {
|
||||
result.adopted.push(runtimeId);
|
||||
continue;
|
||||
}
|
||||
// Orphan: owned by us but no longer desired.
|
||||
const handle = input.handlesByRuntimeId.get(runtimeId);
|
||||
if (!handle) {
|
||||
result.unremovableOrphans.push(runtimeId);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const removed = await deps.broker.remove(runtimeId, handle);
|
||||
result.removedOrphanPorts.push(...removed.removedPorts);
|
||||
} catch (error) {
|
||||
const code = brokerErrorCode(error);
|
||||
if (ALREADY_GONE_CODES.has(code)) {
|
||||
// Already gone — reconciled, count its ports as removed for reporting.
|
||||
result.removedOrphanPorts.push(...ports);
|
||||
} else {
|
||||
result.errors.push({ runtimeId, code });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Best-effort remove used on rollback paths; swallows broker errors. */
|
||||
async function safeRemove(broker: BrokerClient, runtimeId: string, handle: string): Promise<void> {
|
||||
try {
|
||||
await broker.remove(runtimeId, handle);
|
||||
} catch {
|
||||
/* best effort — the outer flow has already decided to fail */
|
||||
}
|
||||
}
|
||||
|
||||
function brokerErrorCode(error: unknown): string {
|
||||
if (error instanceof BrokerClientError) return error.code;
|
||||
return "internal_error";
|
||||
}
|
||||
|
||||
/**
|
||||
* The broker's own explanation of a denial, when it adds anything to the code.
|
||||
*
|
||||
* The wire protocol already carries it (`BrokerClientError.message`, e.g.
|
||||
* "listener on 42003 is not loopback-only"); dropping it is what made
|
||||
* `listener_ownership_mismatch` unactionable in PAP-17254.
|
||||
*/
|
||||
function brokerErrorDetail(error: unknown): string | null {
|
||||
if (!(error instanceof BrokerClientError)) {
|
||||
return error instanceof Error ? error.message : null;
|
||||
}
|
||||
const message = error.message.trim();
|
||||
return message && message !== error.code ? message : null;
|
||||
}
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
import net from "node:net";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { RUNTIME_EXPOSURE_APP_PORT_MIN, deriveViteHmrPort } from "@paperclipai/shared";
|
||||
|
||||
import {
|
||||
diagnoseRuntimeListenerBinds,
|
||||
formatProcAddressHex,
|
||||
listenerBindFactsForPort,
|
||||
parseProcNetListeners,
|
||||
} from "./loopback-listener.js";
|
||||
|
||||
// Real header and row shape, copied from a live /proc/net/tcp{,6} on the host
|
||||
// that produced the PAP-17256 failures. Port 42003 is A40B; 52003 is CB23.
|
||||
const TCP_HEADER =
|
||||
" sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ";
|
||||
const TCP6_HEADER =
|
||||
" sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode";
|
||||
|
||||
function tcpRow(addrHex: string, portHex: string, state = "0A") {
|
||||
return ` 0: ${addrHex}:${portHex} 00000000:0000 ${state} 00000000:00000000 00:00000000 00000000 999 0 74815359 1 0000000000000000 100 0 0 10 0`;
|
||||
}
|
||||
|
||||
function tcp6Row(addrHex: string, portHex: string, state = "0A") {
|
||||
return ` 0: ${addrHex}:${portHex} 00000000000000000000000000000000:0000 ${state} 00000000:00000000 00:00000000 00000000 999 0 74815360 2 0000000000000000 100 0 0 10 0`;
|
||||
}
|
||||
|
||||
/** The app port from the PAP-17256 lane, and its `/proc` hex form. */
|
||||
const APP_PORT = 42_003;
|
||||
const portHex = (port: number) => port.toString(16).toUpperCase().padStart(4, "0");
|
||||
const APP_PORT_HEX = portHex(APP_PORT);
|
||||
|
||||
describe("formatProcAddressHex", () => {
|
||||
it("byte-swaps IPv4 words", () => {
|
||||
expect(formatProcAddressHex("0100007F")).toBe("127.0.0.1");
|
||||
expect(formatProcAddressHex("00000000")).toBe("0.0.0.0");
|
||||
expect(formatProcAddressHex("0400007F")).toBe("127.0.0.4");
|
||||
});
|
||||
|
||||
it("renders the IPv6 loopback and wildcard forms /proc actually stores", () => {
|
||||
expect(formatProcAddressHex("00000000000000000000000001000000")).toBe("::1");
|
||||
expect(formatProcAddressHex("00000000000000000000000000000000")).toBe("::");
|
||||
});
|
||||
});
|
||||
|
||||
describe("listenerBindFactsForPort", () => {
|
||||
it("accepts an IPv4 loopback listener", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
parseProcNetListeners(`${TCP_HEADER}\n${tcpRow("0100007F", APP_PORT_HEX)}\n`),
|
||||
[],
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts).toEqual({ present: true, loopbackOnly: true, addresses: ["127.0.0.1"] });
|
||||
});
|
||||
|
||||
it("accepts an IPv6 loopback listener in the little-endian ::1 form", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
[],
|
||||
parseProcNetListeners(
|
||||
`${TCP6_HEADER}\n${tcp6Row("00000000000000000000000001000000", APP_PORT_HEX)}\n`,
|
||||
),
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts).toEqual({ present: true, loopbackOnly: true, addresses: ["::1"] });
|
||||
});
|
||||
|
||||
it("rejects the 0.0.0.0 wildcard bind that broke every managed lane", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
parseProcNetListeners(`${TCP_HEADER}\n${tcpRow("00000000", APP_PORT_HEX)}\n`),
|
||||
[],
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts).toEqual({ present: true, loopbackOnly: false, addresses: ["0.0.0.0"] });
|
||||
});
|
||||
|
||||
it("rejects the :: wildcard bind", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
[],
|
||||
parseProcNetListeners(
|
||||
`${TCP6_HEADER}\n${tcp6Row("00000000000000000000000000000000", APP_PORT_HEX)}\n`,
|
||||
),
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts.loopbackOnly).toBe(false);
|
||||
expect(facts.addresses).toEqual(["::"]);
|
||||
});
|
||||
|
||||
it("rejects a non-loopback unicast bind", () => {
|
||||
// 100.123.243.20 — the tailnet address, stored little-endian.
|
||||
const facts = listenerBindFactsForPort(
|
||||
parseProcNetListeners(`${TCP_HEADER}\n${tcpRow("14F37B64", APP_PORT_HEX)}\n`),
|
||||
[],
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts.loopbackOnly).toBe(false);
|
||||
expect(facts.addresses).toEqual(["100.123.243.20"]);
|
||||
});
|
||||
|
||||
it("reports absent for a port with no LISTEN row", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
// Same port, but ESTABLISHED (01) rather than LISTEN (0A).
|
||||
parseProcNetListeners(`${TCP_HEADER}\n${tcpRow("0100007F", APP_PORT_HEX, "01")}\n`),
|
||||
[],
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts).toEqual({ present: false, loopbackOnly: true, addresses: [] });
|
||||
});
|
||||
|
||||
it("ignores rows for other ports", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
parseProcNetListeners(`${TCP_HEADER}\n${tcpRow("00000000", portHex(52_003))}\n`),
|
||||
[],
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts.present).toBe(false);
|
||||
});
|
||||
|
||||
it("is not loopback-only when a wildcard row accompanies a loopback row", () => {
|
||||
const facts = listenerBindFactsForPort(
|
||||
parseProcNetListeners(
|
||||
`${TCP_HEADER}\n${tcpRow("0100007F", APP_PORT_HEX)}\n${tcpRow("00000000", APP_PORT_HEX)}\n`,
|
||||
),
|
||||
[],
|
||||
APP_PORT,
|
||||
);
|
||||
expect(facts.loopbackOnly).toBe(false);
|
||||
expect(facts.addresses).toEqual(["127.0.0.1", "0.0.0.0"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("diagnoseRuntimeListenerBinds against live listeners", () => {
|
||||
const appPort = RUNTIME_EXPOSURE_APP_PORT_MIN + 900;
|
||||
const hmrPort = deriveViteHmrPort(appPort);
|
||||
|
||||
async function withListener<T>(
|
||||
port: number,
|
||||
host: string | undefined,
|
||||
body: () => Promise<T>,
|
||||
): Promise<T> {
|
||||
const server = net.createServer();
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
if (host === undefined) server.listen(port, () => resolve());
|
||||
else server.listen(port, host, () => resolve());
|
||||
});
|
||||
try {
|
||||
return await body();
|
||||
} finally {
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
}
|
||||
}
|
||||
|
||||
it("stays silent for a real loopback listener", async () => {
|
||||
await withListener(appPort, "127.0.0.1", async () => {
|
||||
expect(await diagnoseRuntimeListenerBinds([appPort])).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
it("names the port and the wildcard address for a real 0.0.0.0 listener", async () => {
|
||||
await withListener(appPort, undefined, async () => {
|
||||
const diagnosis = await diagnoseRuntimeListenerBinds([appPort]);
|
||||
expect(diagnosis).toContain(`port ${appPort}`);
|
||||
// Node's hostless listen is dual-stack, so /proc shows :: and/or 0.0.0.0.
|
||||
expect(diagnosis).toMatch(/0\.0\.0\.0|::/);
|
||||
expect(diagnosis).toContain("--bind custom --bind-host 127.0.0.1");
|
||||
});
|
||||
});
|
||||
|
||||
it("catches the HMR companion port too, not just the app port", async () => {
|
||||
await withListener(appPort, "127.0.0.1", async () => {
|
||||
await withListener(hmrPort, undefined, async () => {
|
||||
const diagnosis = await diagnoseRuntimeListenerBinds([appPort, hmrPort]);
|
||||
expect(diagnosis).toContain(`port ${hmrPort}`);
|
||||
expect(diagnosis).not.toContain(`port ${appPort} is bound`);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("stays silent for a port with no listener, leaving the verdict to the broker", async () => {
|
||||
expect(await diagnoseRuntimeListenerBinds([appPort])).toBeNull();
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
/**
|
||||
* Server-side loopback listener diagnosis for the `tailscale_https` mode
|
||||
* (PAP-17256).
|
||||
*
|
||||
* The broker already refuses to expose a port whose listener is not
|
||||
* loopback-only, and it must keep doing that check itself — it cannot trust the
|
||||
* caller. But the broker only answers with a stable machine code, so when a
|
||||
* managed lane bound `0.0.0.0` the operator saw a bare
|
||||
* `listener_ownership_mismatch` with no port, no address, and no hint about
|
||||
* which side was wrong. That opacity, not the denial, is what turned one bug
|
||||
* into three diagnostic cycles.
|
||||
*
|
||||
* So the server runs the same read *before* it calls the broker, purely to
|
||||
* produce a sentence a human can act on. This is deliberately a duplicate of
|
||||
* the broker's `proc-listener` logic rather than a shared import: the broker is
|
||||
* host-privileged and its gate must stay independent of anything the server
|
||||
* says. If the two ever disagree, the broker still wins — this one only
|
||||
* explains.
|
||||
*
|
||||
* Parsing helpers are exported pure so they can be tested without /proc.
|
||||
*/
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
/** One listening row from `/proc/net/tcp` or `/proc/net/tcp6`. */
|
||||
export interface ProcListenerRow {
|
||||
localAddressHex: string;
|
||||
localPortHex: string;
|
||||
state: string;
|
||||
}
|
||||
|
||||
export interface ListenerBindFacts {
|
||||
present: boolean;
|
||||
loopbackOnly: boolean;
|
||||
/** Bound addresses in human-readable form, for the diagnosis message. */
|
||||
addresses: string[];
|
||||
}
|
||||
|
||||
const TCP_STATE_LISTEN = "0A";
|
||||
|
||||
/** Parse one `/proc/net/tcp{,6}` table into its listening rows. */
|
||||
export function parseProcNetListeners(content: string): ProcListenerRow[] {
|
||||
const rows: ProcListenerRow[] = [];
|
||||
for (const line of content.split("\n").slice(1)) {
|
||||
const cols = line.trim().split(/\s+/);
|
||||
if (cols.length < 10) continue;
|
||||
const [addr, port] = cols[1].split(":");
|
||||
if (!addr || !port) continue;
|
||||
rows.push({ localAddressHex: addr, localPortHex: port, state: cols[3] });
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
/** True for a wildcard bind (`0.0.0.0` / `::`), reachable off-loopback. */
|
||||
function isWildcardHex(addrHex: string): boolean {
|
||||
return /^0+$/.test(addrHex);
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a `/proc` address for humans. Both tables store 32-bit words in host
|
||||
* (little-endian) byte order, so each 8-hex-digit word is byte-swapped.
|
||||
*/
|
||||
export function formatProcAddressHex(addrHex: string): string {
|
||||
if (addrHex.length === 8) {
|
||||
const octets = [6, 4, 2, 0].map((offset) => parseInt(addrHex.slice(offset, offset + 2), 16));
|
||||
return octets.join(".");
|
||||
}
|
||||
if (addrHex.length === 32) {
|
||||
const bytes: number[] = [];
|
||||
for (let word = 0; word < 4; word += 1) {
|
||||
const hex = addrHex.slice(word * 8, word * 8 + 8);
|
||||
for (let offset = 6; offset >= 0; offset -= 2) {
|
||||
bytes.push(parseInt(hex.slice(offset, offset + 2), 16));
|
||||
}
|
||||
}
|
||||
if (bytes.every((byte) => byte === 0)) return "::";
|
||||
if (bytes.slice(0, 15).every((byte) => byte === 0) && bytes[15] === 1) return "::1";
|
||||
const groups: string[] = [];
|
||||
for (let index = 0; index < 16; index += 2) {
|
||||
groups.push(((bytes[index] << 8) | bytes[index + 1]).toString(16));
|
||||
}
|
||||
return groups.join(":");
|
||||
}
|
||||
return `0x${addrHex}`;
|
||||
}
|
||||
|
||||
/** True when a `/proc` IPv4 hex address is in 127.0.0.0/8. */
|
||||
function isLoopbackIpv4Hex(addrHex: string): boolean {
|
||||
if (addrHex.length !== 8) return false;
|
||||
return parseInt(addrHex.slice(6, 8), 16) === 127;
|
||||
}
|
||||
|
||||
/** `::1` as `/proc/net/tcp6` stores it — final word byte-swapped. */
|
||||
const LOOPBACK_IPV6_PROC_HEX = "00000000000000000000000001000000";
|
||||
|
||||
function isLoopbackIpv6Hex(addrHex: string): boolean {
|
||||
return addrHex.toUpperCase() === LOOPBACK_IPV6_PROC_HEX;
|
||||
}
|
||||
|
||||
/** Reduce both `/proc` tables to bind facts for one port. */
|
||||
export function listenerBindFactsForPort(
|
||||
tcp: ProcListenerRow[],
|
||||
tcp6: ProcListenerRow[],
|
||||
port: number,
|
||||
): ListenerBindFacts {
|
||||
const wantHex = port.toString(16).toUpperCase().padStart(4, "0");
|
||||
const addresses: string[] = [];
|
||||
let present = false;
|
||||
let loopbackOnly = true;
|
||||
const scan = (rows: ProcListenerRow[], isLoopback: (hex: string) => boolean) => {
|
||||
for (const row of rows) {
|
||||
if (row.localPortHex.toUpperCase() !== wantHex || row.state !== TCP_STATE_LISTEN) continue;
|
||||
present = true;
|
||||
addresses.push(formatProcAddressHex(row.localAddressHex));
|
||||
if (isWildcardHex(row.localAddressHex) || !isLoopback(row.localAddressHex)) {
|
||||
loopbackOnly = false;
|
||||
}
|
||||
}
|
||||
};
|
||||
scan(tcp, isLoopbackIpv4Hex);
|
||||
scan(tcp6, isLoopbackIpv6Hex);
|
||||
return { present, loopbackOnly, addresses: [...new Set(addresses)] };
|
||||
}
|
||||
|
||||
/** Read live bind facts for one port. Unreadable `/proc` yields "unknown". */
|
||||
export async function readListenerBindFacts(port: number): Promise<ListenerBindFacts | null> {
|
||||
let tcp: ProcListenerRow[];
|
||||
try {
|
||||
tcp = parseProcNetListeners(await readFile("/proc/net/tcp", "utf8"));
|
||||
} catch {
|
||||
// No /proc (non-Linux, or a restricted sandbox): the server simply cannot
|
||||
// explain, so it stays silent and lets the broker's own gate decide.
|
||||
return null;
|
||||
}
|
||||
let tcp6: ProcListenerRow[] = [];
|
||||
try {
|
||||
tcp6 = parseProcNetListeners(await readFile("/proc/net/tcp6", "utf8"));
|
||||
} catch {
|
||||
/* optional table */
|
||||
}
|
||||
return listenerBindFactsForPort(tcp, tcp6, port);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the operator-facing diagnosis for a set of ports about to be exposed, or
|
||||
* null when nothing is provably wrong.
|
||||
*
|
||||
* Only a *proven* violation is reported. A missing listener is not treated as a
|
||||
* violation here: readiness already gates on the bind, and guessing would turn a
|
||||
* benign race into a misleading error.
|
||||
*/
|
||||
export async function diagnoseRuntimeListenerBinds(ports: number[]): Promise<string | null> {
|
||||
const violations: string[] = [];
|
||||
for (const port of ports) {
|
||||
const facts = await readListenerBindFacts(port);
|
||||
if (!facts || !facts.present || facts.loopbackOnly) continue;
|
||||
violations.push(`port ${port} is bound to ${facts.addresses.join(", ")}`);
|
||||
}
|
||||
if (violations.length === 0) return null;
|
||||
return (
|
||||
`${violations.join("; ")} instead of loopback only. The broker will not expose a listener `
|
||||
+ "reachable off-loopback. This means the workspace checkout's dev server ignored the managed "
|
||||
+ "loopback bind — a checkout that predates managed HTTPS exposure overwrites PAPERCLIP_BIND "
|
||||
+ "from its own --bind argv, so the start command must pass --bind custom --bind-host 127.0.0.1."
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,109 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
RUNTIME_EXPOSURE_APP_PORT_MIN,
|
||||
RUNTIME_EXPOSURE_HMR_PORT_OFFSET,
|
||||
} from "@paperclipai/shared";
|
||||
|
||||
import { allocateExposurePortPair } from "./port-pair.js";
|
||||
|
||||
describe("allocateExposurePortPair", () => {
|
||||
it("returns the first free app port with a free HMR companion", async () => {
|
||||
const pair = await allocateExposurePortPair({ isPortAvailable: async () => true });
|
||||
expect(pair.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN);
|
||||
expect(pair.hmrPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN + RUNTIME_EXPOSURE_HMR_PORT_OFFSET);
|
||||
});
|
||||
|
||||
it("skips an app port whose HMR companion is busy", async () => {
|
||||
const busyHmr = RUNTIME_EXPOSURE_APP_PORT_MIN + RUNTIME_EXPOSURE_HMR_PORT_OFFSET;
|
||||
const pair = await allocateExposurePortPair({
|
||||
isPortAvailable: async (port) => port !== busyHmr,
|
||||
});
|
||||
// First app port's companion is busy, so it advances to the next app port.
|
||||
expect(pair.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN + 1);
|
||||
expect(pair.hmrPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN + 1 + RUNTIME_EXPOSURE_HMR_PORT_OFFSET);
|
||||
});
|
||||
|
||||
it("skips a busy app port", async () => {
|
||||
const pair = await allocateExposurePortPair({
|
||||
isPortAvailable: async (port) => port !== RUNTIME_EXPOSURE_APP_PORT_MIN,
|
||||
});
|
||||
expect(pair.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN + 1);
|
||||
});
|
||||
|
||||
it("never returns a reserved (e.g. quarantined) app or HMR port", async () => {
|
||||
const reserved = new Set<number>([
|
||||
RUNTIME_EXPOSURE_APP_PORT_MIN,
|
||||
// reserve the SECOND app port's HMR companion too
|
||||
RUNTIME_EXPOSURE_APP_PORT_MIN + 1 + RUNTIME_EXPOSURE_HMR_PORT_OFFSET,
|
||||
]);
|
||||
const pair = await allocateExposurePortPair({
|
||||
isPortAvailable: async () => true,
|
||||
reserved,
|
||||
});
|
||||
expect(pair.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN + 2);
|
||||
});
|
||||
|
||||
describe("preferredAppPort (keep existing runtime ports when safe)", () => {
|
||||
it("keeps a preferred in-range port instead of restarting the scan", async () => {
|
||||
const pair = await allocateExposurePortPair({
|
||||
isPortAvailable: async () => true,
|
||||
preferredAppPort: 42_500,
|
||||
});
|
||||
expect(pair.appPort).toBe(42_500);
|
||||
expect(pair.hmrPort).toBe(42_500 + RUNTIME_EXPOSURE_HMR_PORT_OFFSET);
|
||||
});
|
||||
|
||||
it("ignores a legacy pinned port outside the dedicated range", async () => {
|
||||
// 45439 is the pre-feature Paperclip App port; the broker can never
|
||||
// publish it, so the allocator must relocate rather than fail.
|
||||
const pair = await allocateExposurePortPair({
|
||||
isPortAvailable: async () => true,
|
||||
preferredAppPort: 45_439,
|
||||
});
|
||||
expect(pair.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN);
|
||||
});
|
||||
|
||||
it("falls back to the scan when the preferred port is busy or reserved", async () => {
|
||||
const busy = await allocateExposurePortPair({
|
||||
isPortAvailable: async (port) => port !== 42_500,
|
||||
preferredAppPort: 42_500,
|
||||
});
|
||||
expect(busy.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN);
|
||||
|
||||
const reservedCompanion = await allocateExposurePortPair({
|
||||
isPortAvailable: async () => true,
|
||||
reserved: new Set([42_500 + RUNTIME_EXPOSURE_HMR_PORT_OFFSET]),
|
||||
preferredAppPort: 42_500,
|
||||
});
|
||||
expect(reservedCompanion.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN);
|
||||
});
|
||||
|
||||
it("ignores a preferred HMR-range port (never an app port)", async () => {
|
||||
const pair = await allocateExposurePortPair({
|
||||
isPortAvailable: async () => true,
|
||||
preferredAppPort: RUNTIME_EXPOSURE_APP_PORT_MIN + RUNTIME_EXPOSURE_HMR_PORT_OFFSET,
|
||||
});
|
||||
expect(pair.appPort).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN);
|
||||
});
|
||||
});
|
||||
|
||||
it("throws when the dedicated range is exhausted", async () => {
|
||||
await expect(
|
||||
allocateExposurePortPair({ isPortAvailable: async () => false }),
|
||||
).rejects.toThrow(/no free app\/HMR port pair/);
|
||||
});
|
||||
|
||||
it("only probes the companion after the app port is free (short-circuits)", async () => {
|
||||
const probed: number[] = [];
|
||||
await allocateExposurePortPair({
|
||||
isPortAvailable: async (port) => {
|
||||
probed.push(port);
|
||||
return true;
|
||||
},
|
||||
});
|
||||
// First two probes are the first app port and its companion, in order.
|
||||
expect(probed[0]).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN);
|
||||
expect(probed[1]).toBe(RUNTIME_EXPOSURE_APP_PORT_MIN + RUNTIME_EXPOSURE_HMR_PORT_OFFSET);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,87 @@
|
|||
/**
|
||||
* Paired app + Vite HMR port allocation for the `tailscale_https` exposure mode.
|
||||
*
|
||||
* A managed runtime that opts into HTTPS exposure needs TWO deterministically
|
||||
* related loopback ports reserved together (PAP-17049 plan, PAP-17050 verdict
|
||||
* requirement #2): the app port and its Paperclip Vite HMR companion at a fixed
|
||||
* offset. Allocating them as a pair — and only from the dedicated allowlisted
|
||||
* range — means a compromised caller can never ask the broker to publish an
|
||||
* arbitrary existing loopback service, and the HMR listener is never orphaned
|
||||
* from its app listener.
|
||||
*
|
||||
* Pure orchestration over an injected availability probe: no sockets here, so
|
||||
* the scan order and skip logic are unit-testable.
|
||||
*/
|
||||
import {
|
||||
RUNTIME_EXPOSURE_APP_PORT_MIN,
|
||||
RUNTIME_EXPOSURE_APP_PORT_MAX,
|
||||
deriveViteHmrPort,
|
||||
isRuntimeExposureAppPort,
|
||||
} from "@paperclipai/shared";
|
||||
|
||||
export interface ExposurePortPair {
|
||||
appPort: number;
|
||||
hmrPort: number;
|
||||
}
|
||||
|
||||
export interface AllocateExposurePortPairInput {
|
||||
/**
|
||||
* Returns true if the port can currently be bound loopback-only. Both the app
|
||||
* port and its HMR companion must pass before a pair is returned.
|
||||
*/
|
||||
isPortAvailable: (port: number) => Promise<boolean>;
|
||||
/**
|
||||
* Ports that must never be handed out — e.g. quarantined after an ambiguous
|
||||
* cleanup, or already reserved by other live runtimes this cycle.
|
||||
*/
|
||||
reserved?: ReadonlySet<number>;
|
||||
/**
|
||||
* The port this runtime is already using, if any. Tried first so a restart or
|
||||
* a backfilled service keeps its port instead of drifting across the range on
|
||||
* every deploy ("keep existing runtime ports when safe", PAP-17158).
|
||||
*
|
||||
* Only honored when it is genuinely safe: the port must be an allowlisted app
|
||||
* port, unreserved, and free together with its HMR companion. A legacy pinned
|
||||
* port outside the dedicated range (the Paperclip App template's 45439) can
|
||||
* never be published by the broker, so it is ignored here rather than making
|
||||
* the caller special-case it.
|
||||
*/
|
||||
preferredAppPort?: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return an app port whose HMR companion is also free and unreserved: the
|
||||
* preferred port when it is eligible, otherwise the first such port scanning the
|
||||
* dedicated range in ascending order. Throws when the range is exhausted so the
|
||||
* caller fails closed rather than binding an out-of-range port.
|
||||
*/
|
||||
export async function allocateExposurePortPair(
|
||||
input: AllocateExposurePortPairInput,
|
||||
): Promise<ExposurePortPair> {
|
||||
const reserved = input.reserved ?? new Set<number>();
|
||||
|
||||
const claimIfFree = async (appPort: number): Promise<ExposurePortPair | null> => {
|
||||
if (reserved.has(appPort)) return null;
|
||||
if (!isRuntimeExposureAppPort(appPort)) return null;
|
||||
const hmrPort = deriveViteHmrPort(appPort);
|
||||
if (reserved.has(hmrPort)) return null;
|
||||
// Probe the app port first; short-circuit before probing the companion.
|
||||
if (!(await input.isPortAvailable(appPort))) return null;
|
||||
if (!(await input.isPortAvailable(hmrPort))) return null;
|
||||
return { appPort, hmrPort };
|
||||
};
|
||||
|
||||
if (input.preferredAppPort != null) {
|
||||
const preferred = await claimIfFree(input.preferredAppPort);
|
||||
if (preferred) return preferred;
|
||||
}
|
||||
|
||||
for (let appPort = RUNTIME_EXPOSURE_APP_PORT_MIN; appPort <= RUNTIME_EXPOSURE_APP_PORT_MAX; appPort += 1) {
|
||||
const pair = await claimIfFree(appPort);
|
||||
if (pair) return pair;
|
||||
}
|
||||
throw new Error(
|
||||
`no free app/HMR port pair available in the dedicated runtime exposure range ` +
|
||||
`[${RUNTIME_EXPOSURE_APP_PORT_MIN}, ${RUNTIME_EXPOSURE_APP_PORT_MAX}]`,
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { parseTailscaleDnsName } from "./tailscale-hostname.js";
|
||||
|
||||
describe("parseTailscaleDnsName", () => {
|
||||
it("normalizes the local node MagicDNS hostname", () => {
|
||||
expect(parseTailscaleDnsName({ Self: { DNSName: "Branch-Runner.tail123.ts.net." } }))
|
||||
.toBe("branch-runner.tail123.ts.net");
|
||||
});
|
||||
|
||||
it("rejects missing, single-label, and injected hostnames", () => {
|
||||
expect(() => parseTailscaleDnsName({})).toThrow(/Self\.DNSName/);
|
||||
expect(() => parseTailscaleDnsName({ Self: { DNSName: "localhost" } })).toThrow(/invalid/);
|
||||
expect(() => parseTailscaleDnsName({ Self: { DNSName: "host/evil.ts.net" } })).toThrow(/invalid/);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
import { execFile } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
const DEFAULT_TAILSCALE_BIN = "/usr/bin/tailscale";
|
||||
|
||||
type TailscaleStatus = {
|
||||
Self?: { DNSName?: unknown };
|
||||
};
|
||||
|
||||
/** Parse only this node's MagicDNS name from `tailscale status --json`. */
|
||||
export function parseTailscaleDnsName(value: unknown): string {
|
||||
const dnsName = (value as TailscaleStatus | null)?.Self?.DNSName;
|
||||
if (typeof dnsName !== "string") {
|
||||
throw new Error("tailscale status did not include Self.DNSName");
|
||||
}
|
||||
const normalized = dnsName.trim().replace(/\.$/, "").toLowerCase();
|
||||
if (!/^[a-z0-9](?:[a-z0-9.-]{0,251}[a-z0-9])?$/.test(normalized) || !normalized.includes(".")) {
|
||||
throw new Error("tailscale status returned an invalid Self.DNSName");
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only hostname discovery. This does not require Tailscale operator
|
||||
* authority; all Serve mutations remain isolated in the host broker.
|
||||
*/
|
||||
export async function resolveTailscaleDnsName(input?: {
|
||||
tailscaleBinPath?: string;
|
||||
timeoutMs?: number;
|
||||
}): Promise<string> {
|
||||
const configured = process.env.PAPERCLIP_TAILSCALE_DNS_NAME?.trim();
|
||||
if (configured) return parseTailscaleDnsName({ Self: { DNSName: configured } });
|
||||
|
||||
const tailscaleBinPath = input?.tailscaleBinPath ?? DEFAULT_TAILSCALE_BIN;
|
||||
if (!tailscaleBinPath.startsWith("/")) {
|
||||
throw new Error("tailscale binary path must be absolute");
|
||||
}
|
||||
const { stdout } = await execFileAsync(tailscaleBinPath, ["status", "--json"], {
|
||||
encoding: "utf8",
|
||||
timeout: input?.timeoutMs ?? 3_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
});
|
||||
return parseTailscaleDnsName(JSON.parse(stdout));
|
||||
}
|
||||
|
|
@ -2,14 +2,184 @@ import { randomUUID } from "node:crypto";
|
|||
import type { Db } from "@paperclipai/db";
|
||||
import { workspaceOperations } from "@paperclipai/db";
|
||||
import type { WorkspaceOperation, WorkspaceOperationPhase, WorkspaceOperationStatus } from "@paperclipai/shared";
|
||||
import { asc, desc, eq, inArray, isNull, or, and } from "drizzle-orm";
|
||||
import { notFound } from "../errors.js";
|
||||
import { asc, desc, eq, gte, inArray, isNull, lt, or, and } from "drizzle-orm";
|
||||
import { conflict, notFound } from "../errors.js";
|
||||
import { redactCurrentUserText, redactCurrentUserValue } from "../log-redaction.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
import { getWorkspaceOperationLogStore } from "./workspace-operation-log-store.js";
|
||||
|
||||
type WorkspaceOperationRow = typeof workspaceOperations.$inferSelect;
|
||||
|
||||
/**
|
||||
* Managed runtime control actions. Every one of these mutates the runtime rows and
|
||||
* local listeners of a single execution workspace, so only one may be live at a time.
|
||||
*/
|
||||
const RUNTIME_CONTROL_ACTIONS = new Set(["start", "stop", "restart", "run"]);
|
||||
|
||||
/**
|
||||
* Identity of this server process. A `running` runtime-control operation stamped with a
|
||||
* different owner id can only have been left behind by another (now gone) process, which
|
||||
* is what makes bounded recovery safe: we never guess about our own live operations.
|
||||
*/
|
||||
const RUNTIME_CONTROL_OWNER_ID = randomUUID();
|
||||
|
||||
/** How often a live operation refreshes its ownership stamp. */
|
||||
export const RUNTIME_CONTROL_HEARTBEAT_MS = 10_000;
|
||||
|
||||
/**
|
||||
* How long an operation may go without a heartbeat before another owner may terminalize
|
||||
* it. Deliberately several heartbeat intervals so a slow-but-live start is never stolen.
|
||||
*/
|
||||
export const RUNTIME_CONTROL_STALE_AFTER_MS = 60_000;
|
||||
|
||||
/**
|
||||
* Last-resort ceiling for a managed runtime lifecycle control. Past this the operation is
|
||||
* failed by its own owner so it always reaches a terminal state, even when the underlying
|
||||
* start hangs on an unresponsive process or provider that keeps the owner alive. Generous
|
||||
* enough to cover a cold runtime provision (a full dependency install) on a large repo.
|
||||
*/
|
||||
export const RUNTIME_CONTROL_MAX_DURATION_MS = 30 * 60_000;
|
||||
|
||||
/**
|
||||
* Workspace jobs are operator-authored commands (builds, migrations, test suites) that can
|
||||
* legitimately run far longer than a lifecycle control, so they get a much wider ceiling —
|
||||
* still finite, so a wedged job cannot hold the workspace forever.
|
||||
*/
|
||||
export const WORKSPACE_JOB_MAX_DURATION_MS = 4 * 60 * 60_000;
|
||||
|
||||
function defaultRuntimeControlTimeoutMs(action: string | null) {
|
||||
if (!action) return null;
|
||||
return action === "run" ? WORKSPACE_JOB_MAX_DURATION_MS : RUNTIME_CONTROL_MAX_DURATION_MS;
|
||||
}
|
||||
|
||||
export class WorkspaceOperationTimeoutError extends Error {
|
||||
constructor(public readonly timeoutMs: number, action: string | null) {
|
||||
super(
|
||||
`Managed workspace ${action ?? "runtime"} operation exceeded its ${Math.round(timeoutMs / 1000)}s time budget and was failed so it can be retried or stopped.`,
|
||||
);
|
||||
this.name = "WorkspaceOperationTimeoutError";
|
||||
}
|
||||
}
|
||||
|
||||
type RuntimeControlOwnerStamp = {
|
||||
ownerId: string;
|
||||
pid: number;
|
||||
action: string;
|
||||
heartbeatAt: string;
|
||||
};
|
||||
|
||||
function readRuntimeControlAction(metadata: unknown): string | null {
|
||||
if (!metadata || typeof metadata !== "object") return null;
|
||||
const action = (metadata as Record<string, unknown>).action;
|
||||
if (typeof action !== "string" || !RUNTIME_CONTROL_ACTIONS.has(action)) return null;
|
||||
return action;
|
||||
}
|
||||
|
||||
function readRuntimeControlOwner(metadata: unknown): RuntimeControlOwnerStamp | null {
|
||||
if (!metadata || typeof metadata !== "object") return null;
|
||||
const owner = (metadata as Record<string, unknown>).runtimeControlOwner;
|
||||
if (!owner || typeof owner !== "object") return null;
|
||||
const record = owner as Record<string, unknown>;
|
||||
if (typeof record.ownerId !== "string") return null;
|
||||
return {
|
||||
ownerId: record.ownerId,
|
||||
pid: typeof record.pid === "number" ? record.pid : 0,
|
||||
action: typeof record.action === "string" ? record.action : "start",
|
||||
heartbeatAt: typeof record.heartbeatAt === "string" ? record.heartbeatAt : new Date(0).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Operation ids this process is actively driving. An operation owned by this process but
|
||||
* missing from this set can only be a leftover from a request that died without unwinding,
|
||||
* so it is safe to terminalize immediately instead of waiting out the staleness window.
|
||||
*/
|
||||
const liveRuntimeControlOperationIds = new Set<string>();
|
||||
|
||||
/**
|
||||
* Whether the process that stamped an operation is still alive on this host. A dead owner can
|
||||
* never heartbeat again, so its operation is recoverable immediately instead of after the
|
||||
* staleness window — which is what makes recovery after a crash or restart prompt. Signal 0
|
||||
* only probes; it never touches the process.
|
||||
*/
|
||||
function ownerProcessIsAlive(pid: number): boolean {
|
||||
if (!Number.isInteger(pid) || pid <= 0) return false;
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
// EPERM means the pid exists but belongs to another user: treat it as alive.
|
||||
return (error as NodeJS.ErrnoException)?.code === "EPERM";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare-and-swap predicate for an operation's `updated_at`.
|
||||
*
|
||||
* Postgres stores `timestamptz` at microsecond precision but the driver hands JS a
|
||||
* millisecond-precision `Date`, so an `=` against the value we just read never matches a row
|
||||
* whose timestamp came from the column's `defaultNow()` — which is exactly the shape of every
|
||||
* row a pre-recovery build left behind, i.e. the stranded operations this sweep exists to
|
||||
* clear. Matching the millisecond bucket instead is still a true CAS: a concurrent heartbeat
|
||||
* writes a `Date` in a different millisecond, so it still wins the row.
|
||||
*/
|
||||
function updatedAtUnchanged(updatedAt: Date) {
|
||||
return and(
|
||||
gte(workspaceOperations.updatedAt, updatedAt),
|
||||
lt(workspaceOperations.updatedAt, new Date(updatedAt.getTime() + 1)),
|
||||
);
|
||||
}
|
||||
|
||||
export function resetWorkspaceRuntimeControlStateForTests() {
|
||||
liveRuntimeControlOperationIds.clear();
|
||||
}
|
||||
|
||||
export function workspaceRuntimeControlOwnerIdForTests() {
|
||||
return RUNTIME_CONTROL_OWNER_ID;
|
||||
}
|
||||
|
||||
/**
|
||||
* Same-process serialization of managed runtime controls. The durable sweep below is what
|
||||
* recovers stranded operations across processes; this in-memory claim is the cheaper first
|
||||
* line of defense that rejects an overlapping control in this process before any row is
|
||||
* written. Kept from the runtime-lease chain (PAP-17205) and composed with the ownership
|
||||
* machinery above: the claim is taken before the operation row exists, so a recovery sweep
|
||||
* can never see a live control it is not yet tracking.
|
||||
*/
|
||||
const activeRuntimeControls = new Map<string, { action: string; startedAt: Date }>();
|
||||
|
||||
export async function runExclusiveWorkspaceRuntimeControl<T>(input: {
|
||||
executionWorkspaceId: string;
|
||||
action: string;
|
||||
run: () => Promise<T>;
|
||||
}): Promise<T> {
|
||||
const active = activeRuntimeControls.get(input.executionWorkspaceId);
|
||||
if (active) {
|
||||
throw conflict("A managed runtime control operation is already in progress for this execution workspace.", {
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
activeAction: active.action,
|
||||
requestedAction: input.action,
|
||||
startedAt: active.startedAt.toISOString(),
|
||||
remediation: "Wait for the active operation to reach a terminal state before retrying.",
|
||||
});
|
||||
}
|
||||
|
||||
const claim = { action: input.action, startedAt: new Date() };
|
||||
activeRuntimeControls.set(input.executionWorkspaceId, claim);
|
||||
try {
|
||||
return await input.run();
|
||||
} finally {
|
||||
if (activeRuntimeControls.get(input.executionWorkspaceId) === claim) {
|
||||
activeRuntimeControls.delete(input.executionWorkspaceId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function resetWorkspaceRuntimeControlLocksForTests() {
|
||||
activeRuntimeControls.clear();
|
||||
}
|
||||
|
||||
function toWorkspaceOperation(row: WorkspaceOperationRow): WorkspaceOperation {
|
||||
return {
|
||||
id: row.id,
|
||||
|
|
@ -59,6 +229,11 @@ export interface WorkspaceOperationRecorder {
|
|||
command?: string | null;
|
||||
cwd?: string | null;
|
||||
metadata?: Record<string, unknown> | null;
|
||||
/**
|
||||
* Wall-clock ceiling for `run`. On expiry the operation is failed (terminal) and the
|
||||
* timeout error is rethrown, so a hung provider can never leave an active operation.
|
||||
*/
|
||||
timeoutMs?: number | null;
|
||||
run: () => Promise<{
|
||||
status?: WorkspaceOperationStatus;
|
||||
exitCode?: number | null;
|
||||
|
|
@ -83,9 +258,182 @@ export function workspaceOperationService(db: Db) {
|
|||
return row ? toWorkspaceOperation(row) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Terminalize `running` managed runtime-control operations that no live owner can still
|
||||
* be driving, and return the ones that are genuinely still live.
|
||||
*
|
||||
* Recovery is bounded on both axes: an operation owned by another server process is only
|
||||
* reclaimed after {@link RUNTIME_CONTROL_STALE_AFTER_MS} without a heartbeat, and the
|
||||
* terminalizing UPDATE is a compare-and-swap on `updated_at`, so an owner that heartbeats
|
||||
* concurrently keeps its operation. Only this process's own abandoned operations (owned
|
||||
* by us but no longer tracked in-process) are reclaimed without waiting.
|
||||
*/
|
||||
async function sweepRuntimeControlOperations(
|
||||
executionWorkspaceId: string | null | undefined,
|
||||
options?: { now?: Date; staleAfterMs?: number },
|
||||
) {
|
||||
const now = options?.now ?? new Date();
|
||||
const staleAfterMs = options?.staleAfterMs ?? RUNTIME_CONTROL_STALE_AFTER_MS;
|
||||
const candidates = await db
|
||||
.select()
|
||||
.from(workspaceOperations)
|
||||
.where(
|
||||
executionWorkspaceId
|
||||
? and(
|
||||
eq(workspaceOperations.executionWorkspaceId, executionWorkspaceId),
|
||||
eq(workspaceOperations.status, "running"),
|
||||
)
|
||||
: eq(workspaceOperations.status, "running"),
|
||||
);
|
||||
|
||||
const reconciledIds: string[] = [];
|
||||
const live: WorkspaceOperation[] = [];
|
||||
|
||||
for (const candidate of candidates) {
|
||||
const action = readRuntimeControlAction(candidate.metadata);
|
||||
if (!action) continue;
|
||||
|
||||
const owner = readRuntimeControlOwner(candidate.metadata);
|
||||
const lastSignalAt = owner
|
||||
? new Date(owner.heartbeatAt)
|
||||
: (candidate.updatedAt ?? candidate.startedAt ?? new Date(0));
|
||||
const silentForMs = now.getTime() - lastSignalAt.getTime();
|
||||
|
||||
const ownedByThisProcess = owner?.ownerId === RUNTIME_CONTROL_OWNER_ID;
|
||||
const abandonedByThisProcess = ownedByThisProcess && !liveRuntimeControlOperationIds.has(candidate.id);
|
||||
const ownedHereAndLive = ownedByThisProcess && liveRuntimeControlOperationIds.has(candidate.id);
|
||||
// A previous server process that no longer exists cannot heartbeat again, so its
|
||||
// operations are recoverable at once instead of after the staleness window.
|
||||
const ownerProcessGone = owner !== null && !ownedByThisProcess && !ownerProcessIsAlive(owner.pid);
|
||||
|
||||
if (ownedHereAndLive) {
|
||||
// This process is driving it right now; its own time budget is what terminalizes it.
|
||||
live.push(toWorkspaceOperation(candidate));
|
||||
continue;
|
||||
}
|
||||
if (!abandonedByThisProcess && !ownerProcessGone && silentForMs < staleAfterMs) {
|
||||
live.push(toWorkspaceOperation(candidate));
|
||||
continue;
|
||||
}
|
||||
|
||||
const finishedAt = new Date();
|
||||
const reconciliationMessage =
|
||||
`Reconciled stale managed runtime ${action} operation: ${
|
||||
ownerProcessGone
|
||||
? `the owning server process (pid ${owner?.pid}) is gone`
|
||||
: abandonedByThisProcess
|
||||
? "the owning request no longer exists"
|
||||
: `no live owner has reported progress for ${Math.round(silentForMs / 1000)}s`
|
||||
}, so the operation was failed to release the execution workspace for retry or stop.\n`;
|
||||
// Compare-and-swap on `updated_at`: a live owner that heartbeats between the read and
|
||||
// this write moves the column and keeps its operation.
|
||||
const claimed = await db
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
status: "failed",
|
||||
stderrExcerpt: appendExcerpt(candidate.stderrExcerpt ?? "", reconciliationMessage),
|
||||
metadata: combineMetadata(candidate.metadata as Record<string, unknown> | null, {
|
||||
reconciled: true,
|
||||
reconciliationReason: abandonedByThisProcess
|
||||
? "abandoned_runtime_control"
|
||||
: "orphaned_runtime_control",
|
||||
reconciledAt: finishedAt.toISOString(),
|
||||
}),
|
||||
finishedAt,
|
||||
updatedAt: finishedAt,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(workspaceOperations.id, candidate.id),
|
||||
eq(workspaceOperations.status, "running"),
|
||||
candidate.updatedAt
|
||||
? updatedAtUnchanged(candidate.updatedAt)
|
||||
: isNull(workspaceOperations.updatedAt),
|
||||
),
|
||||
)
|
||||
.returning({ id: workspaceOperations.id })
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!claimed) {
|
||||
// Someone else moved the row; re-read so callers see the current truth.
|
||||
const refreshed = await getById(candidate.id);
|
||||
if (refreshed?.status === "running") live.push(refreshed);
|
||||
continue;
|
||||
}
|
||||
|
||||
liveRuntimeControlOperationIds.delete(candidate.id);
|
||||
reconciledIds.push(candidate.id);
|
||||
|
||||
if (candidate.logStore && candidate.logRef) {
|
||||
const handle = {
|
||||
store: candidate.logStore as "local_file",
|
||||
logRef: candidate.logRef,
|
||||
};
|
||||
try {
|
||||
await logStore.append(handle, {
|
||||
stream: "stderr",
|
||||
chunk: reconciliationMessage,
|
||||
ts: finishedAt.toISOString(),
|
||||
});
|
||||
const finalized = await logStore.finalize(handle);
|
||||
await db
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
logBytes: finalized.bytes,
|
||||
logSha256: finalized.sha256,
|
||||
logCompressed: finalized.compressed,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(workspaceOperations.id, candidate.id));
|
||||
} catch {
|
||||
// The terminal row and stderr excerpt stay inspectable even when the external log
|
||||
// file from the previous process is gone.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { reconciled: reconciledIds.length, operationIds: reconciledIds, live };
|
||||
}
|
||||
|
||||
return {
|
||||
getById,
|
||||
|
||||
/**
|
||||
* Bounded recovery entrypoint. Safe to call on startup and before every managed control.
|
||||
*/
|
||||
async reconcileStaleRuntimeControlOperations(
|
||||
executionWorkspaceId?: string | null,
|
||||
options?: { now?: Date; staleAfterMs?: number },
|
||||
) {
|
||||
const result = await sweepRuntimeControlOperations(executionWorkspaceId, options);
|
||||
return { reconciled: result.reconciled, operationIds: result.operationIds };
|
||||
},
|
||||
|
||||
/**
|
||||
* Refuse a managed runtime control while another one is genuinely live for the same
|
||||
* execution workspace. Stale operations are recovered first, so a workspace stranded by
|
||||
* a dead request or a previous server process becomes controllable again on its own.
|
||||
*/
|
||||
async assertRuntimeControlAvailable(input: {
|
||||
executionWorkspaceId: string;
|
||||
action: string;
|
||||
options?: { now?: Date; staleAfterMs?: number };
|
||||
}) {
|
||||
const { live } = await sweepRuntimeControlOperations(input.executionWorkspaceId, input.options);
|
||||
const blocking = live[0];
|
||||
if (!blocking) return;
|
||||
const owner = readRuntimeControlOwner(blocking.metadata);
|
||||
throw conflict("A managed runtime control operation is already in progress for this execution workspace.", {
|
||||
code: "workspace_runtime_control_in_progress",
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
activeAction: readRuntimeControlAction(blocking.metadata) ?? owner?.action ?? null,
|
||||
requestedAction: input.action,
|
||||
activeOperationId: blocking.id,
|
||||
startedAt: blocking.startedAt instanceof Date ? blocking.startedAt.toISOString() : blocking.startedAt,
|
||||
remediation:
|
||||
"Wait for the active operation to reach a terminal state before retrying; abandoned operations are recovered automatically.",
|
||||
});
|
||||
},
|
||||
|
||||
createRecorder(input: {
|
||||
companyId: string;
|
||||
heartbeatRunId?: string | null;
|
||||
|
|
@ -133,28 +481,91 @@ export function workspaceOperationService(db: Db) {
|
|||
});
|
||||
};
|
||||
|
||||
await db.insert(workspaceOperations).values({
|
||||
id,
|
||||
companyId: input.companyId,
|
||||
executionWorkspaceId,
|
||||
heartbeatRunId: input.heartbeatRunId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
phase: recordInput.phase,
|
||||
command: recordInput.command ?? null,
|
||||
cwd: recordInput.cwd ?? null,
|
||||
status: "running",
|
||||
logStore: handle.store,
|
||||
logRef: handle.logRef,
|
||||
metadata: redactCurrentUserValue(
|
||||
recordInput.metadata ?? null,
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
startedAt,
|
||||
});
|
||||
createdIds.push(id);
|
||||
// Managed runtime controls get an ownership stamp so bounded recovery can tell a
|
||||
// slow-but-live operation from one abandoned by a dead request or server process.
|
||||
const runtimeControlAction = readRuntimeControlAction(recordInput.metadata);
|
||||
const insertedMetadata = runtimeControlAction
|
||||
? {
|
||||
...(recordInput.metadata ?? {}),
|
||||
runtimeControlOwner: {
|
||||
ownerId: RUNTIME_CONTROL_OWNER_ID,
|
||||
pid: process.pid,
|
||||
action: runtimeControlAction,
|
||||
heartbeatAt: startedAt.toISOString(),
|
||||
} satisfies RuntimeControlOwnerStamp,
|
||||
}
|
||||
: recordInput.metadata ?? null;
|
||||
|
||||
// Claim in-process before the row exists, so there is no ordering in which a recovery
|
||||
// sweep can observe an operation stamped by this process but not yet tracked by it —
|
||||
// which is the one state that would let recovery terminalize a genuinely live start.
|
||||
if (runtimeControlAction) liveRuntimeControlOperationIds.add(id);
|
||||
|
||||
try {
|
||||
const result = await recordInput.run();
|
||||
await db.insert(workspaceOperations).values({
|
||||
id,
|
||||
companyId: input.companyId,
|
||||
executionWorkspaceId,
|
||||
heartbeatRunId: input.heartbeatRunId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
phase: recordInput.phase,
|
||||
command: recordInput.command ?? null,
|
||||
cwd: recordInput.cwd ?? null,
|
||||
status: "running",
|
||||
logStore: handle.store,
|
||||
logRef: handle.logRef,
|
||||
metadata: redactCurrentUserValue(
|
||||
insertedMetadata,
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
startedAt,
|
||||
updatedAt: startedAt,
|
||||
});
|
||||
} catch (insertError) {
|
||||
liveRuntimeControlOperationIds.delete(id);
|
||||
throw insertError;
|
||||
}
|
||||
createdIds.push(id);
|
||||
|
||||
let heartbeatTimer: NodeJS.Timeout | null = null;
|
||||
let timeoutTimer: NodeJS.Timeout | null = null;
|
||||
if (runtimeControlAction) {
|
||||
heartbeatTimer = setInterval(() => {
|
||||
const heartbeatAt = new Date();
|
||||
void db
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
metadata: combineMetadata(insertedMetadata, {
|
||||
runtimeControlOwner: {
|
||||
ownerId: RUNTIME_CONTROL_OWNER_ID,
|
||||
pid: process.pid,
|
||||
action: runtimeControlAction,
|
||||
heartbeatAt: heartbeatAt.toISOString(),
|
||||
} satisfies RuntimeControlOwnerStamp,
|
||||
}),
|
||||
updatedAt: heartbeatAt,
|
||||
})
|
||||
.where(and(eq(workspaceOperations.id, id), eq(workspaceOperations.status, "running")))
|
||||
.catch(() => undefined);
|
||||
}, RUNTIME_CONTROL_HEARTBEAT_MS);
|
||||
heartbeatTimer.unref?.();
|
||||
}
|
||||
|
||||
const timeoutMs = recordInput.timeoutMs ?? defaultRuntimeControlTimeoutMs(runtimeControlAction);
|
||||
const settle = async () => {
|
||||
if (!timeoutMs || timeoutMs <= 0) return await recordInput.run();
|
||||
const timeout = new Promise<never>((_resolve, reject) => {
|
||||
timeoutTimer = setTimeout(
|
||||
() => reject(new WorkspaceOperationTimeoutError(timeoutMs, runtimeControlAction)),
|
||||
timeoutMs,
|
||||
);
|
||||
timeoutTimer.unref?.();
|
||||
});
|
||||
return await Promise.race([recordInput.run(), timeout]);
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await settle();
|
||||
await append("system", result.system ?? null);
|
||||
await append("stdout", result.stdout ?? null);
|
||||
await append("stderr", result.stderr ?? null);
|
||||
|
|
@ -172,7 +583,7 @@ export function workspaceOperationService(db: Db) {
|
|||
logSha256: finalized.sha256,
|
||||
logCompressed: finalized.compressed,
|
||||
metadata: redactCurrentUserValue(
|
||||
combineMetadata(recordInput.metadata, result.metadata),
|
||||
combineMetadata(insertedMetadata, result.metadata),
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
finishedAt,
|
||||
|
|
@ -197,11 +608,31 @@ export function workspaceOperationService(db: Db) {
|
|||
logBytes: finalized?.bytes ?? null,
|
||||
logSha256: finalized?.sha256 ?? null,
|
||||
logCompressed: finalized?.compressed ?? false,
|
||||
// Only managed controls carry a failure reason; other phases keep the metadata
|
||||
// they were recorded with.
|
||||
...(runtimeControlAction
|
||||
? {
|
||||
metadata: redactCurrentUserValue(
|
||||
combineMetadata(insertedMetadata, {
|
||||
failureReason: error instanceof WorkspaceOperationTimeoutError
|
||||
? "runtime_control_timeout"
|
||||
: "runtime_control_error",
|
||||
}),
|
||||
currentUserRedactionOptions,
|
||||
) as Record<string, unknown> | null,
|
||||
}
|
||||
: {}),
|
||||
finishedAt,
|
||||
updatedAt: finishedAt,
|
||||
})
|
||||
.where(eq(workspaceOperations.id, id));
|
||||
throw error;
|
||||
} finally {
|
||||
// Releasing the in-process claim before the request unwinds is what lets the very
|
||||
// next control call proceed, and lets recovery treat a leftover row as abandoned.
|
||||
if (heartbeatTimer) clearInterval(heartbeatTimer);
|
||||
if (timeoutTimer) clearTimeout(timeoutTimer);
|
||||
liveRuntimeControlOperationIds.delete(id);
|
||||
}
|
||||
},
|
||||
};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,182 @@
|
|||
/**
|
||||
* Backfill decision coverage for PAP-17158.
|
||||
*
|
||||
* `decideManagedRuntimeExposureBackfill` is the whole contract for which
|
||||
* pre-feature workspaces get upgraded to HTTPS in place and which are left
|
||||
* alone, so each branch is asserted by its reason code rather than only by the
|
||||
* resulting action.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
decideManagedRuntimeExposureBackfill,
|
||||
decideStaleExposureReclaim,
|
||||
} from "./workspace-runtime.js";
|
||||
|
||||
/** A persisted, running, pre-feature Paperclip App dev runtime. */
|
||||
function httpOnlyRunningRow(overrides: Partial<Parameters<typeof decideManagedRuntimeExposureBackfill>[0]> = {}) {
|
||||
return {
|
||||
mode: "auto" as const,
|
||||
brokerAvailable: true,
|
||||
provider: "local_process",
|
||||
serviceName: "paperclip-dev",
|
||||
command: "pnpm dev --bind lan",
|
||||
status: "running",
|
||||
hasExposure: false,
|
||||
declaredIntent: "unset" as const,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("decideManagedRuntimeExposureBackfill", () => {
|
||||
it("reprovisions a running HTTP-only managed worktree runtime", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow())).toEqual({
|
||||
action: "reprovision",
|
||||
reason: "http_only_managed_service",
|
||||
});
|
||||
});
|
||||
|
||||
it("reprovisions a service that is mid-start as well as one already running", () => {
|
||||
for (const status of ["provisioning", "starting", "running"]) {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ status })).action).toBe("reprovision");
|
||||
}
|
||||
});
|
||||
|
||||
it("leaves a stopped service to pick the default up on its next start", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ status: "stopped" }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "stopped_defaults_on_next_start",
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves a deliberate opt-out", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ declaredIntent: "disabled" }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "deliberate_opt_out",
|
||||
});
|
||||
});
|
||||
|
||||
it("is idempotent: a row that already carries exposure state is never re-driven", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ hasExposure: true }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "already_exposed",
|
||||
});
|
||||
// Repeated deploys see the same row and must keep converging on "keep".
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ hasExposure: true })).action).toBe("keep");
|
||||
});
|
||||
|
||||
it("leaves unmanaged and custom services alone", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({
|
||||
serviceName: "preview",
|
||||
command: "pnpm vite",
|
||||
}))).toEqual({ action: "keep", reason: "unmanaged_or_custom_service" });
|
||||
});
|
||||
|
||||
it("leaves external (non local_process) runtime services alone", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ provider: "external" }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "not_a_managed_local_process",
|
||||
});
|
||||
});
|
||||
|
||||
it("does nothing when the automatic default is switched off", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ mode: "off" }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "https_default_disabled",
|
||||
});
|
||||
});
|
||||
|
||||
it("skips the backfill when the host broker is unavailable, unless forced", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ brokerAvailable: false }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "broker_unavailable",
|
||||
});
|
||||
expect(
|
||||
decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ brokerAvailable: false, mode: "force" })).action,
|
||||
).toBe("reprovision");
|
||||
});
|
||||
|
||||
it("never takes down a row it cannot restart from a configured entry", () => {
|
||||
expect(decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ declaredIntent: null }))).toEqual({
|
||||
action: "keep",
|
||||
reason: "no_configured_service_entry",
|
||||
});
|
||||
});
|
||||
|
||||
it("reprovisions an explicit opt-in that is somehow running without exposure", () => {
|
||||
expect(
|
||||
decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({ declaredIntent: "enabled" })).action,
|
||||
).toBe("reprovision");
|
||||
});
|
||||
|
||||
it("checks the opt-out before broker availability so an opt-out never depends on host state", () => {
|
||||
expect(
|
||||
decideManagedRuntimeExposureBackfill(httpOnlyRunningRow({
|
||||
declaredIntent: "disabled",
|
||||
brokerAvailable: false,
|
||||
})).reason,
|
||||
).toBe("deliberate_opt_out");
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* PAP-17285 regression coverage for the global startup exposure sweep.
|
||||
*
|
||||
* A server restart at 12:25:49 UTC deleted the operator-preserved `42000/52000`
|
||||
* Serve mappings by issuing broker removals for two stale `stopped` rows — 2 and
|
||||
* 3 days old, both claiming the same recycled pair — purely on those rows'
|
||||
* authority. The sweep spans every execution workspace and company on the host
|
||||
* and runs on every start, so "trust the row" is an unbounded delete primitive.
|
||||
* These pin the corroboration contract that replaced it.
|
||||
*/
|
||||
const RUNTIME = "c4a0f1d8-be27-4f95-970e-443fe4a517b7";
|
||||
const OTHER_RUNTIME = "c0cca855-0d56-47fc-beca-9f3e3b8d341f";
|
||||
|
||||
describe("decideStaleExposureReclaim", () => {
|
||||
it("defers instead of removing when the broker cannot be reached", () => {
|
||||
// Unreachable broker proves nothing. Fail closed toward preservation: a
|
||||
// mapping we cannot attribute must never be deleted on a guess.
|
||||
expect(decideStaleExposureReclaim({ runtimeId: RUNTIME, ownedListeners: null })).toEqual({
|
||||
action: "defer",
|
||||
reason: "broker_unreachable",
|
||||
});
|
||||
});
|
||||
|
||||
it("clears stale bookkeeping WITHOUT a Serve mutation when the broker owns nothing for the runtime", () => {
|
||||
// The 12:28:19 row: 3 days old, claiming a pair that had since been recycled.
|
||||
// Nothing of ours is published under this runtime, so there is nothing to
|
||||
// remove — only local bookkeeping to tidy.
|
||||
expect(decideStaleExposureReclaim({ runtimeId: RUNTIME, ownedListeners: [] })).toEqual({
|
||||
action: "clear_bookkeeping",
|
||||
reason: "not_owned_by_broker",
|
||||
});
|
||||
expect(
|
||||
decideStaleExposureReclaim({
|
||||
runtimeId: RUNTIME,
|
||||
ownedListeners: [{ runtimeId: OTHER_RUNTIME, port: 42000 }],
|
||||
}),
|
||||
).toEqual({ action: "clear_bookkeeping", reason: "not_owned_by_broker" });
|
||||
});
|
||||
|
||||
it("reclaims only when the broker still attributes a listener to that exact runtime", () => {
|
||||
// Genuine orphan GC — the behaviour PAP-17207 row 7 requires — is preserved.
|
||||
expect(
|
||||
decideStaleExposureReclaim({
|
||||
runtimeId: RUNTIME,
|
||||
ownedListeners: [{ runtimeId: RUNTIME, port: 42001 }],
|
||||
}),
|
||||
).toEqual({ action: "reclaim", reason: "owned_by_broker" });
|
||||
});
|
||||
|
||||
it("attributes by runtimeId and never by port, so a recycled port cannot cross-authorize", () => {
|
||||
// Two rows claiming the same recycled pair is exactly the observed state. If
|
||||
// this matched on port, either row could authorize deleting the other's
|
||||
// mapping — which is the shape of the original loss.
|
||||
expect(
|
||||
decideStaleExposureReclaim({
|
||||
runtimeId: RUNTIME,
|
||||
ownedListeners: [{ runtimeId: OTHER_RUNTIME, port: 42000 }, { runtimeId: OTHER_RUNTIME, port: 52000 }],
|
||||
}).action,
|
||||
).toBe("clear_bookkeeping");
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,542 @@
|
|||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import type { BrokerClient, BrokerListenerRequest } from "./runtime-exposure/broker-client.js";
|
||||
import { diagnoseRuntimeListenerBinds } from "./runtime-exposure/loopback-listener.js";
|
||||
import {
|
||||
resetRuntimeServicesForTests,
|
||||
setWorkspaceRuntimeExposureDepsForTests,
|
||||
startRuntimeServicesForWorkspaceControl,
|
||||
stopRuntimeServicesForExecutionWorkspace,
|
||||
} from "./workspace-runtime.js";
|
||||
|
||||
const EXECUTION_WORKSPACE_ID = "11111111-2222-4333-8444-555566667777";
|
||||
const HANDLE = "handle-abcdef1234567890";
|
||||
|
||||
// The shared test setup pins the automatic default off so unrelated suites do
|
||||
// not probe for a real host broker. This suite is about the default, so it opts
|
||||
// back in and restores the harness value afterwards.
|
||||
let previousHttpsMode: string | undefined;
|
||||
beforeEach(() => {
|
||||
previousHttpsMode = process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS;
|
||||
process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS = "auto";
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
if (previousHttpsMode === undefined) delete process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS;
|
||||
else process.env.PAPERCLIP_MANAGED_RUNTIME_HTTPS = previousHttpsMode;
|
||||
// These tests spawn real loopback backends on dedicated-range ports; reap them
|
||||
// rather than leaving one squatting 42xxx/52xxx for every test in the file.
|
||||
await resetRuntimeServicesForTests({ terminateProcesses: true });
|
||||
});
|
||||
|
||||
function serviceCommand() {
|
||||
return `node -e 'const http=require("http");const p=Number(process.env.PORT);for(const q of [p,p+10000])http.createServer((_,r)=>{r.statusCode=200;r.end("ok")}).listen(q,"127.0.0.1");setInterval(()=>{},1000)'`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fake guest dev runners written to disk (PAP-17256).
|
||||
*
|
||||
* They must be real files named `dev-runner*.mjs` for two reasons: the command
|
||||
* has to look like a dev-runner invocation for the rewrite to apply at all, and
|
||||
* `node <file> --bind lan` hands the flags to the script, which is exactly how a
|
||||
* real `pnpm dev --bind lan` reaches `scripts/dev-runner.ts`.
|
||||
*/
|
||||
let guestDir: string;
|
||||
|
||||
/**
|
||||
* A guest checkout from *before* managed HTTPS exposure existed.
|
||||
*
|
||||
* Reproduces the behaviour that actually broke the lanes: `scripts/dev-runner.ts`
|
||||
* on plain master derives its bind mode from its own `--bind` / `--bind-host`
|
||||
* argv and **ignores `PAPERCLIP_BIND` / `PAPERCLIP_MANAGED_RUNTIME_EXPOSURE`
|
||||
* entirely** — so env-only hardening cannot reach it. `--bind lan` therefore
|
||||
* means `0.0.0.0`, which the broker must refuse.
|
||||
*/
|
||||
const PRE_MANAGED_EXPOSURE_GUEST = `
|
||||
import http from "node:http";
|
||||
const argv = process.argv.slice(2);
|
||||
const valueOf = (flag) => {
|
||||
const at = argv.indexOf(flag);
|
||||
const value = at >= 0 ? argv[at + 1] : undefined;
|
||||
return value && !value.startsWith("--") ? value : undefined;
|
||||
};
|
||||
const mode = valueOf("--bind") ?? "loopback";
|
||||
const host = mode === "custom" ? (valueOf("--bind-host") ?? "127.0.0.1")
|
||||
: mode === "lan" ? "0.0.0.0"
|
||||
: "127.0.0.1";
|
||||
const p = Number(process.env.PORT);
|
||||
for (const q of [p, p + 10000]) {
|
||||
http.createServer((_, r) => { r.statusCode = 200; r.end("ok"); }).listen(q, host);
|
||||
}
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
|
||||
/**
|
||||
* The true shape of the deployed failure: a guest that honours the bind argv for
|
||||
* its own HTTP listener but whose Vite gets `hmr.port` with no `hmr.server` and
|
||||
* no `server.host`, so Vite opens its *own* HMR websocket on the IPv6 wildcard
|
||||
* regardless of the bind mode. Plain master's `server/src/app.ts` does exactly
|
||||
* this, which is why forcing the bind is necessary but not sufficient there.
|
||||
*/
|
||||
const WILDCARD_HMR_GUEST = `
|
||||
import http from "node:http";
|
||||
const argv = process.argv.slice(2);
|
||||
const at = argv.indexOf("--bind-host");
|
||||
const host = at >= 0 ? argv[at + 1] : "127.0.0.1";
|
||||
const p = Number(process.env.PORT);
|
||||
http.createServer((_, r) => { r.statusCode = 200; r.end("ok"); }).listen(p, host);
|
||||
// No host argument: Vite's own HMR listener lands on the wildcard.
|
||||
http.createServer((_, r) => { r.statusCode = 426; r.end(); }).listen(p + 10000);
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
|
||||
/** A guest so old it has no bind flags at all and always binds the wildcard. */
|
||||
const ALWAYS_WILDCARD_GUEST = `
|
||||
import http from "node:http";
|
||||
const p = Number(process.env.PORT);
|
||||
for (const q of [p, p + 10000]) {
|
||||
http.createServer((_, r) => { r.statusCode = 200; r.end("ok"); }).listen(q, "0.0.0.0");
|
||||
}
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
|
||||
beforeAll(async () => {
|
||||
guestDir = await fs.mkdtemp(path.join(os.tmpdir(), "pap-17256-guest-"));
|
||||
await fs.writeFile(path.join(guestDir, "dev-runner.mjs"), PRE_MANAGED_EXPOSURE_GUEST);
|
||||
await fs.writeFile(path.join(guestDir, "dev-runner-legacy.mjs"), ALWAYS_WILDCARD_GUEST);
|
||||
await fs.writeFile(path.join(guestDir, "dev-runner-wildcard-hmr.mjs"), WILDCARD_HMR_GUEST);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(guestDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const guestCommand = (file: string) => `node ${path.join(guestDir, file)}`;
|
||||
|
||||
function createBroker() {
|
||||
const calls: string[] = [];
|
||||
let listeners: BrokerListenerRequest[] = [];
|
||||
const broker: BrokerClient = {
|
||||
async reserve(_runtimeId, requested) {
|
||||
calls.push("reserve");
|
||||
listeners = requested;
|
||||
return { handle: HANDLE, reservedPorts: requested.map((listener) => listener.port) };
|
||||
},
|
||||
async expose() {
|
||||
calls.push("expose");
|
||||
return { handle: HANDLE, publicPorts: listeners.map((listener) => listener.port) };
|
||||
},
|
||||
async remove() {
|
||||
calls.push("remove");
|
||||
return { removedPorts: listeners.map((listener) => listener.port) };
|
||||
},
|
||||
async list() {
|
||||
return [];
|
||||
},
|
||||
};
|
||||
return { broker, calls };
|
||||
}
|
||||
|
||||
function installDeps(overrides: {
|
||||
broker: BrokerClient;
|
||||
probeHealth?: () => Promise<boolean>;
|
||||
isBrokerAvailable?: () => Promise<boolean>;
|
||||
isPortAvailable?: (port: number) => Promise<boolean>;
|
||||
diagnoseListenerBinds?: (ports: number[]) => Promise<string | null>;
|
||||
}) {
|
||||
setWorkspaceRuntimeExposureDepsForTests({
|
||||
broker: overrides.broker,
|
||||
isPortAvailable: overrides.isPortAvailable ?? (async () => true),
|
||||
isBrokerAvailable: overrides.isBrokerAvailable ?? (async () => true),
|
||||
resolveHostname: async () => "runner.tail123.ts.net",
|
||||
probeHealth: overrides.probeHealth ?? (async () => true),
|
||||
now: () => "2026-08-11T00:00:00.000Z",
|
||||
// The real /proc-backed diagnosis, not a fake: the fake broker below always
|
||||
// says yes, so this is what has to catch an off-loopback bind here.
|
||||
diagnoseListenerBinds: overrides.diagnoseListenerBinds ?? diagnoseRuntimeListenerBinds,
|
||||
});
|
||||
}
|
||||
|
||||
const DECLARED_EXPOSE = {
|
||||
type: "tailscale_https",
|
||||
hostname: "auto",
|
||||
publicPort: "same",
|
||||
includePaperclipViteHmr: true,
|
||||
failurePolicy: "fail_closed",
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* The pre-feature Paperclip App project template, verbatim: a hard-coded HTTP
|
||||
* `urlTemplate`, a pinned port outside the broker's dedicated range, and no
|
||||
* exposure declaration at all.
|
||||
*/
|
||||
const LEGACY_HTTP_EXPOSE = {
|
||||
type: "url",
|
||||
urlTemplate: "http://paperclip-dev:{{port}}",
|
||||
} as const;
|
||||
|
||||
function startInput(options?: {
|
||||
serviceName?: string;
|
||||
expose?: Record<string, unknown> | null;
|
||||
port?: Record<string, unknown> | number;
|
||||
command?: string;
|
||||
}) {
|
||||
const expose = options?.expose === undefined ? DECLARED_EXPOSE : options.expose;
|
||||
return {
|
||||
invocationId: "runtime-exposure-test",
|
||||
actor: { id: null, name: "Paperclip", companyId: "aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee" },
|
||||
issue: null,
|
||||
workspace: {
|
||||
baseCwd: process.cwd(),
|
||||
source: "project_primary" as const,
|
||||
projectId: null,
|
||||
workspaceId: null,
|
||||
repoUrl: null,
|
||||
repoRef: null,
|
||||
strategy: "project_primary" as const,
|
||||
cwd: process.cwd(),
|
||||
branchName: "test",
|
||||
worktreePath: null,
|
||||
warnings: [],
|
||||
created: false,
|
||||
},
|
||||
executionWorkspaceId: EXECUTION_WORKSPACE_ID,
|
||||
config: {
|
||||
workspaceRuntime: {
|
||||
services: [{
|
||||
name: options?.serviceName ?? "preview",
|
||||
command: options?.command ?? serviceCommand(),
|
||||
port: options?.port ?? { type: "auto", envKey: "PORT" },
|
||||
readiness: { type: "http", urlTemplate: "http://127.0.0.1:{{port}}", timeoutSec: 5 },
|
||||
...(expose ? { expose } : {}),
|
||||
}],
|
||||
},
|
||||
},
|
||||
adapterEnv: {},
|
||||
};
|
||||
}
|
||||
|
||||
describe("workspace runtime tailscale_https lifecycle", () => {
|
||||
it("reserves before spawn, exposes after backend readiness, and removes on stop", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput());
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
expect(runtime.port).toBeGreaterThanOrEqual(42000);
|
||||
expect(runtime.url).toBe(`https://runner.tail123.ts.net:${runtime.port}`);
|
||||
expect(runtime.exposure?.state).toBe("ready");
|
||||
|
||||
await stopRuntimeServicesForExecutionWorkspace({
|
||||
executionWorkspaceId: EXECUTION_WORKSPACE_ID,
|
||||
runtimeServiceId: runtime.id,
|
||||
});
|
||||
expect(calls).toEqual(["reserve", "expose", "remove"]);
|
||||
}, 15_000);
|
||||
|
||||
it("fails closed and removes the mapping when external HTTPS validation fails", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker, probeHealth: async () => false });
|
||||
|
||||
await expect(startRuntimeServicesForWorkspaceControl(startInput())).rejects.toThrow(/HTTPS exposure failed/);
|
||||
expect(calls).toEqual(["reserve", "expose", "remove"]);
|
||||
}, 15_000);
|
||||
});
|
||||
|
||||
describe("automatic tailscale_https default for managed worktree runtimes", () => {
|
||||
it("defaults a legacy paperclip-dev service with no exposure block, relocating its pinned port", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
// Exactly the persisted pre-feature shape: pinned 45439 + HTTP urlTemplate.
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: 45_439,
|
||||
}));
|
||||
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
// 45439 is outside the broker allowlist, so the pinned port cannot be kept.
|
||||
expect(runtime.port).not.toBe(45_439);
|
||||
expect(runtime.port).toBeGreaterThanOrEqual(42_000);
|
||||
expect(runtime.port).toBeLessThanOrEqual(42_999);
|
||||
// The canonical URL is the verified HTTPS origin; HTTP is never retained.
|
||||
expect(runtime.url).toBe(`https://runner.tail123.ts.net:${runtime.port}`);
|
||||
expect(runtime.url).not.toContain("http://");
|
||||
expect(runtime.exposure?.state).toBe("ready");
|
||||
}, 15_000);
|
||||
|
||||
it("keeps an existing runtime port that is already inside the dedicated range", async () => {
|
||||
const { broker } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: 42_500,
|
||||
}));
|
||||
|
||||
expect(runtime.port).toBe(42_500);
|
||||
expect(runtime.url).toBe("https://runner.tail123.ts.net:42500");
|
||||
}, 15_000);
|
||||
|
||||
it("preserves a deliberate opt-out and leaves the service on plain HTTP", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
expose: { ...LEGACY_HTTP_EXPOSE, tailscaleHttps: false },
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}));
|
||||
|
||||
expect(calls).toEqual([]);
|
||||
expect(runtime.exposure ?? null).toBeNull();
|
||||
expect(runtime.url).toBe(`http://paperclip-dev:${runtime.port}`);
|
||||
}, 15_000);
|
||||
|
||||
it("leaves an unmanaged/custom service untouched", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "preview",
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}));
|
||||
|
||||
expect(calls).toEqual([]);
|
||||
expect(runtime.exposure ?? null).toBeNull();
|
||||
expect(runtime.url).toBe(`http://paperclip-dev:${runtime.port}`);
|
||||
}, 15_000);
|
||||
|
||||
it("does not default when the host broker is unavailable", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker, isBrokerAvailable: async () => false });
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}));
|
||||
|
||||
expect(calls).toEqual([]);
|
||||
expect(runtime.exposure ?? null).toBeNull();
|
||||
}, 15_000);
|
||||
|
||||
it("still honors an explicit opt-in when the broker socket is missing, failing loudly", async () => {
|
||||
// A missing broker must never silently downgrade a requested HTTPS preview.
|
||||
const { broker, calls } = createBroker();
|
||||
const failing: BrokerClient = {
|
||||
...broker,
|
||||
async reserve() {
|
||||
calls.push("reserve");
|
||||
throw new Error("ENOENT: broker socket missing");
|
||||
},
|
||||
};
|
||||
installDeps({ broker: failing, isBrokerAvailable: async () => false });
|
||||
|
||||
await expect(
|
||||
startRuntimeServicesForWorkspaceControl(startInput({ serviceName: "paperclip-dev" })),
|
||||
).rejects.toThrow();
|
||||
expect(calls).toEqual(["reserve"]);
|
||||
}, 15_000);
|
||||
|
||||
it("is idempotent across repeated starts: the same port pair is reserved again", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const [first] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: 45_439,
|
||||
}));
|
||||
const firstPort = first.port;
|
||||
await stopRuntimeServicesForExecutionWorkspace({
|
||||
executionWorkspaceId: EXECUTION_WORKSPACE_ID,
|
||||
runtimeServiceId: first.id,
|
||||
});
|
||||
calls.length = 0;
|
||||
|
||||
const [second] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: 45_439,
|
||||
}));
|
||||
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
expect(second.port).toBe(firstPort);
|
||||
expect(second.url).toBe(`https://runner.tail123.ts.net:${firstPort}`);
|
||||
}, 25_000);
|
||||
});
|
||||
|
||||
/**
|
||||
* PAP-17256: the deployed regression. A fresh managed lane on service index 0
|
||||
* failed every start with `listener_ownership_mismatch` because the branch
|
||||
* checkout it launched bound `0.0.0.0`, and the server only asked for loopback
|
||||
* via env vars that checkout never read.
|
||||
*/
|
||||
describe("loopback bind is forced on the guest, not merely requested (PAP-17256)", () => {
|
||||
it("starts a fresh service-index-0 lane whose guest only honours argv", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
// Verbatim the command every failing lane recorded, modulo the fake guest.
|
||||
command: `${guestCommand("dev-runner.mjs")} --bind lan`,
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}));
|
||||
|
||||
// The launched command carries the loopback bind, replacing `--bind lan`.
|
||||
expect(runtime.command).toContain("--bind custom --bind-host 127.0.0.1");
|
||||
expect(runtime.command).not.toContain("--bind lan");
|
||||
|
||||
// And because the guest honoured it, both listeners are loopback-only, so
|
||||
// the exposure completes instead of being denied.
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
expect(runtime.exposure?.state).toBe("ready");
|
||||
expect(runtime.exposure?.lastError ?? null).toBeNull();
|
||||
expect(runtime.port).toBeGreaterThanOrEqual(42_000);
|
||||
expect(runtime.url).toBe(`https://runner.tail123.ts.net:${runtime.port}`);
|
||||
|
||||
// Independent confirmation on the live listeners, using the same /proc read
|
||||
// the broker's gate performs.
|
||||
const hmrPort = runtime.port! + 10_000;
|
||||
expect(await diagnoseRuntimeListenerBinds([runtime.port!, hmrPort])).toBeNull();
|
||||
|
||||
await stopRuntimeServicesForExecutionWorkspace({
|
||||
executionWorkspaceId: EXECUTION_WORKSPACE_ID,
|
||||
runtimeServiceId: runtime.id,
|
||||
});
|
||||
expect(calls).toEqual(["reserve", "expose", "remove"]);
|
||||
}, 20_000);
|
||||
|
||||
it("reaches a terminal failure naming the port and address when a guest still binds the wildcard", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
// A guest with no bind flags at all: the argv rewrite cannot reach it, so
|
||||
// this is the residual case that must fail loudly rather than expose.
|
||||
await expect(startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
command: guestCommand("dev-runner-legacy.mjs"),
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}))).rejects.toThrow(/listener_ownership_mismatch.*(?:0\.0\.0\.0|::)/s);
|
||||
|
||||
// Terminal recovery: the reservation is released and nothing was exposed.
|
||||
// The diagnosis runs before the broker, so `expose` is never attempted.
|
||||
expect(calls).toEqual(["reserve", "remove"]);
|
||||
}, 20_000);
|
||||
|
||||
it("explains rather than only coding the failure, so the next operator can act", async () => {
|
||||
const { broker } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const error = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
command: guestCommand("dev-runner-legacy.mjs"),
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
})).then(() => null, (err: unknown) => err as Error);
|
||||
|
||||
expect(error).not.toBeNull();
|
||||
// The bare code alone is what cost PAP-17254 three diagnostic cycles.
|
||||
expect(error!.message).toContain("listener_ownership_mismatch");
|
||||
expect(error!.message).toContain("loopback");
|
||||
expect(error!.message).toContain("--bind custom --bind-host 127.0.0.1");
|
||||
}, 20_000);
|
||||
|
||||
it("leaves a non-Paperclip service's --bind argument alone", async () => {
|
||||
// `--bind` means something entirely different to the HTTPS probe canaries
|
||||
// (`python3 -m http.server --bind 127.0.0.1`); rewriting it would break them.
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const declared = `${serviceCommand()} -- --bind 127.0.0.1`;
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "https-probe-canary",
|
||||
command: declared,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}));
|
||||
|
||||
expect(runtime.command).toBe(declared);
|
||||
expect(runtime.command).not.toContain("--bind custom");
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
expect(runtime.exposure?.state).toBe("ready");
|
||||
}, 20_000);
|
||||
|
||||
it("does not rewrite a Paperclip dev command when the service is not exposed", async () => {
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const declared = `${guestCommand("dev-runner.mjs")} --bind lan`;
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
command: declared,
|
||||
expose: { ...LEGACY_HTTP_EXPOSE, tailscaleHttps: false },
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}));
|
||||
|
||||
expect(calls).toEqual([]);
|
||||
expect(runtime.command).toBe(declared);
|
||||
}, 20_000);
|
||||
});
|
||||
|
||||
describe("readiness probes loopback for an exposed runtime (PAP-17256)", () => {
|
||||
it("starts even when the only readiness target is the MagicDNS display URL", async () => {
|
||||
// The live project config declares a loopback readiness URL, but when it does
|
||||
// not, readiness falls back to `expose.urlTemplate` — a MagicDNS name that
|
||||
// resolves off-loopback. That only ever answered because the guest was bound
|
||||
// to the wildcard, so it must be normalised alongside the bind fix.
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const input = startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
command: `${guestCommand("dev-runner.mjs")} --bind lan`,
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
});
|
||||
// Drop the explicit loopback readiness URL so the fallback path is exercised.
|
||||
const service = input.config.workspaceRuntime.services[0] as Record<string, unknown>;
|
||||
service.readiness = { type: "http", timeoutSec: 10, intervalMs: 100 };
|
||||
|
||||
const [runtime] = await startRuntimeServicesForWorkspaceControl(input);
|
||||
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
expect(runtime.exposure?.state).toBe("ready");
|
||||
expect(runtime.url).toBe(`https://runner.tail123.ts.net:${runtime.port}`);
|
||||
}, 20_000);
|
||||
});
|
||||
|
||||
describe("the deployed failure shape: loopback app port, wildcard HMR (PAP-17256)", () => {
|
||||
it("fails terminally naming the HMR port, because forcing the bind cannot reach Vite's own listener", async () => {
|
||||
// Plain master's app.ts passes Vite `hmr.port` without `hmr.server` or
|
||||
// `server.host`, so the HMR websocket binds `::` no matter what the bind mode
|
||||
// is. The argv rewrite fixes the app port; only the preflight catches this.
|
||||
const { broker, calls } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
const error = await startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
command: `${guestCommand("dev-runner-wildcard-hmr.mjs")} --bind lan`,
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
})).then(() => null, (err: unknown) => err as Error);
|
||||
|
||||
expect(error).not.toBeNull();
|
||||
expect(error!.message).toContain("listener_ownership_mismatch");
|
||||
// The app port obeyed the forced bind; the HMR companion is the offender, and
|
||||
// the diagnosis has to say so rather than blaming the lane as a whole.
|
||||
expect(error!.message).toMatch(/port 5\d{4} is bound to/);
|
||||
expect(error!.message).not.toMatch(/port 4\d{4} is bound to/);
|
||||
expect(calls).toEqual(["reserve", "remove"]);
|
||||
}, 20_000);
|
||||
});
|
||||
|
|
@ -0,0 +1,317 @@
|
|||
import { and, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { executionWorkspaceRuntimeLeases, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { conflict } from "../errors.js";
|
||||
|
||||
type ExecutionWorkspaceRuntimeLeaseRow = typeof executionWorkspaceRuntimeLeases.$inferSelect;
|
||||
type LeaseTx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
|
||||
/**
|
||||
* Issue statuses that keep an issue eligible to drive workspace runtime controls.
|
||||
* Shared with the runtime authorization path so "still authorized" and "still owns
|
||||
* the lease" cannot drift apart.
|
||||
*/
|
||||
export const WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES: readonly string[] = [
|
||||
"backlog",
|
||||
"todo",
|
||||
"in_progress",
|
||||
"in_review",
|
||||
"blocked",
|
||||
];
|
||||
|
||||
const TERMINAL_HEARTBEAT_RUN_STATUSES = new Set([
|
||||
"succeeded",
|
||||
"interrupted",
|
||||
"failed",
|
||||
"cancelled",
|
||||
"timed_out",
|
||||
]);
|
||||
|
||||
/** Runtime control actions that mutate the workspace and therefore need the lease. */
|
||||
export const LEASED_WORKSPACE_RUNTIME_ACTIONS: readonly string[] = ["start", "stop", "restart"];
|
||||
|
||||
/**
|
||||
* Upper bound on how long a lease survives without the owner touching it. Recovery
|
||||
* from a stale owner is otherwise driven by owner eligibility, which is cheaper and
|
||||
* more precise; the TTL only bounds the cases eligibility cannot see (for example an
|
||||
* agent-scoped owner with no run or issue identity).
|
||||
*/
|
||||
export const WORKSPACE_RUNTIME_LEASE_TTL_MS = 30 * 60 * 1000;
|
||||
|
||||
export type WorkspaceRuntimeLeaseOwner = {
|
||||
actorType: string;
|
||||
agentId?: string | null;
|
||||
runId?: string | null;
|
||||
issueId?: string | null;
|
||||
};
|
||||
|
||||
export type WorkspaceRuntimeLeaseStaleReason =
|
||||
| "lease_expired"
|
||||
| "owner_issue_missing"
|
||||
| "owner_issue_hidden"
|
||||
| "owner_issue_terminal"
|
||||
| "owner_run_missing"
|
||||
| "owner_run_terminal";
|
||||
|
||||
export type WorkspaceRuntimeLeaseClaim = {
|
||||
outcome: "created" | "renewed" | "reclaimed" | "bypassed";
|
||||
ownerKey: string | null;
|
||||
lease: ExecutionWorkspaceRuntimeLeaseRow | null;
|
||||
reclaimedFrom: { ownerKey: string; reason: WorkspaceRuntimeLeaseStaleReason } | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Durable owner identity for a lease. The controlling issue is preferred because a
|
||||
* canary lane outlives any single heartbeat run; runs and bare agent keys are only
|
||||
* used when no issue is in scope.
|
||||
*/
|
||||
export function buildWorkspaceRuntimeLeaseOwnerKey(owner: WorkspaceRuntimeLeaseOwner): string | null {
|
||||
if (owner.issueId) return `issue:${owner.issueId}`;
|
||||
if (owner.runId) return `run:${owner.runId}`;
|
||||
if (owner.agentId) return `agent:${owner.agentId}`;
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
|
||||
const separator = ownerKey.indexOf(":");
|
||||
if (separator <= 0) return null;
|
||||
const kind = ownerKey.slice(0, separator);
|
||||
const id = ownerKey.slice(separator + 1);
|
||||
if (!id) return null;
|
||||
if (kind !== "issue" && kind !== "run" && kind !== "agent") return null;
|
||||
return { kind, id };
|
||||
}
|
||||
|
||||
async function evaluateStaleOwner(
|
||||
tx: LeaseTx,
|
||||
lease: ExecutionWorkspaceRuntimeLeaseRow,
|
||||
now: Date,
|
||||
): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
|
||||
if (lease.expiresAt.getTime() <= now.getTime()) return "lease_expired";
|
||||
|
||||
const owner = parseOwnerKey(lease.ownerKey);
|
||||
if (owner?.kind === "issue") {
|
||||
const ownerIssue = await tx
|
||||
.select({ status: issues.status, hiddenAt: issues.hiddenAt })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.id, owner.id), eq(issues.companyId, lease.companyId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!ownerIssue) return "owner_issue_missing";
|
||||
if (ownerIssue.hiddenAt) return "owner_issue_hidden";
|
||||
if (!WORKSPACE_RUNTIME_ELIGIBLE_ISSUE_STATUSES.includes(ownerIssue.status)) return "owner_issue_terminal";
|
||||
return null;
|
||||
}
|
||||
|
||||
if (owner?.kind === "run") {
|
||||
const ownerRun = await tx
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.id, owner.id), eq(heartbeatRuns.companyId, lease.companyId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!ownerRun) return "owner_run_missing";
|
||||
if (TERMINAL_HEARTBEAT_RUN_STATUSES.has(ownerRun.status)) return "owner_run_terminal";
|
||||
return null;
|
||||
}
|
||||
|
||||
// Agent-scoped and unparseable owners have no lifecycle to inspect, so the TTL
|
||||
// checked above is their only recovery path.
|
||||
return null;
|
||||
}
|
||||
|
||||
function throwLeaseConflict(input: {
|
||||
lease: ExecutionWorkspaceRuntimeLeaseRow;
|
||||
requestedAction: string;
|
||||
requestedOwnerKey: string;
|
||||
}): never {
|
||||
const { lease } = input;
|
||||
throw conflict(
|
||||
"Another issue run holds the exclusive runtime lease for this execution workspace.",
|
||||
{
|
||||
code: "workspace_runtime_lease_conflict",
|
||||
executionWorkspaceId: lease.executionWorkspaceId,
|
||||
requestedAction: input.requestedAction,
|
||||
requestedOwnerKey: input.requestedOwnerKey,
|
||||
ownerKey: lease.ownerKey,
|
||||
ownerIssueId: lease.ownerIssueId,
|
||||
ownerRunId: lease.ownerRunId,
|
||||
ownerAgentId: lease.ownerAgentId,
|
||||
lastAction: lease.lastAction,
|
||||
claimedAt: lease.claimedAt.toISOString(),
|
||||
renewedAt: lease.renewedAt.toISOString(),
|
||||
expiresAt: lease.expiresAt.toISOString(),
|
||||
remediation:
|
||||
"Do not retry against this execution workspace. Wait for the owning issue to reach a terminal status, ask its owner to release the workspace, or wait for the lease to expire, then retry.",
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
export function workspaceRuntimeLeaseService(db: Db) {
|
||||
return {
|
||||
async get(executionWorkspaceId: string) {
|
||||
return await db
|
||||
.select()
|
||||
.from(executionWorkspaceRuntimeLeases)
|
||||
.where(eq(executionWorkspaceRuntimeLeases.executionWorkspaceId, executionWorkspaceId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
},
|
||||
|
||||
/**
|
||||
* Atomically claim (or renew, or reclaim from a stale owner) the exclusive runtime
|
||||
* lease for an execution workspace. Cross-process exclusivity comes from the unique
|
||||
* constraint on execution_workspace_id plus a `SELECT ... FOR UPDATE` on the existing
|
||||
* row, so competing claims serialize in the database rather than in one process.
|
||||
*
|
||||
* Board/operator actors bypass the lease entirely: they keep unconditional control
|
||||
* and never take the lane away from an agent run.
|
||||
*/
|
||||
async claim(input: {
|
||||
companyId: string;
|
||||
executionWorkspaceId: string;
|
||||
action: string;
|
||||
owner: WorkspaceRuntimeLeaseOwner;
|
||||
now?: Date;
|
||||
ttlMs?: number;
|
||||
}): Promise<WorkspaceRuntimeLeaseClaim> {
|
||||
if (input.owner.actorType !== "agent") {
|
||||
return { outcome: "bypassed", ownerKey: null, lease: null, reclaimedFrom: null };
|
||||
}
|
||||
|
||||
const ownerKey = buildWorkspaceRuntimeLeaseOwnerKey(input.owner);
|
||||
if (!ownerKey) {
|
||||
return { outcome: "bypassed", ownerKey: null, lease: null, reclaimedFrom: null };
|
||||
}
|
||||
|
||||
const now = input.now ?? new Date();
|
||||
const expiresAt = new Date(now.getTime() + (input.ttlMs ?? WORKSPACE_RUNTIME_LEASE_TTL_MS));
|
||||
const ownerColumns = {
|
||||
ownerKey,
|
||||
ownerIssueId: input.owner.issueId ?? null,
|
||||
ownerRunId: input.owner.runId ?? null,
|
||||
ownerAgentId: input.owner.agentId ?? null,
|
||||
};
|
||||
|
||||
return await db.transaction(async (tx) => {
|
||||
const created = await tx
|
||||
.insert(executionWorkspaceRuntimeLeases)
|
||||
.values({
|
||||
companyId: input.companyId,
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
...ownerColumns,
|
||||
lastAction: input.action,
|
||||
claimedAt: now,
|
||||
renewedAt: now,
|
||||
expiresAt,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.onConflictDoNothing({ target: executionWorkspaceRuntimeLeases.executionWorkspaceId })
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (created) {
|
||||
return { outcome: "created" as const, ownerKey, lease: created, reclaimedFrom: null };
|
||||
}
|
||||
|
||||
// The insert above conflicted, so a row exists (and, if a competing claim was
|
||||
// mid-flight, the conflict already waited for it to commit). Lock it before
|
||||
// deciding whether this caller may take or keep the lane.
|
||||
const current = await tx
|
||||
.select()
|
||||
.from(executionWorkspaceRuntimeLeases)
|
||||
.where(eq(executionWorkspaceRuntimeLeases.executionWorkspaceId, input.executionWorkspaceId))
|
||||
.for("update")
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!current) {
|
||||
// The holder released between the conflict and the lock; the caller can retry
|
||||
// immediately rather than mutating a workspace it does not own.
|
||||
throw conflict("The execution workspace runtime lease changed while it was being claimed.", {
|
||||
code: "workspace_runtime_lease_contended",
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
requestedAction: input.action,
|
||||
remediation: "Retry the runtime control request.",
|
||||
});
|
||||
}
|
||||
|
||||
if (current.ownerKey === ownerKey) {
|
||||
const renewed = await tx
|
||||
.update(executionWorkspaceRuntimeLeases)
|
||||
.set({
|
||||
...ownerColumns,
|
||||
lastAction: input.action,
|
||||
renewedAt: now,
|
||||
expiresAt,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(eq(executionWorkspaceRuntimeLeases.id, current.id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? current);
|
||||
return { outcome: "renewed" as const, ownerKey, lease: renewed, reclaimedFrom: null };
|
||||
}
|
||||
|
||||
const staleReason = await evaluateStaleOwner(tx, current, now);
|
||||
if (!staleReason) {
|
||||
throwLeaseConflict({ lease: current, requestedAction: input.action, requestedOwnerKey: ownerKey });
|
||||
}
|
||||
|
||||
const reclaimed = await tx
|
||||
.update(executionWorkspaceRuntimeLeases)
|
||||
.set({
|
||||
companyId: input.companyId,
|
||||
...ownerColumns,
|
||||
lastAction: input.action,
|
||||
claimedAt: now,
|
||||
renewedAt: now,
|
||||
expiresAt,
|
||||
updatedAt: now,
|
||||
metadata: {
|
||||
reclaimedFromOwnerKey: current.ownerKey,
|
||||
reclaimReason: staleReason,
|
||||
reclaimedAt: now.toISOString(),
|
||||
},
|
||||
})
|
||||
.where(eq(executionWorkspaceRuntimeLeases.id, current.id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!reclaimed) {
|
||||
throwLeaseConflict({ lease: current, requestedAction: input.action, requestedOwnerKey: ownerKey });
|
||||
}
|
||||
|
||||
return {
|
||||
outcome: "reclaimed" as const,
|
||||
ownerKey,
|
||||
lease: reclaimed,
|
||||
reclaimedFrom: { ownerKey: current.ownerKey, reason: staleReason },
|
||||
};
|
||||
});
|
||||
},
|
||||
|
||||
/**
|
||||
* Give up the lane. `force` is for operator/lifecycle paths (workspace archive);
|
||||
* owner-scoped releases only remove a lease the caller actually holds.
|
||||
*/
|
||||
async release(input: {
|
||||
executionWorkspaceId: string;
|
||||
owner?: WorkspaceRuntimeLeaseOwner;
|
||||
force?: boolean;
|
||||
}): Promise<{ released: boolean; ownerKey: string | null }> {
|
||||
const ownerKey = input.owner ? buildWorkspaceRuntimeLeaseOwnerKey(input.owner) : null;
|
||||
if (!input.force && !ownerKey) return { released: false, ownerKey: null };
|
||||
|
||||
const condition = input.force
|
||||
? eq(executionWorkspaceRuntimeLeases.executionWorkspaceId, input.executionWorkspaceId)
|
||||
: and(
|
||||
eq(executionWorkspaceRuntimeLeases.executionWorkspaceId, input.executionWorkspaceId),
|
||||
eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey!),
|
||||
);
|
||||
|
||||
const deleted = await db
|
||||
.delete(executionWorkspaceRuntimeLeases)
|
||||
.where(condition)
|
||||
.returning({ ownerKey: executionWorkspaceRuntimeLeases.ownerKey })
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
return { released: Boolean(deleted), ownerKey: deleted?.ownerKey ?? null };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export type WorkspaceRuntimeLeaseService = ReturnType<typeof workspaceRuntimeLeaseService>;
|
||||
|
|
@ -56,6 +56,7 @@ function runtimeService(
|
|||
stoppedAt: null,
|
||||
stopPolicy: null,
|
||||
healthStatus: "healthy",
|
||||
exposure: null,
|
||||
reused: false,
|
||||
...overrides,
|
||||
};
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -860,6 +860,18 @@ export function ProjectProperties({ project, onUpdate, onFieldUpdate, getFieldSa
|
|||
service.command ?? "No URL"
|
||||
)}
|
||||
</div>
|
||||
{service.exposure && service.exposure.state !== "removed" ? (
|
||||
<div
|
||||
className={cn(
|
||||
"text-(length:--text-nano)",
|
||||
service.exposure.state === "failed" || service.exposure.state === "cleanup_pending"
|
||||
? "text-destructive"
|
||||
: "text-muted-foreground",
|
||||
)}
|
||||
>
|
||||
HTTPS {service.exposure.state.replace("_", " ")}
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
<div className="text-(length:--text-nano) text-muted-foreground whitespace-nowrap">
|
||||
{service.lifecycle}
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ function createRuntimeService(overrides: Partial<WorkspaceRuntimeService> = {}):
|
|||
stoppedAt: overrides.stoppedAt ?? null,
|
||||
stopPolicy: overrides.stopPolicy ?? null,
|
||||
healthStatus: overrides.healthStatus ?? "unknown",
|
||||
exposure: overrides.exposure ?? null,
|
||||
configIndex: overrides.configIndex ?? null,
|
||||
createdAt: overrides.createdAt ?? new Date("2026-04-12T00:00:00.000Z"),
|
||||
updatedAt: overrides.updatedAt ?? new Date("2026-04-12T00:00:00.000Z"),
|
||||
|
|
@ -423,6 +424,61 @@ describe("WorkspaceRuntimeControls", () => {
|
|||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it.each([
|
||||
["failed", "external HTTPS health probe did not validate", "HTTPS unavailable", "Check the Tailscale broker and node HTTPS configuration."],
|
||||
["cleanup_pending", "host broker cleanup confirmation timed out", "HTTPS cleanup pending", "Restart the host broker before reusing this port."],
|
||||
] as const)(
|
||||
"shows %s exposure state, last error, and remediation on service cards",
|
||||
(state, lastError, label, remediation) => {
|
||||
const sections = buildWorkspaceRuntimeControlSections({
|
||||
runtimeConfig: {
|
||||
commands: [
|
||||
{ id: "web", name: "web", kind: "service", command: "pnpm dev" },
|
||||
],
|
||||
},
|
||||
runtimeServices: [
|
||||
createRuntimeService({
|
||||
id: "service-web",
|
||||
serviceName: "web",
|
||||
status: "stopped",
|
||||
exposure: {
|
||||
provider: "tailscale_https",
|
||||
state,
|
||||
publicUrl: null,
|
||||
hostname: "paperclip-dev.tail29c1aa.ts.net",
|
||||
listeners: [{ purpose: "app", publicPort: 42002, targetPort: 42002 }],
|
||||
brokerRef: "service-web",
|
||||
lastError,
|
||||
updatedAt: "2026-08-12T00:00:00.000Z",
|
||||
},
|
||||
}),
|
||||
],
|
||||
canStartServices: true,
|
||||
});
|
||||
|
||||
const root = createRoot(container);
|
||||
act(() => {
|
||||
root.render(
|
||||
<WorkspaceRuntimeControls
|
||||
sections={sections}
|
||||
onAction={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
|
||||
const alert = container.querySelector('[role="alert"]');
|
||||
const summary = alert?.firstElementChild;
|
||||
expect(alert?.classList.contains("text-destructive")).toBe(true);
|
||||
expect(summary?.classList.contains("line-clamp-3")).toBe(true);
|
||||
expect(summary?.getAttribute("title")).toBe(lastError);
|
||||
expect(alert?.textContent).toContain(label);
|
||||
expect(alert?.textContent).toContain(lastError);
|
||||
expect(alert?.textContent).toContain(remediation);
|
||||
|
||||
act(() => root.unmount());
|
||||
},
|
||||
);
|
||||
|
||||
it("can render square plain surfaces for embedded configuration pages", () => {
|
||||
const sections = buildWorkspaceRuntimeControlSections({
|
||||
runtimeConfig: {
|
||||
|
|
@ -627,6 +683,98 @@ describe("buildWorkspaceServiceControlEntries", () => {
|
|||
expect(entries[0].state).toBe("failed");
|
||||
expect(entries[0].failureDetail).toMatch(/^Service failed · /);
|
||||
});
|
||||
|
||||
it("surfaces HTTPS failure independently while the backend remains running", () => {
|
||||
const running = createRuntimeService({
|
||||
status: "running",
|
||||
healthStatus: "healthy",
|
||||
port: 42000,
|
||||
url: null,
|
||||
exposure: {
|
||||
provider: "tailscale_https",
|
||||
state: "failed",
|
||||
publicUrl: null,
|
||||
hostname: "runner.tail123.ts.net",
|
||||
listeners: [{ purpose: "app", publicPort: 42000, targetPort: 42000 }],
|
||||
brokerRef: "service-1",
|
||||
lastError: "cli_error",
|
||||
updatedAt: "2026-08-11T00:00:00.000Z",
|
||||
},
|
||||
});
|
||||
const built = buildWorkspaceRuntimeControlSections({
|
||||
runtimeConfig: { commands: [{ id: "web", name: "web", kind: "service", command: "pnpm dev" }] },
|
||||
runtimeServices: [running],
|
||||
canStartServices: true,
|
||||
});
|
||||
const [entry] = buildWorkspaceServiceControlEntries({ sections: built, runtimeServices: [running] });
|
||||
|
||||
expect(entry.state).toBe("running");
|
||||
expect(entry.exposureState).toBe("failed");
|
||||
expect(entry.exposureDetail).toMatch(/^HTTPS unavailable/);
|
||||
expect(entry.exposureDetail).not.toContain("cli_error");
|
||||
});
|
||||
|
||||
it("carries the verified HTTPS URL into the launch entry, and no HTTP fallback while pending", () => {
|
||||
// PAP-17158: the workspace/project/issue launch links are rendered from these
|
||||
// entries, so the tailnet HTTPS URL has to survive the mapping intact — and a
|
||||
// service whose exposure is not yet verified must offer no URL at all rather
|
||||
// than the loopback backend it is really listening on.
|
||||
const httpsUrl = "https://paperclip-dev.tail29c1aa.ts.net:42010";
|
||||
const buildEntry = (service: ReturnType<typeof createRuntimeService>) => {
|
||||
const sections = buildWorkspaceRuntimeControlSections({
|
||||
runtimeConfig: { commands: [{ id: "web", name: "web", kind: "service", command: "pnpm dev" }] },
|
||||
runtimeServices: [service],
|
||||
canStartServices: true,
|
||||
});
|
||||
return buildWorkspaceServiceControlEntries({ sections, runtimeServices: [service] })[0];
|
||||
};
|
||||
|
||||
const ready = buildEntry(createRuntimeService({
|
||||
status: "running",
|
||||
healthStatus: "healthy",
|
||||
port: 42_010,
|
||||
url: httpsUrl,
|
||||
exposure: {
|
||||
provider: "tailscale_https",
|
||||
state: "ready",
|
||||
publicUrl: httpsUrl,
|
||||
hostname: "paperclip-dev.tail29c1aa.ts.net",
|
||||
listeners: [
|
||||
{ purpose: "app", publicPort: 42_010, targetPort: 42_010 },
|
||||
{ purpose: "vite_hmr", publicPort: 52_010, targetPort: 52_010 },
|
||||
],
|
||||
brokerRef: "service-1",
|
||||
lastError: null,
|
||||
updatedAt: "2026-08-12T00:00:00.000Z",
|
||||
},
|
||||
}));
|
||||
expect(ready.state).toBe("running");
|
||||
expect(ready.url).toBe(httpsUrl);
|
||||
expect(ready.exposureState).toBe("ready");
|
||||
// A ready exposure reports plainly and never as a remediation prompt.
|
||||
expect(ready.exposureDetail).toBe("HTTPS ready");
|
||||
expect(ready.exposureDetail).not.toMatch(/unavailable|cleanup|Check the/i);
|
||||
|
||||
const pending = buildEntry(createRuntimeService({
|
||||
status: "running",
|
||||
healthStatus: "healthy",
|
||||
port: 42_020,
|
||||
url: null,
|
||||
exposure: {
|
||||
provider: "tailscale_https",
|
||||
state: "pending",
|
||||
publicUrl: null,
|
||||
hostname: "paperclip-dev.tail29c1aa.ts.net",
|
||||
listeners: [],
|
||||
brokerRef: null,
|
||||
lastError: null,
|
||||
updatedAt: "2026-08-12T00:00:00.000Z",
|
||||
},
|
||||
}));
|
||||
expect(pending.url ?? null).toBeNull();
|
||||
expect(pending.exposureDetail).toBe("Provisioning HTTPS…");
|
||||
expect(JSON.stringify(pending)).not.toContain("http://");
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveWorkspaceServiceControlRequests", () => {
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import type {
|
||||
WorkspaceCommandDefinition,
|
||||
RuntimeExposureStatus,
|
||||
WorkspaceRuntimeControlTarget,
|
||||
WorkspaceRuntimeService,
|
||||
} from "@paperclipai/shared";
|
||||
|
|
@ -30,6 +31,7 @@ export type WorkspaceRuntimeControlItem = {
|
|||
statusLabel: string;
|
||||
lifecycle: "shared" | "ephemeral" | null;
|
||||
healthStatus: "unknown" | "healthy" | "unhealthy" | null;
|
||||
exposure: RuntimeExposureStatus | null;
|
||||
command: string | null;
|
||||
cwd: string | null;
|
||||
port: number | null;
|
||||
|
|
@ -96,6 +98,7 @@ function buildServiceItem(
|
|||
statusLabel: runtimeService?.status ?? "stopped",
|
||||
lifecycle: runtimeService?.lifecycle ?? command.lifecycle,
|
||||
healthStatus: runtimeService?.healthStatus ?? "unknown",
|
||||
exposure: runtimeService?.exposure ?? null,
|
||||
command: runtimeService?.command ?? command.command,
|
||||
cwd: runtimeService?.cwd ?? command.cwd,
|
||||
port: runtimeService?.port ?? null,
|
||||
|
|
@ -120,6 +123,7 @@ function buildJobItem(
|
|||
statusLabel: "run once",
|
||||
lifecycle: null,
|
||||
healthStatus: null,
|
||||
exposure: null,
|
||||
command: command.command,
|
||||
cwd: command.cwd,
|
||||
port: null,
|
||||
|
|
@ -169,6 +173,7 @@ export function buildWorkspaceRuntimeControlSections(input: {
|
|||
statusLabel: runtimeService.status,
|
||||
lifecycle: runtimeService.lifecycle,
|
||||
healthStatus: runtimeService.healthStatus,
|
||||
exposure: runtimeService.exposure ?? null,
|
||||
command: runtimeService.command ?? null,
|
||||
cwd: runtimeService.cwd ?? null,
|
||||
port: runtimeService.port ?? null,
|
||||
|
|
@ -213,6 +218,39 @@ function isActiveStatusLabel(statusLabel: string) {
|
|||
return statusLabel === "running" || statusLabel === "starting" || statusLabel === "provisioning";
|
||||
}
|
||||
|
||||
function exposureFailureCopy(exposure: RuntimeExposureStatus | null) {
|
||||
if (exposure?.state === "failed") {
|
||||
return {
|
||||
label: "HTTPS unavailable",
|
||||
remediation: "Check the Tailscale broker and node HTTPS configuration.",
|
||||
};
|
||||
}
|
||||
if (exposure?.state === "cleanup_pending") {
|
||||
return {
|
||||
label: "HTTPS cleanup pending",
|
||||
remediation: "Restart the host broker before reusing this port.",
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function ExposureFailureDetail({ exposure }: { exposure: RuntimeExposureStatus | null }) {
|
||||
const copy = exposureFailureCopy(exposure);
|
||||
if (!copy) return null;
|
||||
return (
|
||||
<div className="space-y-1 break-words text-xs text-destructive" role="alert">
|
||||
<div
|
||||
className="line-clamp-3 font-medium"
|
||||
title={exposure?.lastError ?? undefined}
|
||||
>
|
||||
{copy.label}
|
||||
{exposure?.lastError ? ` · ${exposure.lastError}` : ""}
|
||||
</div>
|
||||
<div>{copy.remediation}</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Maps runtime control sections onto the fixed-geometry service control bar
|
||||
* model. In-flight mutations overlay the transitional states (starting /
|
||||
|
|
@ -260,6 +298,15 @@ export function buildWorkspaceServiceControlEntries(input: {
|
|||
const failureDetail = state === "failed"
|
||||
? `Service failed${runtimeService?.stoppedAt ? ` · ${timeAgo(runtimeService.stoppedAt)}` : ""}`
|
||||
: null;
|
||||
const exposure = runtimeService?.exposure ?? item.exposure;
|
||||
const exposureFailure = exposureFailureCopy(exposure);
|
||||
const exposureDetail = exposure?.state === "pending"
|
||||
? "Provisioning HTTPS…"
|
||||
: exposure?.state === "ready"
|
||||
? "HTTPS ready"
|
||||
: exposureFailure
|
||||
? `${exposureFailure.label} · ${exposureFailure.remediation}`
|
||||
: null;
|
||||
|
||||
return {
|
||||
key: item.key,
|
||||
|
|
@ -269,6 +316,8 @@ export function buildWorkspaceServiceControlEntries(input: {
|
|||
url: item.url,
|
||||
port: item.port,
|
||||
failureDetail,
|
||||
exposureState: exposure?.state ?? null,
|
||||
exposureDetail,
|
||||
canStart: item.canStart,
|
||||
};
|
||||
});
|
||||
|
|
@ -456,6 +505,7 @@ function CommandSection({
|
|||
{item.cwd ? <div className="break-all font-mono">{item.cwd}</div> : null}
|
||||
{item.disabledReason ? <div>{item.disabledReason}</div> : null}
|
||||
</div>
|
||||
<ExposureFailureDetail exposure={item.exposure} />
|
||||
{item.healthStatus && item.statusLabel !== "stopped" ? (
|
||||
<div className="flex items-center gap-2">
|
||||
<Badge variant="outline" className={cn(
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
// @vitest-environment jsdom
|
||||
|
||||
import { act } from "react";
|
||||
import { flushSync } from "react-dom";
|
||||
import { createRoot, type Root } from "react-dom/client";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { WorkspaceServiceControlBar } from "./WorkspaceServiceControlBar";
|
||||
|
|
@ -8,6 +8,17 @@ import { WorkspaceServiceControlBar } from "./WorkspaceServiceControlBar";
|
|||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true;
|
||||
|
||||
async function act(callback: () => void | Promise<void>) {
|
||||
let pending: void | Promise<void>;
|
||||
flushSync(() => {
|
||||
pending = callback();
|
||||
});
|
||||
await pending!;
|
||||
await Promise.resolve();
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
flushSync(() => {});
|
||||
}
|
||||
|
||||
describe("WorkspaceServiceControlBar", () => {
|
||||
let container: HTMLDivElement;
|
||||
let root: Root;
|
||||
|
|
@ -105,4 +116,116 @@ describe("WorkspaceServiceControlBar", () => {
|
|||
expect(runningSegment).not.toBeNull();
|
||||
expect(runningSegment?.className).toBe(stoppedSegment?.className);
|
||||
});
|
||||
|
||||
it("links a managed HTTPS runtime at its tailnet URL and offers no link before it is verified", async () => {
|
||||
// PAP-17158: this bar renders the launch link on the workspace, project, and
|
||||
// issue surfaces. The href must be the verified tailnet HTTPS URL including
|
||||
// its non-standard port, and an unverified exposure must render no anchor at
|
||||
// all — an `http://` fallback link is the failure this feature prevents.
|
||||
const httpsUrl = "https://paperclip-dev.tail29c1aa.ts.net:42010";
|
||||
const renderExposed = async (url: string | null, exposureState: "ready" | "pending") => {
|
||||
await act(() => {
|
||||
root.render(
|
||||
<WorkspaceServiceControlBar
|
||||
services={[{
|
||||
key: "paperclip-dev",
|
||||
name: "paperclip-dev",
|
||||
state: "running",
|
||||
healthStatus: "healthy",
|
||||
url,
|
||||
port: 42_010,
|
||||
exposureState,
|
||||
}]}
|
||||
onAction={() => {}}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
};
|
||||
|
||||
await renderExposed(httpsUrl, "ready");
|
||||
const links = Array.from(container.querySelectorAll<HTMLAnchorElement>("a[href]"));
|
||||
expect(links.length).toBeGreaterThan(0);
|
||||
for (const link of links) expect(link.getAttribute("href")).toBe(httpsUrl);
|
||||
// The port is what makes a preview reachable, so it must stay visible.
|
||||
expect(container.textContent).toContain("paperclip-dev.tail29c1aa.ts.net:42010");
|
||||
// Scoped to hrefs, titles, and text: an SVG `xmlns` is not a launch link.
|
||||
const advertisedUrls = () => [
|
||||
...Array.from(container.querySelectorAll("a[href]"), (el) => el.getAttribute("href")),
|
||||
...Array.from(container.querySelectorAll("[title]"), (el) => el.getAttribute("title")),
|
||||
container.textContent,
|
||||
].join(" ");
|
||||
expect(advertisedUrls()).not.toContain("http://");
|
||||
|
||||
const copyButton = container.querySelector<HTMLButtonElement>('button[aria-label="Copy URL"]')!;
|
||||
await act(async () => {
|
||||
copyButton.click();
|
||||
await Promise.resolve();
|
||||
});
|
||||
expect(writeText).toHaveBeenCalledWith(httpsUrl);
|
||||
|
||||
// Exposure still pending: the backend is up but has no publishable URL yet.
|
||||
await renderExposed(null, "pending");
|
||||
expect(container.querySelectorAll("a[href]")).toHaveLength(0);
|
||||
expect(advertisedUrls()).not.toContain("http://");
|
||||
expect(container.textContent).toContain(":42010");
|
||||
});
|
||||
|
||||
it("renders HTTPS cleanup separately from process running state", async () => {
|
||||
await act(() => {
|
||||
root.render(
|
||||
<WorkspaceServiceControlBar
|
||||
services={[{
|
||||
key: "web",
|
||||
name: "Web",
|
||||
state: "running",
|
||||
healthStatus: "healthy",
|
||||
exposureState: "cleanup_pending",
|
||||
exposureDetail: "HTTPS cleanup pending · Restart the host broker before reusing this port.",
|
||||
}]}
|
||||
onAction={() => {}}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
|
||||
expect(container.textContent).toContain("Running");
|
||||
expect(container.textContent).toContain("HTTPS cleanup pending");
|
||||
expect(container.querySelector(".text-destructive")).not.toBeNull();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["failed", "HTTPS unavailable · Check the Tailscale broker and node HTTPS configuration."],
|
||||
["cleanup_pending", "HTTPS cleanup pending · Restart the host broker before reusing this port."],
|
||||
] as const)("keeps the full %s remediation visible in the multi-service popover", async (exposureState, exposureDetail) => {
|
||||
await act(() => {
|
||||
root.render(
|
||||
<WorkspaceServiceControlBar
|
||||
services={[
|
||||
{
|
||||
key: "web",
|
||||
name: "Web",
|
||||
state: "running",
|
||||
healthStatus: "healthy",
|
||||
exposureState,
|
||||
exposureDetail,
|
||||
},
|
||||
{
|
||||
key: "api",
|
||||
name: "API",
|
||||
state: "stopped",
|
||||
port: 42001,
|
||||
},
|
||||
]}
|
||||
defaultServicesOpen
|
||||
onAction={() => {}}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
|
||||
const detail = Array.from(document.body.querySelectorAll("span"))
|
||||
.find((element) => element.textContent === exposureDetail);
|
||||
expect(detail).toBeDefined();
|
||||
expect(detail?.classList.contains("text-destructive")).toBe(true);
|
||||
expect(detail?.classList.contains("truncate")).toBe(false);
|
||||
expect(detail?.classList.contains("whitespace-normal")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -35,6 +35,9 @@ export type WorkspaceServiceControlEntry = {
|
|||
port?: number | null;
|
||||
/** Short human-readable failure summary, e.g. "dev exited with code 1, 12s ago". */
|
||||
failureDetail?: string | null;
|
||||
/** HTTPS exposure lifecycle, intentionally separate from process health. */
|
||||
exposureState?: "pending" | "ready" | "failed" | "cleanup_pending" | "removed" | null;
|
||||
exposureDetail?: string | null;
|
||||
canStart?: boolean;
|
||||
};
|
||||
|
||||
|
|
@ -276,18 +279,20 @@ function ActionSlots({
|
|||
);
|
||||
}
|
||||
|
||||
function FailureDetail({
|
||||
function ServiceDetail({
|
||||
entry,
|
||||
onViewLogs,
|
||||
}: {
|
||||
entry: WorkspaceServiceControlEntry;
|
||||
onViewLogs?: () => void;
|
||||
}) {
|
||||
if (entry.state !== "failed" || !entry.failureDetail) return null;
|
||||
const detail = entry.exposureDetail ?? (entry.state === "failed" ? entry.failureDetail : null);
|
||||
if (!detail) return null;
|
||||
const exposureFailed = entry.exposureState === "failed" || entry.exposureState === "cleanup_pending";
|
||||
return (
|
||||
<div className="flex items-center gap-1 text-xs text-muted-foreground">
|
||||
<span>{entry.failureDetail}</span>
|
||||
{onViewLogs ? (
|
||||
<div className={cn("flex items-center gap-1 text-xs", exposureFailed ? "text-destructive" : "text-muted-foreground")}>
|
||||
<span>{detail}</span>
|
||||
{onViewLogs && entry.state === "failed" ? (
|
||||
<>
|
||||
<span aria-hidden>·</span>
|
||||
<button
|
||||
|
|
@ -339,7 +344,7 @@ function SingleServiceBar({
|
|||
<UrlSegment entry={entry} compact />
|
||||
</div>
|
||||
</div>
|
||||
<FailureDetail entry={entry} onViewLogs={onViewLogs} />
|
||||
<ServiceDetail entry={entry} onViewLogs={onViewLogs} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -354,8 +359,11 @@ function ServicePopoverRow({
|
|||
const meta = statusMeta(entry);
|
||||
const displayUrl = formatServiceUrl(entry.url);
|
||||
const live = entry.state === "running" && Boolean(entry.url);
|
||||
const secondary = live
|
||||
? displayUrl
|
||||
const exposureFailed = entry.exposureState === "failed" || entry.exposureState === "cleanup_pending";
|
||||
const secondary = entry.exposureDetail
|
||||
? entry.exposureDetail
|
||||
: live
|
||||
? displayUrl
|
||||
: entry.state === "starting" && entry.port
|
||||
? `starting on :${entry.port}…`
|
||||
: entry.state === "failed" && entry.failureDetail
|
||||
|
|
@ -382,7 +390,16 @@ function ServicePopoverRow({
|
|||
<CopyUrlButton url={entry.url} />
|
||||
</>
|
||||
) : (
|
||||
<span className="min-w-0 truncate text-xs text-muted-foreground">{secondary}</span>
|
||||
<span
|
||||
className={cn(
|
||||
"min-w-0 text-xs",
|
||||
exposureFailed
|
||||
? "whitespace-normal break-words text-destructive"
|
||||
: "truncate text-muted-foreground",
|
||||
)}
|
||||
>
|
||||
{secondary}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
Loading…
Reference in New Issue