fix(server): preserve hot restart intent across path upgrade (#10593)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server can preserve eligible agent runs during a controlled hot restart. > - A path change moved restart state from the Paperclip home root to the instance root. > - A staged update can therefore make the old server and the new server read different intent files. > - The old server then drains live runs, while the new server can start without a shutdown snapshot. > - This pull request adds a correlated compatibility handoff and records the live preflight set. > - It also verifies the target process instance on Linux, macOS, and Windows. > - The benefit is complete and safe run classification across the path upgrade. ## Linked Issues or Issue Description No public GitHub issue covers this defect. **What happened?** A staged hot restart can run an older server that reads `hot-restart-intent.json` from the Paperclip home root and a new server that writes the file under the instance root. The old server misses the request and uses graceful drain. The new server later finds its marker without a shutdown snapshot. Before this change, that state could produce an empty loss list even when live runs existed before restart. **Expected behavior** The old server must receive the PID-targeted restart request at its legacy path. The new server must correlate the legacy shutdown snapshot with its instance-scoped request. Every run that was live during preflight must appear as adopted, finalized while down, or lost. A reused PID must not let a stale marker claim a different process instance. **Steps to reproduce** 1. Start a server version from before the instance-root marker change. 2. Keep one or more local-agent heartbeat runs active. 3. Stage a current build and request a hot restart from that build. 4. Observe that the old server reads only the home-root path while the staged build writes only the instance-root path. 5. Observe graceful drain and a new-server intent that has no shutdown snapshot. **Paperclip version or commit** The path transition entered `master` in #10045. The hot-restart adoption flow came from #9647. This fix targets current `master` and compatibility with the immediately preceding home-root behavior. **Deployment mode** Self-hosted server built from source with controlled service hot restarts. Related work: #9628 is the original broader hot-restart feature PR. #10556 addresses embedded PostgreSQL lifecycle behavior and does not address marker-path compatibility. ## What Changed - Write an authoritative instance-scoped intent and a correlated legacy home-root handoff marker. - Merge a legacy shutdown snapshot only when immutable request identity fields match. - Prevent a non-default instance from consuming an uncorrelated legacy-only marker. - Record preflight running heartbeat IDs and reconcile snapshot omissions from current database state. - Serialize marker claims, snapshot writes, stale recovery, and matching cleanup with recoverable per-path filesystem leases. - Read process start identity on Linux, macOS, and Windows to distinguish a reused PID from the original server. - Require identity for new restart requests and fail closed when a supported platform cannot provide it. - Classify older markers by comparing the replacement server boot time or operating-system process start time with the request time. - Close the preflight database client explicitly and use a root-safe SQL query. - Add focused unit, platform-branch, database-backed, and CI regression coverage. - Document the compatibility handoff, process identity probes, and instance-scoped report path. ## Verification - `pnpm exec vitest run server/src/services/hot-restart.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts -t "hot-restart|old-server legacy|preflight live|preflight run|spawn identity before hot restart"` — 24 tests passed and 90 tests were skipped across 2 files. - `pnpm exec vitest run server/src/services/hot-restart.test.ts` — 17 tests passed. - `pnpm exec vitest run server/src/__tests__/issue-watchdogs-routes.test.ts -t "restarts a stalled claimed run"` — 1 test passed and 10 tests were skipped. - `pnpm exec vitest run server/src/__tests__/agent-action-audit-routes.test.ts -t "allows an agent with issue:delegate"` — 1 test passed and 7 tests were skipped. - `pnpm --filter @paperclipai/server typecheck` — passed. - `git diff --check` — passed. - GitHub Actions — 26 of 26 checks passed at `55a79cb029be8b1dc89926d9d89ccd2181266d5c`. - Greptile — 5/5 at the same head with no unresolved current-head review threads. ## Risks - The legacy handoff path is shared across instances. Exclusive claims and per-path leases prevent overwrite and match-before-delete races. - Process identity uses platform commands as a fallback when the health endpoint has no identity. Linux reads `/proc`, macOS and BSD use `ps`, and Windows uses PowerShell. - A supported-platform identity probe failure aborts the restart. This fails closed instead of replacing an unknown live process. - Older intent files do not contain process identity. The server compares the replacement boot or process start time with the request time when those values are available. - A preflight database read can fail before the marker is written. The command fails closed instead of claiming a restart whose live-run set is unknown. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with GPT-5. The exact deployment model ID and context-window size were not exposed by this runtime. Reasoning, repository editing, shell execution, test execution, GitHub CLI, and Paperclip API capabilities were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
GitHub
parent
ee9d907d01
commit
b163c6c473
|
|
@ -115,7 +115,27 @@ pnpm --filter @paperclipai/server exec tsx ../scripts/request-hot-restart.ts --s
|
|||
systemctl restart paperclip.service
|
||||
```
|
||||
|
||||
Use `--drain-required` only when the deploy intentionally requires the old terminate-and-retry behavior. Without that flag, the old server verifies that the marker targets its own PID, snapshots currently running heartbeat run IDs and child PIDs, and skips the shutdown drain so eligible detached local-agent processes can keep running. On startup the new server writes `$PAPERCLIP_HOME/hot-restart-report.json` with `previousServerPid`, `newServerPid`, `previousServerVersion`, `newServerVersion`, `adoptedRunIds`, `finalizedWhileDownRunIds`, `lostRunIds`, and per-run classifications before the normal orphan reaper runs.
|
||||
The staged command records the target server's boot identity and operating
|
||||
system process start time with the PID. It reads process metadata through
|
||||
`/proc` on Linux, `ps` on macOS and BSD, and PowerShell on Windows. These
|
||||
identities let a later request reclaim an abandoned marker after the operating
|
||||
system recycles the numeric PID. Older markers stay compatible and use process
|
||||
start metadata when available. When OS metadata is unavailable, the current
|
||||
server's health-reported boot time can still prove that a legacy marker predates
|
||||
the process now using its PID. Paperclip refuses to create a new request without
|
||||
at least one identity source. Supported-platform process probes fail explicitly
|
||||
instead of silently treating a live PID as either the original owner or a
|
||||
recycled process when identity cannot be established.
|
||||
|
||||
Use `--drain-required` only when the deploy intentionally requires the old terminate-and-retry behavior. Without that flag, the old server verifies that the marker targets its own PID, snapshots currently running heartbeat run IDs and child PIDs, and skips the shutdown drain so eligible detached local-agent processes can keep running. On startup the new server writes `$PAPERCLIP_HOME/instances/${PAPERCLIP_INSTANCE_ID:-default}/hot-restart-report.json` with `previousServerPid`, `newServerPid`, `previousServerVersion`, `newServerVersion`, `adoptedRunIds`, `finalizedWhileDownRunIds`, `lostRunIds`, and per-run classifications before the normal orphan reaper runs.
|
||||
|
||||
The request command records the preflight set of running heartbeat IDs and writes
|
||||
an instance-scoped marker plus a PID-targeted legacy home-root handoff marker.
|
||||
This lets a previous server version capture its snapshot at the old path while
|
||||
the new server correlates that snapshot back to the authoritative instance
|
||||
request. If any preflight run ID is absent from the shutdown snapshot, the
|
||||
startup report includes it in `lostRunIds`; a missing snapshot therefore cannot
|
||||
look like a zero-loss restart.
|
||||
|
||||
A healthy guarded deploy must compare the report against `/api/health` (`version` or `serverVersion`) and treat any `lostRunIds` entry as a continuity failure that needs recovery before marking deployment complete.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,6 @@
|
|||
#!/usr/bin/env -S node --import tsx
|
||||
import { createDb } from "../packages/db/src/index.js";
|
||||
import { loadConfig } from "../server/src/config.js";
|
||||
import {
|
||||
resolveHotRestartIntentPath,
|
||||
writeHotRestartIntent,
|
||||
|
|
@ -8,7 +10,7 @@ function usage(): never {
|
|||
console.error([
|
||||
"Usage: tsx scripts/request-hot-restart.ts --server-pid <pid> [--drain-required]",
|
||||
"",
|
||||
"Writes a one-shot hot-restart intent marker under PAPERCLIP_HOME.",
|
||||
"Writes an instance-scoped hot-restart intent plus a legacy home-root handoff marker.",
|
||||
].join("\n"));
|
||||
process.exit(2);
|
||||
}
|
||||
|
|
@ -47,37 +49,72 @@ function normalizeApiBase(raw: string | undefined) {
|
|||
return trimmed.replace(/\/+$/, "").replace(/\/api$/, "");
|
||||
}
|
||||
|
||||
async function readPreviousServerVersion() {
|
||||
async function readPreviousServerInfo() {
|
||||
const apiBase = normalizeApiBase(process.env.PAPERCLIP_API_URL);
|
||||
if (!apiBase) return null;
|
||||
if (!apiBase) return { version: null, identity: null };
|
||||
try {
|
||||
const apiKey = process.env.PAPERCLIP_API_KEY?.trim();
|
||||
const response = await fetch(`${apiBase}/api/health`, {
|
||||
headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : undefined,
|
||||
signal: AbortSignal.timeout(2_000),
|
||||
});
|
||||
if (!response.ok) return null;
|
||||
if (!response.ok) return { version: null, identity: null };
|
||||
const body = await response.json() as Record<string, unknown>;
|
||||
return typeof body.serverVersion === "string"
|
||||
? body.serverVersion
|
||||
: typeof body.version === "string"
|
||||
? body.version
|
||||
: null;
|
||||
const serverInfo = body.serverInfo && typeof body.serverInfo === "object"
|
||||
? body.serverInfo as Record<string, unknown>
|
||||
: null;
|
||||
return {
|
||||
version: typeof body.serverVersion === "string"
|
||||
? body.serverVersion
|
||||
: typeof body.version === "string"
|
||||
? body.version
|
||||
: null,
|
||||
identity: typeof serverInfo?.processStartedAt === "string"
|
||||
? serverInfo.processStartedAt
|
||||
: null,
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
return { version: null, identity: null };
|
||||
}
|
||||
}
|
||||
|
||||
async function readPreflightActiveRunIds() {
|
||||
const config = loadConfig();
|
||||
const dbUrl = process.env.DATABASE_URL?.trim()
|
||||
|| config.databaseUrl
|
||||
|| `postgres://paperclip:paperclip@127.0.0.1:${config.embeddedPostgresPort}/paperclip`;
|
||||
const db = createDb(dbUrl);
|
||||
try {
|
||||
const rows = await db.$client<{ id: string }[]>`
|
||||
SELECT id
|
||||
FROM heartbeat_runs
|
||||
WHERE status = 'running'
|
||||
`;
|
||||
return rows.map((row) => row.id);
|
||||
} finally {
|
||||
await db.$client.end({ timeout: 1 });
|
||||
}
|
||||
}
|
||||
|
||||
const { serverPid, drainRequired } = readArgs(process.argv.slice(2));
|
||||
const preflightActiveRunIds = drainRequired ? [] : await readPreflightActiveRunIds();
|
||||
const previousServerInfo = await readPreviousServerInfo();
|
||||
const intent = await writeHotRestartIntent({
|
||||
previousServerPid: serverPid,
|
||||
previousServerVersion: await readPreviousServerVersion(),
|
||||
previousServerIdentity: previousServerInfo.identity,
|
||||
previousServerVersion: previousServerInfo.version,
|
||||
drainRequired,
|
||||
requestedByRunId: process.env.PAPERCLIP_RUN_ID?.trim() || null,
|
||||
preflightActiveRunIds,
|
||||
});
|
||||
|
||||
console.log(JSON.stringify({
|
||||
status: "hot_restart_intent_written",
|
||||
intentPath: resolveHotRestartIntentPath(),
|
||||
previousServerPid: intent.previousServerPid,
|
||||
previousServerIdentity: intent.previousServerIdentity,
|
||||
previousServerStartedAt: intent.previousServerStartedAt,
|
||||
previousServerVersion: intent.previousServerVersion,
|
||||
drainRequired: intent.drainRequired,
|
||||
preflightActiveRunIds: intent.preflightActiveRunIds,
|
||||
}, null, 2));
|
||||
|
|
|
|||
|
|
@ -127,7 +127,7 @@ describePostgres("agent action audit routes", () => {
|
|||
})).get(`/api/companies/${company.id}/audit/agent-actions`);
|
||||
expect(boardResponse.status).toBe(403);
|
||||
expect(boardResponse.body.error).toContain("audit:view_agent_actions");
|
||||
});
|
||||
}, 30_000);
|
||||
|
||||
it("returns a client error for invalid audit query parameters", async () => {
|
||||
const { company } = await seed();
|
||||
|
|
|
|||
|
|
@ -185,8 +185,9 @@ describePg("decisionService", () => {
|
|||
});
|
||||
|
||||
it("expires a decision atomically instead of executing after its deadline", async () => {
|
||||
const created = await createCommentDecision("lenient", { expiresAt: new Date(Date.now() + 5) });
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
const created = await createCommentDecision("lenient");
|
||||
await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1) })
|
||||
.where(eq(decisions.id, created.id));
|
||||
await expect(service().decide({ id: created.id, optionId: "yes", decidedByUserId, userActor: boardActor() }))
|
||||
.rejects.toThrow("decision_expired");
|
||||
expect((await service().get(created.id))?.status).toBe("expired");
|
||||
|
|
|
|||
|
|
@ -106,6 +106,7 @@ import {
|
|||
} from "../services/heartbeat.ts";
|
||||
import {
|
||||
readHotRestartIntent,
|
||||
resolveLegacyHotRestartIntentPath,
|
||||
resolveHotRestartReportPath,
|
||||
writeHotRestartIntent,
|
||||
} from "../services/hot-restart.ts";
|
||||
|
|
@ -1489,6 +1490,153 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("adopts an old-server legacy snapshot written for a new instance-scoped marker", async () => {
|
||||
const child = spawnAliveProcess();
|
||||
childProcesses.add(child);
|
||||
expect(child.pid).toBeGreaterThan(0);
|
||||
const { companyId, agentId, issueId, runId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
processPid: child.pid ?? null,
|
||||
processGroupId: null,
|
||||
});
|
||||
|
||||
await withTempPaperclipHome(async (home) => {
|
||||
await writeHotRestartIntent({
|
||||
previousServerPid: process.pid,
|
||||
previousServerVersion: "old-home-root-version",
|
||||
requestedAt: new Date("2026-08-01T00:05:00.000Z"),
|
||||
requestedByRunId: "deploy-run",
|
||||
preflightActiveRunIds: [runId],
|
||||
});
|
||||
|
||||
// Simulate the previous binary: it reads and rewrites only the legacy
|
||||
// home-root marker, and its parser drops fields introduced by the new binary.
|
||||
const legacyPath = resolveLegacyHotRestartIntentPath(home);
|
||||
const legacyIntent = JSON.parse(await fs.readFile(legacyPath, "utf8")) as Record<string, unknown>;
|
||||
delete legacyIntent.preflightActiveRunIds;
|
||||
legacyIntent.shutdownSnapshot = {
|
||||
capturedAt: "2026-08-01T00:06:00.000Z",
|
||||
signal: "SIGTERM",
|
||||
activeRuns: [{
|
||||
runId,
|
||||
companyId,
|
||||
agentId,
|
||||
adapterType: "codex_local",
|
||||
status: "running",
|
||||
processPid: child.pid,
|
||||
processGroupId: null,
|
||||
issueId,
|
||||
}],
|
||||
};
|
||||
await fs.writeFile(legacyPath, `${JSON.stringify(legacyIntent, null, 2)}\n`, "utf8");
|
||||
|
||||
const mergedIntent = await readHotRestartIntent();
|
||||
expect(mergedIntent).toMatchObject({
|
||||
preflightActiveRunIds: [runId],
|
||||
shutdownSnapshot: {
|
||||
activeRuns: [expect.objectContaining({ runId, processPid: child.pid })],
|
||||
},
|
||||
});
|
||||
|
||||
const heartbeat = heartbeatService(db);
|
||||
const adoption = await heartbeat.reconcileHotRestartAdoption(
|
||||
new Date("2026-08-01T00:07:00.000Z"),
|
||||
);
|
||||
expect(adoption).toMatchObject({
|
||||
mode: "reported",
|
||||
adoptedRunIds: [runId],
|
||||
finalizedWhileDownRunIds: [],
|
||||
lostRunIds: [],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("reports preflight live runs as lost when the shutdown snapshot is missing", async () => {
|
||||
const { runId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
processPid: process.pid,
|
||||
processGroupId: null,
|
||||
});
|
||||
|
||||
await withTempPaperclipHome(async (home) => {
|
||||
await writeHotRestartIntent({
|
||||
previousServerPid: process.pid,
|
||||
previousServerVersion: "missing-snapshot-version",
|
||||
requestedAt: new Date("2026-08-01T01:05:00.000Z"),
|
||||
preflightActiveRunIds: [runId],
|
||||
});
|
||||
|
||||
const heartbeat = heartbeatService(db);
|
||||
const adoption = await heartbeat.reconcileHotRestartAdoption(
|
||||
new Date("2026-08-01T01:07:00.000Z"),
|
||||
);
|
||||
expect(adoption).toMatchObject({
|
||||
mode: "reported",
|
||||
adoptedRunIds: [],
|
||||
finalizedWhileDownRunIds: [],
|
||||
lostRunIds: [runId],
|
||||
});
|
||||
|
||||
const report = JSON.parse(
|
||||
await fs.readFile(resolveHotRestartReportPath(home), "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
expect(report).toMatchObject({
|
||||
adoptedRunIds: [],
|
||||
finalizedWhileDownRunIds: [],
|
||||
lostRunIds: [runId],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("reports a preflight run that finished before snapshot capture as finalized", async () => {
|
||||
const { runId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
processPid: process.pid,
|
||||
processGroupId: null,
|
||||
});
|
||||
|
||||
await withTempPaperclipHome(async (home) => {
|
||||
await writeHotRestartIntent({
|
||||
previousServerPid: process.pid,
|
||||
previousServerVersion: "preflight-race-version",
|
||||
requestedAt: new Date("2026-08-01T01:08:00.000Z"),
|
||||
preflightActiveRunIds: [runId],
|
||||
});
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
status: "succeeded",
|
||||
finishedAt: new Date("2026-08-01T01:08:01.000Z"),
|
||||
updatedAt: new Date("2026-08-01T01:08:01.000Z"),
|
||||
})
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
|
||||
const heartbeat = heartbeatService(db);
|
||||
const adoption = await heartbeat.reconcileHotRestartAdoption(
|
||||
new Date("2026-08-01T01:09:00.000Z"),
|
||||
);
|
||||
expect(adoption).toMatchObject({
|
||||
mode: "reported",
|
||||
adoptedRunIds: [],
|
||||
finalizedWhileDownRunIds: [runId],
|
||||
lostRunIds: [],
|
||||
});
|
||||
|
||||
const report = JSON.parse(
|
||||
await fs.readFile(resolveHotRestartReportPath(home), "utf8"),
|
||||
) as { runs?: Array<Record<string, unknown>> };
|
||||
expect(report.runs).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
runId,
|
||||
classification: "finalized_while_down",
|
||||
reason: "run_status_succeeded",
|
||||
}),
|
||||
]),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("persists codex_local spawn identity before hot restart and never loses the live run for missing metadata", async () => {
|
||||
let releaseAdapter: (() => void) | null = null;
|
||||
let spawnedPid: number | null = null;
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
|
|||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
agentRuntimeState,
|
||||
|
|
@ -10,6 +10,7 @@ import {
|
|||
agents,
|
||||
companies,
|
||||
companyMemberships,
|
||||
companySkills,
|
||||
createDb,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
|
|
@ -24,9 +25,35 @@ import {
|
|||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { runningProcesses } from "../adapters/index.ts";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { ensureHumanRoleDefaultGrants } from "../services/principal-access-compatibility.js";
|
||||
import { taskWatchdogService } from "../services/task-watchdogs.js";
|
||||
import { drainHeartbeatRunsToQuiescence } from "./helpers/drain-heartbeat-runs.js";
|
||||
|
||||
const mockAdapterExecute = vi.hoisted(() =>
|
||||
vi.fn(async () => ({
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
errorMessage: null,
|
||||
summary: "Issue watchdog route test run.",
|
||||
provider: "test",
|
||||
model: "test-model",
|
||||
})),
|
||||
);
|
||||
|
||||
vi.mock("../adapters/index.ts", async () => {
|
||||
const actual = await vi.importActual<typeof import("../adapters/index.ts")>("../adapters/index.ts");
|
||||
return {
|
||||
...actual,
|
||||
getServerAdapter: vi.fn(() => ({
|
||||
supportsLocalAgentJwt: false,
|
||||
execute: mockAdapterExecute,
|
||||
})),
|
||||
};
|
||||
});
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
|
@ -47,6 +74,9 @@ describeEmbeddedPostgres("issue watchdog routes", () => {
|
|||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
mockAdapterExecute.mockClear();
|
||||
runningProcesses.clear();
|
||||
await drainHeartbeatRunsToQuiescence(db, heartbeatService(db));
|
||||
await db.delete(activityLog);
|
||||
await db.delete(issueComments);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
|
|
@ -57,6 +87,7 @@ describeEmbeddedPostgres("issue watchdog routes", () => {
|
|||
await db.delete(issueWatchdogs);
|
||||
await db.delete(issues);
|
||||
await db.delete(agents);
|
||||
await db.delete(companySkills);
|
||||
await db.delete(principalPermissionGrants);
|
||||
await db.delete(companyMemberships);
|
||||
await db.delete(companies);
|
||||
|
|
|
|||
|
|
@ -263,6 +263,7 @@ import {
|
|||
touchHeartbeatRunRuntimeStatus,
|
||||
} from "./heartbeat-run-runtime-status.js";
|
||||
import {
|
||||
findMissingHotRestartSnapshotRunIds,
|
||||
readHotRestartIntent,
|
||||
removeHotRestartIntent,
|
||||
shouldHonorHotRestartIntentForProcess,
|
||||
|
|
@ -9721,12 +9722,19 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
|
||||
if (!intent.shutdownSnapshot) {
|
||||
logger.warn(
|
||||
{ previousServerPid: intent.previousServerPid },
|
||||
{
|
||||
previousServerPid: intent.previousServerPid,
|
||||
preflightActiveRunIds: intent.preflightActiveRunIds,
|
||||
},
|
||||
"hot-restart intent present but shutdown snapshot is missing; no runs can be adopted",
|
||||
);
|
||||
}
|
||||
const candidates = intent.shutdownSnapshot?.activeRuns ?? [];
|
||||
const currentRows = candidates.length > 0
|
||||
const missingSnapshotRunIds = findMissingHotRestartSnapshotRunIds(intent);
|
||||
const reconciliationRunIds = [
|
||||
...new Set([...candidates.map((run) => run.runId), ...missingSnapshotRunIds]),
|
||||
];
|
||||
const currentRows = reconciliationRunIds.length > 0
|
||||
? await db
|
||||
.select({
|
||||
run: heartbeatRuns,
|
||||
|
|
@ -9734,7 +9742,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
})
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
|
||||
.where(inArray(heartbeatRuns.id, candidates.map((run) => run.runId)))
|
||||
.where(inArray(heartbeatRuns.id, reconciliationRunIds))
|
||||
: [];
|
||||
const currentByRunId = new Map(currentRows.map((row) => [row.run.id, row]));
|
||||
|
||||
|
|
@ -9758,6 +9766,28 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
else skippedRunIds.push(candidate.runId);
|
||||
};
|
||||
|
||||
for (const runId of missingSnapshotRunIds) {
|
||||
const current = currentByRunId.get(runId);
|
||||
if (!current) {
|
||||
finalizedWhileDownRunIds.push(runId);
|
||||
continue;
|
||||
}
|
||||
|
||||
const candidate = toHotRestartIntentRun(current);
|
||||
if (current.run.status !== "running") {
|
||||
classify(candidate, "finalized_while_down", `run_status_${current.run.status}`);
|
||||
} else {
|
||||
classify(candidate, "lost", "missing_shutdown_snapshot");
|
||||
}
|
||||
}
|
||||
|
||||
if (lostRunIds.length > 0) {
|
||||
logger.error(
|
||||
{ previousServerPid: intent.previousServerPid, lostRunIds },
|
||||
"hot-restart shutdown snapshot omitted live preflight runs; reporting them as lost",
|
||||
);
|
||||
}
|
||||
|
||||
for (const candidate of candidates) {
|
||||
const current = currentByRunId.get(candidate.runId);
|
||||
if (!current) {
|
||||
|
|
@ -9868,7 +9898,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
skippedRunIds,
|
||||
runs: reportRuns,
|
||||
});
|
||||
await removeHotRestartIntent();
|
||||
await removeHotRestartIntent(undefined, intent);
|
||||
|
||||
logger.info(
|
||||
{
|
||||
|
|
@ -9877,6 +9907,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
adoptedRunIds,
|
||||
finalizedWhileDownRunIds,
|
||||
lostRunIds,
|
||||
missingSnapshotRunIds,
|
||||
skippedRunIds,
|
||||
},
|
||||
"hot-restart adoption report written",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,476 @@
|
|||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
findMissingHotRestartSnapshotRunIds,
|
||||
isObservedHotRestartTargetAlive,
|
||||
readHotRestartIntent,
|
||||
readProcessStartedAt,
|
||||
removeHotRestartIntent,
|
||||
resolveHotRestartIntentPath,
|
||||
resolveLegacyHotRestartIntentPath,
|
||||
writeHotRestartIntent,
|
||||
} from "./hot-restart.js";
|
||||
|
||||
const originalInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
|
||||
afterEach(() => {
|
||||
if (originalInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
|
||||
else process.env.PAPERCLIP_INSTANCE_ID = originalInstanceId;
|
||||
});
|
||||
|
||||
async function withTempHome<T>(fn: (homeDir: string) => Promise<T>) {
|
||||
const homeDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-hot-restart-paths-"));
|
||||
try {
|
||||
return await fn(homeDir);
|
||||
} finally {
|
||||
await fs.rm(homeDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
describe("hot-restart path compatibility", () => {
|
||||
it("reads Linux process start time from proc metadata", async () => {
|
||||
await expect(
|
||||
readProcessStartedAt(123, {
|
||||
platform: "linux",
|
||||
stat: async (target) => {
|
||||
expect(target).toBe("/proc/123");
|
||||
return {
|
||||
ctimeMs: Date.parse("2026-08-01T01:00:00.123Z"),
|
||||
};
|
||||
},
|
||||
}),
|
||||
).resolves.toBe("2026-08-01T01:00:00.123Z");
|
||||
});
|
||||
|
||||
it("reads macOS process start time through ps", async () => {
|
||||
await expect(
|
||||
readProcessStartedAt(456, {
|
||||
platform: "darwin",
|
||||
runCommand: async (command, args) => {
|
||||
expect([command, ...args]).toEqual([
|
||||
"ps",
|
||||
"-o",
|
||||
"lstart=",
|
||||
"-p",
|
||||
"456",
|
||||
]);
|
||||
return "Fri Aug 1 01:02:03 2026\n";
|
||||
},
|
||||
}),
|
||||
).resolves.toBe(new Date("Fri Aug 1 01:02:03 2026").toISOString());
|
||||
});
|
||||
|
||||
it("reads Windows process start time through PowerShell", async () => {
|
||||
await expect(
|
||||
readProcessStartedAt(789, {
|
||||
platform: "win32",
|
||||
runCommand: async (command, args) => {
|
||||
expect(command).toBe("powershell.exe");
|
||||
expect(args.at(-1)).toContain("Get-Process -Id 789");
|
||||
return "2026-08-01T01:02:03.456Z\r\n";
|
||||
},
|
||||
}),
|
||||
).resolves.toBe("2026-08-01T01:02:03.456Z");
|
||||
});
|
||||
|
||||
it("falls back from Windows PowerShell to pwsh", async () => {
|
||||
const commands: string[] = [];
|
||||
await expect(
|
||||
readProcessStartedAt(790, {
|
||||
platform: "win32",
|
||||
runCommand: async (command) => {
|
||||
commands.push(command);
|
||||
if (command === "powershell.exe") throw new Error("not installed");
|
||||
return "2026-08-01T01:02:04.000Z\n";
|
||||
},
|
||||
}),
|
||||
).resolves.toBe("2026-08-01T01:02:04.000Z");
|
||||
expect(commands).toEqual(["powershell.exe", "pwsh.exe"]);
|
||||
});
|
||||
|
||||
it("surfaces supported-platform process identity probe failures", async () => {
|
||||
await expect(readProcessStartedAt(791, {
|
||||
platform: "darwin",
|
||||
runCommand: async () => {
|
||||
throw new Error("ps unavailable");
|
||||
},
|
||||
})).rejects.toThrow("ps unavailable");
|
||||
});
|
||||
|
||||
it("distinguishes recycled PIDs and rejects unknown live claim classification", () => {
|
||||
const intent = {
|
||||
version: 1 as const,
|
||||
requestedAt: "2026-08-01T01:05:00.000Z",
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: "server-boot-a",
|
||||
previousServerStartedAt: "2026-08-01T01:00:00.000Z",
|
||||
previousServerVersion: "old-version",
|
||||
drainRequired: false,
|
||||
requestedByRunId: null,
|
||||
preflightActiveRunIds: [],
|
||||
};
|
||||
|
||||
expect(isObservedHotRestartTargetAlive(intent, {
|
||||
alive: true,
|
||||
startedAt: "2026-08-01T01:00:00.000Z",
|
||||
replacement: {
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: "server-boot-a",
|
||||
},
|
||||
})).toBe(true);
|
||||
expect(isObservedHotRestartTargetAlive(intent, {
|
||||
alive: true,
|
||||
startedAt: null,
|
||||
replacement: {
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: "server-boot-b",
|
||||
},
|
||||
})).toBe(false);
|
||||
|
||||
const legacyIntent = {
|
||||
...intent,
|
||||
previousServerIdentity: null,
|
||||
previousServerStartedAt: null,
|
||||
};
|
||||
expect(isObservedHotRestartTargetAlive(legacyIntent, {
|
||||
alive: true,
|
||||
startedAt: "2026-08-01T01:06:00.000Z",
|
||||
})).toBe(false);
|
||||
expect(() => isObservedHotRestartTargetAlive(legacyIntent, {
|
||||
alive: true,
|
||||
startedAt: null,
|
||||
})).toThrow("Cannot establish process identity");
|
||||
expect(isObservedHotRestartTargetAlive(legacyIntent, {
|
||||
alive: true,
|
||||
startedAt: null,
|
||||
replacement: {
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: "2026-08-01T01:04:00.000Z",
|
||||
},
|
||||
})).toBe(true);
|
||||
expect(isObservedHotRestartTargetAlive(legacyIntent, {
|
||||
alive: true,
|
||||
startedAt: null,
|
||||
replacement: {
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: "2026-08-01T01:06:00.000Z",
|
||||
},
|
||||
})).toBe(false);
|
||||
expect(isObservedHotRestartTargetAlive(legacyIntent, {
|
||||
alive: true,
|
||||
startedAt: null,
|
||||
replacement: {
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: null,
|
||||
previousServerStartedAt: "2026-08-01T01:04:00.000Z",
|
||||
},
|
||||
})).toBe(true);
|
||||
expect(isObservedHotRestartTargetAlive(legacyIntent, {
|
||||
alive: true,
|
||||
startedAt: null,
|
||||
replacement: {
|
||||
previousServerPid: 123,
|
||||
previousServerIdentity: null,
|
||||
previousServerStartedAt: "2026-08-01T01:06:00.000Z",
|
||||
},
|
||||
})).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses to create an unidentifiable process claim", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
await expect(writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: 2_147_483_647,
|
||||
previousServerStartedAt: null,
|
||||
})).rejects.toThrow("process start time are unavailable");
|
||||
|
||||
await expect(fs.stat(resolveHotRestartIntentPath(homeDir))).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
await expect(fs.stat(resolveLegacyHotRestartIntentPath(homeDir))).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("reclaims a live recycled PID when server boot identities differ", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
previousServerIdentity: "server-boot-a",
|
||||
requestedByRunId: "blue-deploy",
|
||||
});
|
||||
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "green";
|
||||
await expect(writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
previousServerIdentity: "server-boot-b",
|
||||
requestedByRunId: "green-deploy",
|
||||
})).resolves.toMatchObject({
|
||||
previousServerIdentity: "server-boot-b",
|
||||
requestedByRunId: "green-deploy",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("writes both paths but does not merge a legacy snapshot from another request", async () => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
|
||||
await withTempHome(async (homeDir) => {
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: 101,
|
||||
previousServerStartedAt: "2026-08-01T01:00:00.000Z",
|
||||
requestedAt: new Date("2026-08-01T02:00:00.000Z"),
|
||||
requestedByRunId: "blue-deploy",
|
||||
preflightActiveRunIds: ["blue-run"],
|
||||
});
|
||||
await expect(fs.stat(resolveHotRestartIntentPath(homeDir))).resolves.toBeDefined();
|
||||
await expect(fs.stat(resolveLegacyHotRestartIntentPath(homeDir))).resolves.toBeDefined();
|
||||
|
||||
const unrelatedLegacyIntent = {
|
||||
version: 1,
|
||||
requestedAt: "2026-08-01T02:00:01.000Z",
|
||||
previousServerPid: 202,
|
||||
previousServerVersion: "other-instance",
|
||||
drainRequired: false,
|
||||
requestedByRunId: "other-deploy",
|
||||
shutdownSnapshot: {
|
||||
capturedAt: "2026-08-01T02:00:02.000Z",
|
||||
signal: "SIGTERM",
|
||||
activeRuns: [],
|
||||
},
|
||||
};
|
||||
await fs.writeFile(
|
||||
resolveLegacyHotRestartIntentPath(homeDir),
|
||||
`${JSON.stringify(unrelatedLegacyIntent, null, 2)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
|
||||
await expect(readHotRestartIntent(homeDir)).resolves.toMatchObject({
|
||||
previousServerPid: 101,
|
||||
requestedByRunId: "blue-deploy",
|
||||
preflightActiveRunIds: ["blue-run"],
|
||||
});
|
||||
expect((await readHotRestartIntent(homeDir))?.shutdownSnapshot).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
it("does not let a non-default instance consume an uncorrelated legacy-only marker", async () => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "green";
|
||||
|
||||
await withTempHome(async (homeDir) => {
|
||||
await fs.writeFile(
|
||||
resolveLegacyHotRestartIntentPath(homeDir),
|
||||
`${JSON.stringify({
|
||||
version: 1,
|
||||
requestedAt: "2026-08-01T03:00:00.000Z",
|
||||
previousServerPid: 303,
|
||||
previousServerVersion: "legacy",
|
||||
drainRequired: false,
|
||||
requestedByRunId: null,
|
||||
})}\n`,
|
||||
"utf8",
|
||||
);
|
||||
|
||||
await expect(readHotRestartIntent(homeDir)).resolves.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
it("ignores a malformed legacy marker when the instance marker is valid", async () => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
|
||||
await withTempHome(async (homeDir) => {
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: 304,
|
||||
previousServerStartedAt: "2026-08-01T03:00:00.000Z",
|
||||
requestedAt: new Date("2026-08-01T03:30:00.000Z"),
|
||||
preflightActiveRunIds: ["blue-run"],
|
||||
});
|
||||
await fs.writeFile(resolveLegacyHotRestartIntentPath(homeDir), "not-json", "utf8");
|
||||
|
||||
await expect(readHotRestartIntent(homeDir)).resolves.toMatchObject({
|
||||
previousServerPid: 304,
|
||||
preflightActiveRunIds: ["blue-run"],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("does not overwrite another instance's active legacy handoff", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedAt: new Date("2026-08-01T03:40:00.000Z"),
|
||||
requestedByRunId: "blue-deploy",
|
||||
});
|
||||
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "green";
|
||||
await expect(writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: 502,
|
||||
previousServerStartedAt: "2026-08-01T03:00:00.000Z",
|
||||
requestedAt: new Date("2026-08-01T03:40:01.000Z"),
|
||||
requestedByRunId: "green-deploy",
|
||||
})).rejects.toMatchObject({ code: "EEXIST" });
|
||||
|
||||
const legacyIntent = JSON.parse(
|
||||
await fs.readFile(resolveLegacyHotRestartIntentPath(homeDir), "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
expect(legacyIntent).toMatchObject({
|
||||
previousServerPid: process.pid,
|
||||
requestedByRunId: "blue-deploy",
|
||||
});
|
||||
await expect(fs.stat(resolveHotRestartIntentPath(homeDir))).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("reclaims an abandoned legacy handoff after its target process exits", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: 2_147_483_647,
|
||||
previousServerStartedAt: "2026-08-01T03:00:00.000Z",
|
||||
requestedAt: new Date("2026-08-01T03:50:00.000Z"),
|
||||
requestedByRunId: "abandoned-deploy",
|
||||
});
|
||||
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "green";
|
||||
await expect(writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedAt: new Date("2026-08-01T03:51:00.000Z"),
|
||||
requestedByRunId: "green-deploy",
|
||||
})).resolves.toMatchObject({ requestedByRunId: "green-deploy" });
|
||||
|
||||
await expect(readHotRestartIntent(homeDir)).resolves.toMatchObject({
|
||||
previousServerPid: process.pid,
|
||||
requestedByRunId: "green-deploy",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("reclaims an abandoned handoff when its PID belongs to a newer process", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedAt: new Date("2020-01-01T00:00:00.000Z"),
|
||||
requestedByRunId: "expired-deploy",
|
||||
});
|
||||
const legacyPath = resolveLegacyHotRestartIntentPath(homeDir);
|
||||
const legacyIntent = JSON.parse(
|
||||
await fs.readFile(legacyPath, "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
delete legacyIntent.previousServerStartedAt;
|
||||
await fs.writeFile(
|
||||
legacyPath,
|
||||
`${JSON.stringify(legacyIntent, null, 2)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "green";
|
||||
await expect(writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedByRunId: "green-deploy",
|
||||
})).resolves.toMatchObject({ requestedByRunId: "green-deploy" });
|
||||
|
||||
await expect(readHotRestartIntent(homeDir)).resolves.toMatchObject({
|
||||
requestedByRunId: "green-deploy",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps an old handoff while its original target process is alive", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedAt: new Date(),
|
||||
requestedByRunId: "blue-deploy",
|
||||
});
|
||||
|
||||
vi.useFakeTimers({ now: Date.now() + 10 * 60_000 });
|
||||
try {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "green";
|
||||
await expect(writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedByRunId: "green-deploy",
|
||||
})).rejects.toMatchObject({ code: "EEXIST" });
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
it("does not let matching cleanup delete replacement intent markers", async () => {
|
||||
await withTempHome(async (homeDir) => {
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "blue";
|
||||
const abandonedIntent = await writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: 2_147_483_647,
|
||||
previousServerStartedAt: "2026-08-01T03:00:00.000Z",
|
||||
requestedAt: new Date("2026-08-01T03:55:00.000Z"),
|
||||
requestedByRunId: "abandoned-deploy",
|
||||
});
|
||||
|
||||
await Promise.all([
|
||||
removeHotRestartIntent(homeDir, abandonedIntent),
|
||||
writeHotRestartIntent({
|
||||
homeDir,
|
||||
previousServerPid: process.pid,
|
||||
requestedAt: new Date("2026-08-01T03:56:00.000Z"),
|
||||
requestedByRunId: "replacement-deploy",
|
||||
}),
|
||||
]);
|
||||
|
||||
const legacyIntent = JSON.parse(
|
||||
await fs.readFile(resolveLegacyHotRestartIntentPath(homeDir), "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
expect(legacyIntent).toMatchObject({ requestedByRunId: "replacement-deploy" });
|
||||
await expect(readHotRestartIntent(homeDir)).resolves.toMatchObject({
|
||||
requestedByRunId: "replacement-deploy",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("treats every preflight run omitted from the shutdown snapshot as missing", () => {
|
||||
expect(findMissingHotRestartSnapshotRunIds({
|
||||
version: 1,
|
||||
requestedAt: "2026-08-01T04:00:00.000Z",
|
||||
previousServerPid: 404,
|
||||
previousServerVersion: "old-version",
|
||||
drainRequired: false,
|
||||
requestedByRunId: null,
|
||||
preflightActiveRunIds: ["captured-run", "missing-run"],
|
||||
shutdownSnapshot: {
|
||||
capturedAt: "2026-08-01T04:00:01.000Z",
|
||||
signal: "SIGTERM",
|
||||
activeRuns: [{
|
||||
runId: "captured-run",
|
||||
companyId: "company",
|
||||
agentId: "agent",
|
||||
adapterType: "codex_local",
|
||||
status: "running",
|
||||
processPid: 405,
|
||||
processGroupId: null,
|
||||
issueId: "issue",
|
||||
}],
|
||||
},
|
||||
})).toEqual(["missing-run"]);
|
||||
});
|
||||
});
|
||||
|
|
@ -1,9 +1,20 @@
|
|||
import { execFile } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
import {
|
||||
resolvePaperclipHomeDir,
|
||||
resolvePaperclipInstanceId,
|
||||
resolvePaperclipInstanceRoot,
|
||||
} from "../home-paths.js";
|
||||
|
||||
export const HOT_RESTART_INTENT_FILENAME = "hot-restart-intent.json";
|
||||
export const HOT_RESTART_REPORT_FILENAME = "hot-restart-report.json";
|
||||
const HOT_RESTART_LOCK_SUFFIX = ".lock";
|
||||
const HOT_RESTART_LOCK_STALE_MS = 30_000;
|
||||
const HOT_RESTART_LOCK_TIMEOUT_MS = 10_000;
|
||||
|
||||
type ProcessCommandRunner = (command: string, args: string[]) => Promise<string>;
|
||||
type ProcessStatReader = (target: string) => Promise<{ ctimeMs: number }>;
|
||||
|
||||
export type HotRestartIntentRun = {
|
||||
runId: string;
|
||||
|
|
@ -20,9 +31,12 @@ export type HotRestartIntent = {
|
|||
version: 1;
|
||||
requestedAt: string;
|
||||
previousServerPid: number;
|
||||
previousServerIdentity?: string | null;
|
||||
previousServerStartedAt?: string | null;
|
||||
previousServerVersion: string | null;
|
||||
drainRequired: boolean;
|
||||
requestedByRunId: string | null;
|
||||
preflightActiveRunIds: string[];
|
||||
shutdownSnapshot?: {
|
||||
capturedAt: string;
|
||||
signal: "SIGINT" | "SIGTERM";
|
||||
|
|
@ -59,10 +73,18 @@ function resolveHotRestartPath(filename: string, homeDir?: string) {
|
|||
return path.join(resolvePaperclipInstanceRoot({ homeDir }), filename);
|
||||
}
|
||||
|
||||
function resolveLegacyHotRestartPath(filename: string, homeDir?: string) {
|
||||
return path.join(resolvePaperclipHomeDir(homeDir), filename);
|
||||
}
|
||||
|
||||
export function resolveHotRestartIntentPath(homeDir?: string) {
|
||||
return resolveHotRestartPath(HOT_RESTART_INTENT_FILENAME, homeDir);
|
||||
}
|
||||
|
||||
export function resolveLegacyHotRestartIntentPath(homeDir?: string) {
|
||||
return resolveLegacyHotRestartPath(HOT_RESTART_INTENT_FILENAME, homeDir);
|
||||
}
|
||||
|
||||
export function resolveHotRestartReportPath(homeDir?: string) {
|
||||
return resolveHotRestartPath(HOT_RESTART_REPORT_FILENAME, homeDir);
|
||||
}
|
||||
|
|
@ -74,6 +96,17 @@ async function writeJsonFileAtomic(filePath: string, value: unknown) {
|
|||
await fs.rename(tempPath, filePath);
|
||||
}
|
||||
|
||||
async function writeJsonFileExclusiveAtomic(filePath: string, value: unknown) {
|
||||
await fs.mkdir(path.dirname(filePath), { recursive: true });
|
||||
const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp`;
|
||||
await fs.writeFile(tempPath, `${JSON.stringify(value, null, 2)}\n`, "utf8");
|
||||
try {
|
||||
await fs.link(tempPath, filePath);
|
||||
} finally {
|
||||
await fs.unlink(tempPath).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
|
@ -90,6 +123,261 @@ function asBoolean(value: unknown): boolean {
|
|||
return value === true;
|
||||
}
|
||||
|
||||
function asDateString(value: unknown): string | null {
|
||||
const candidate = asString(value);
|
||||
if (!candidate) return null;
|
||||
const timestamp = Date.parse(candidate);
|
||||
return Number.isFinite(timestamp) ? new Date(timestamp).toISOString() : null;
|
||||
}
|
||||
|
||||
function asStringArray(value: unknown): string[] {
|
||||
if (!Array.isArray(value)) return [];
|
||||
return [...new Set(value.map(asString).filter((entry): entry is string => entry !== null))];
|
||||
}
|
||||
|
||||
function isProcessAlive(pid: number) {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
return (error as NodeJS.ErrnoException | undefined)?.code === "EPERM";
|
||||
}
|
||||
}
|
||||
|
||||
function runProcessCommand(command: string, args: string[]) {
|
||||
return new Promise<string>((resolve, reject) => {
|
||||
execFile(
|
||||
command,
|
||||
args,
|
||||
{
|
||||
encoding: "utf8",
|
||||
timeout: 1_500,
|
||||
windowsHide: true,
|
||||
},
|
||||
(error, stdout) => {
|
||||
if (error) reject(error);
|
||||
else resolve(stdout);
|
||||
},
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
export async function readProcessStartedAt(
|
||||
pid: number,
|
||||
options: {
|
||||
platform?: NodeJS.Platform;
|
||||
stat?: ProcessStatReader;
|
||||
runCommand?: ProcessCommandRunner;
|
||||
} = {},
|
||||
) {
|
||||
const platform = options.platform ?? process.platform;
|
||||
const stat = options.stat ?? fs.stat;
|
||||
const runCommand = options.runCommand ?? runProcessCommand;
|
||||
|
||||
if (platform === "linux") {
|
||||
const processStat = await stat(`/proc/${pid}`);
|
||||
return new Date(processStat.ctimeMs).toISOString();
|
||||
}
|
||||
|
||||
if (["darwin", "freebsd", "openbsd", "aix", "sunos"].includes(platform)) {
|
||||
const stdout = await runCommand("ps", ["-o", "lstart=", "-p", String(pid)]);
|
||||
const startedAt = asDateString(stdout.trim());
|
||||
if (!startedAt) {
|
||||
throw new Error(`Could not parse ${platform} process start time for PID ${pid}`);
|
||||
}
|
||||
return startedAt;
|
||||
}
|
||||
|
||||
if (platform === "win32") {
|
||||
const script = [
|
||||
`$process = Get-Process -Id ${pid} -ErrorAction Stop`,
|
||||
"$process.StartTime.ToUniversalTime().ToString('o')",
|
||||
].join("; ");
|
||||
let stdout: string;
|
||||
try {
|
||||
stdout = await runCommand("powershell.exe", [
|
||||
"-NoLogo",
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
script,
|
||||
]);
|
||||
} catch {
|
||||
stdout = await runCommand("pwsh.exe", [
|
||||
"-NoLogo",
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
script,
|
||||
]);
|
||||
}
|
||||
const startedAt = asDateString(stdout.trim());
|
||||
if (!startedAt) {
|
||||
throw new Error(`Could not parse Windows process start time for PID ${pid}`);
|
||||
}
|
||||
return startedAt;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export function isObservedHotRestartTargetAlive(
|
||||
intent: HotRestartIntent,
|
||||
observation: {
|
||||
alive: boolean;
|
||||
startedAt: string | null;
|
||||
replacement?: Pick<
|
||||
HotRestartIntent,
|
||||
"previousServerPid" | "previousServerIdentity" | "previousServerStartedAt"
|
||||
>;
|
||||
},
|
||||
) {
|
||||
if (!observation.alive) return false;
|
||||
|
||||
if (observation.replacement?.previousServerPid === intent.previousServerPid) {
|
||||
if (
|
||||
observation.replacement.previousServerIdentity
|
||||
&& intent.previousServerIdentity
|
||||
) {
|
||||
return observation.replacement.previousServerIdentity
|
||||
=== intent.previousServerIdentity;
|
||||
}
|
||||
|
||||
if (
|
||||
observation.replacement.previousServerIdentity
|
||||
&& !intent.previousServerIdentity
|
||||
) {
|
||||
const replacementStartedAt = Date.parse(
|
||||
observation.replacement.previousServerIdentity,
|
||||
);
|
||||
const requestedAt = Date.parse(intent.requestedAt);
|
||||
if (Number.isFinite(replacementStartedAt) && Number.isFinite(requestedAt)) {
|
||||
// The health endpoint's processStartedAt value is also the current
|
||||
// server boot identity. If that process started after an older marker
|
||||
// was requested, the shared numeric PID was necessarily recycled.
|
||||
return replacementStartedAt <= requestedAt;
|
||||
}
|
||||
}
|
||||
|
||||
if (!intent.previousServerIdentity) {
|
||||
const replacementStartedAt = observation.replacement.previousServerStartedAt
|
||||
? Date.parse(observation.replacement.previousServerStartedAt)
|
||||
: Number.NaN;
|
||||
const requestedAt = Date.parse(intent.requestedAt);
|
||||
if (Number.isFinite(replacementStartedAt) && Number.isFinite(requestedAt)) {
|
||||
return replacementStartedAt <= requestedAt;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const observedStartedAt = observation.startedAt
|
||||
? Date.parse(observation.startedAt)
|
||||
: Number.NaN;
|
||||
const recordedStartedAt = intent.previousServerStartedAt
|
||||
? Date.parse(intent.previousServerStartedAt)
|
||||
: Number.NaN;
|
||||
if (Number.isFinite(observedStartedAt) && Number.isFinite(recordedStartedAt)) {
|
||||
return observedStartedAt === recordedStartedAt;
|
||||
}
|
||||
|
||||
const requestedAt = Date.parse(intent.requestedAt);
|
||||
if (Number.isFinite(observedStartedAt) && Number.isFinite(requestedAt)) {
|
||||
// Older markers have no recorded process start. A process that started
|
||||
// after the request necessarily reused the marker's numeric PID.
|
||||
return observedStartedAt <= requestedAt;
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`Cannot establish process identity for live hot-restart target PID ${intent.previousServerPid}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function isOriginalServerProcessAlive(
|
||||
intent: HotRestartIntent,
|
||||
replacement: HotRestartIntent,
|
||||
) {
|
||||
const alive = isProcessAlive(intent.previousServerPid);
|
||||
const startedAt = alive
|
||||
? await readProcessStartedAt(intent.previousServerPid)
|
||||
: null;
|
||||
return isObservedHotRestartTargetAlive(intent, {
|
||||
alive,
|
||||
startedAt,
|
||||
replacement,
|
||||
});
|
||||
}
|
||||
|
||||
async function removeStaleHotRestartLock(lockDir: string) {
|
||||
let shouldRemove = false;
|
||||
try {
|
||||
const owner = JSON.parse(
|
||||
await fs.readFile(path.join(lockDir, "owner.json"), "utf8"),
|
||||
) as { pid?: unknown; createdAt?: unknown };
|
||||
const pid = typeof owner.pid === "number" ? owner.pid : 0;
|
||||
const createdAt = typeof owner.createdAt === "string"
|
||||
? Date.parse(owner.createdAt)
|
||||
: Number.NaN;
|
||||
const ageMs = Number.isFinite(createdAt)
|
||||
? Date.now() - createdAt
|
||||
: HOT_RESTART_LOCK_STALE_MS + 1;
|
||||
shouldRemove = !isProcessAlive(pid) || ageMs > HOT_RESTART_LOCK_STALE_MS;
|
||||
} catch {
|
||||
const stat = await fs.stat(lockDir).catch(() => null);
|
||||
shouldRemove = !stat
|
||||
|| Date.now() - stat.mtimeMs > HOT_RESTART_LOCK_STALE_MS;
|
||||
}
|
||||
if (!shouldRemove) return false;
|
||||
await fs.rm(lockDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
return true;
|
||||
}
|
||||
|
||||
async function acquireHotRestartPathLock(filePath: string) {
|
||||
const lockDir = `${filePath}${HOT_RESTART_LOCK_SUFFIX}`;
|
||||
const deadline = Date.now() + HOT_RESTART_LOCK_TIMEOUT_MS;
|
||||
while (true) {
|
||||
try {
|
||||
await fs.mkdir(lockDir);
|
||||
try {
|
||||
await fs.writeFile(
|
||||
path.join(lockDir, "owner.json"),
|
||||
`${JSON.stringify({ pid: process.pid, createdAt: new Date().toISOString() })}\n`,
|
||||
"utf8",
|
||||
);
|
||||
} catch (error) {
|
||||
await fs.rm(lockDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
return async () => {
|
||||
await fs.rm(lockDir, { recursive: true, force: true });
|
||||
};
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
|
||||
if (await removeStaleHotRestartLock(lockDir)) continue;
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`Timed out waiting for hot-restart compatibility lock at ${lockDir}`);
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function withHotRestartPathLock<T>(filePath: string, operation: () => Promise<T>) {
|
||||
await fs.mkdir(path.dirname(filePath), { recursive: true });
|
||||
const release = await acquireHotRestartPathLock(filePath);
|
||||
try {
|
||||
return await operation();
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
}
|
||||
|
||||
function isSameHotRestartRequest(left: HotRestartIntent, right: HotRestartIntent) {
|
||||
return left.requestedAt === right.requestedAt
|
||||
&& left.previousServerPid === right.previousServerPid
|
||||
&& left.drainRequired === right.drainRequired
|
||||
&& left.requestedByRunId === right.requestedByRunId;
|
||||
}
|
||||
|
||||
function parseRun(value: unknown): HotRestartIntentRun | null {
|
||||
if (!isRecord(value)) return null;
|
||||
const runId = asString(value.runId);
|
||||
|
|
@ -120,9 +408,12 @@ export function parseHotRestartIntent(value: unknown): HotRestartIntent | null {
|
|||
version: 1,
|
||||
requestedAt,
|
||||
previousServerPid,
|
||||
previousServerIdentity: asString(value.previousServerIdentity),
|
||||
previousServerStartedAt: asDateString(value.previousServerStartedAt),
|
||||
previousServerVersion: asString(value.previousServerVersion),
|
||||
drainRequired: asBoolean(value.drainRequired),
|
||||
requestedByRunId: asString(value.requestedByRunId),
|
||||
preflightActiveRunIds: asStringArray(value.preflightActiveRunIds),
|
||||
};
|
||||
|
||||
const snapshot = isRecord(value.shutdownSnapshot) ? value.shutdownSnapshot : null;
|
||||
|
|
@ -140,9 +431,9 @@ export function parseHotRestartIntent(value: unknown): HotRestartIntent | null {
|
|||
return intent;
|
||||
}
|
||||
|
||||
export async function readHotRestartIntent(homeDir?: string) {
|
||||
async function readHotRestartIntentAtPath(filePath: string) {
|
||||
try {
|
||||
const raw = await fs.readFile(resolveHotRestartIntentPath(homeDir), "utf8");
|
||||
const raw = await fs.readFile(filePath, "utf8");
|
||||
return parseHotRestartIntent(JSON.parse(raw));
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
|
||||
|
|
@ -150,23 +441,85 @@ export async function readHotRestartIntent(homeDir?: string) {
|
|||
}
|
||||
}
|
||||
|
||||
export async function readHotRestartIntent(homeDir?: string) {
|
||||
const instanceIntent = await readHotRestartIntentAtPath(resolveHotRestartIntentPath(homeDir));
|
||||
let legacyIntent: HotRestartIntent | null;
|
||||
try {
|
||||
legacyIntent = await readHotRestartIntentAtPath(resolveLegacyHotRestartIntentPath(homeDir));
|
||||
} catch (error) {
|
||||
if (instanceIntent) return instanceIntent;
|
||||
throw error;
|
||||
}
|
||||
|
||||
if (!instanceIntent) {
|
||||
// The home-root marker predates instances. Only the default instance may
|
||||
// consume it without an instance-root request to correlate against.
|
||||
return resolvePaperclipInstanceId() === "default" ? legacyIntent : null;
|
||||
}
|
||||
if (!legacyIntent || !isSameHotRestartRequest(instanceIntent, legacyIntent)) {
|
||||
return instanceIntent;
|
||||
}
|
||||
|
||||
// A pre-instance server rewrites only the legacy marker when it captures
|
||||
// its shutdown snapshot. Preserve the instance-scoped request fields while
|
||||
// importing that snapshot only after the immutable request identity matches.
|
||||
return legacyIntent.shutdownSnapshot
|
||||
? { ...instanceIntent, shutdownSnapshot: legacyIntent.shutdownSnapshot }
|
||||
: instanceIntent;
|
||||
}
|
||||
|
||||
export function findMissingHotRestartSnapshotRunIds(intent: HotRestartIntent) {
|
||||
const snapshotRunIds = new Set(intent.shutdownSnapshot?.activeRuns.map((run) => run.runId) ?? []);
|
||||
return intent.preflightActiveRunIds.filter((runId) => !snapshotRunIds.has(runId));
|
||||
}
|
||||
|
||||
export async function writeHotRestartIntent(input: {
|
||||
previousServerPid: number;
|
||||
previousServerIdentity?: string | null;
|
||||
previousServerStartedAt?: string | null;
|
||||
previousServerVersion?: string | null;
|
||||
drainRequired?: boolean;
|
||||
requestedByRunId?: string | null;
|
||||
preflightActiveRunIds?: string[];
|
||||
requestedAt?: Date;
|
||||
homeDir?: string;
|
||||
}) {
|
||||
const previousServerStartedAt = input.previousServerStartedAt === undefined
|
||||
? await readProcessStartedAt(input.previousServerPid)
|
||||
: asDateString(input.previousServerStartedAt);
|
||||
const previousServerIdentity = asString(input.previousServerIdentity);
|
||||
if (!previousServerIdentity && !previousServerStartedAt) {
|
||||
throw new Error(
|
||||
`Cannot create hot-restart intent for PID ${input.previousServerPid}: `
|
||||
+ "server boot identity and operating-system process start time are unavailable",
|
||||
);
|
||||
}
|
||||
const intent: HotRestartIntent = {
|
||||
version: 1,
|
||||
requestedAt: (input.requestedAt ?? new Date()).toISOString(),
|
||||
previousServerPid: input.previousServerPid,
|
||||
previousServerIdentity,
|
||||
previousServerStartedAt,
|
||||
previousServerVersion: input.previousServerVersion ?? null,
|
||||
drainRequired: input.drainRequired ?? false,
|
||||
requestedByRunId: input.requestedByRunId ?? null,
|
||||
preflightActiveRunIds: asStringArray(input.preflightActiveRunIds),
|
||||
};
|
||||
await writeJsonFileAtomic(resolveHotRestartIntentPath(input.homeDir), intent);
|
||||
const instancePath = resolveHotRestartIntentPath(input.homeDir);
|
||||
const legacyPath = resolveLegacyHotRestartIntentPath(input.homeDir);
|
||||
// The legacy location is shared by every instance under PAPERCLIP_HOME.
|
||||
// Claim it without replacement so concurrent staged restarts fail closed
|
||||
// instead of making the first old server consume another instance's PID.
|
||||
await withHotRestartPathLock(legacyPath, () => claimLegacyHotRestartIntent(legacyPath, intent));
|
||||
try {
|
||||
await withHotRestartPathLock(instancePath, () => writeJsonFileAtomic(instancePath, intent));
|
||||
} catch (error) {
|
||||
await withHotRestartPathLock(
|
||||
legacyPath,
|
||||
() => removeMatchingHotRestartIntent(legacyPath, intent),
|
||||
).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
return intent;
|
||||
}
|
||||
|
||||
|
|
@ -185,7 +538,15 @@ export async function writeHotRestartShutdownSnapshot(input: {
|
|||
activeRuns: input.activeRuns,
|
||||
},
|
||||
};
|
||||
await writeJsonFileAtomic(resolveHotRestartIntentPath(input.homeDir), updated);
|
||||
const instancePath = resolveHotRestartIntentPath(input.homeDir);
|
||||
await withHotRestartPathLock(instancePath, () => writeJsonFileAtomic(instancePath, updated));
|
||||
const legacyPath = resolveLegacyHotRestartIntentPath(input.homeDir);
|
||||
await withHotRestartPathLock(legacyPath, async () => {
|
||||
const legacyIntent = await readHotRestartIntentAtPath(legacyPath).catch(() => null);
|
||||
if (legacyIntent && isSameHotRestartRequest(legacyIntent, input.intent)) {
|
||||
await writeJsonFileAtomic(legacyPath, updated);
|
||||
}
|
||||
});
|
||||
return updated;
|
||||
}
|
||||
|
||||
|
|
@ -194,14 +555,54 @@ export async function writeHotRestartReport(report: HotRestartReport, homeDir?:
|
|||
return report;
|
||||
}
|
||||
|
||||
export async function removeHotRestartIntent(homeDir?: string) {
|
||||
async function removeMatchingHotRestartIntent(filePath: string, expected?: HotRestartIntent) {
|
||||
try {
|
||||
await fs.unlink(resolveHotRestartIntentPath(homeDir));
|
||||
if (expected) {
|
||||
const current = await readHotRestartIntentAtPath(filePath);
|
||||
if (!current || !isSameHotRestartRequest(current, expected)) return;
|
||||
}
|
||||
await fs.unlink(filePath);
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function claimLegacyHotRestartIntent(filePath: string, intent: HotRestartIntent) {
|
||||
try {
|
||||
await writeJsonFileExclusiveAtomic(filePath, intent);
|
||||
return;
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
|
||||
|
||||
const existing = await readHotRestartIntentAtPath(filePath).catch(() => null);
|
||||
if (
|
||||
!existing
|
||||
|| await isOriginalServerProcessAlive(existing, intent)
|
||||
) {
|
||||
throw error;
|
||||
}
|
||||
|
||||
// An interrupted restart can leave the shared claim behind after its
|
||||
// target server exits. Remove only that exact abandoned request, then
|
||||
// compete normally for a fresh exclusive claim.
|
||||
await removeMatchingHotRestartIntent(filePath, existing);
|
||||
await writeJsonFileExclusiveAtomic(filePath, intent);
|
||||
}
|
||||
}
|
||||
|
||||
export async function removeHotRestartIntent(homeDir?: string, expected?: HotRestartIntent) {
|
||||
const instancePath = resolveHotRestartIntentPath(homeDir);
|
||||
await withHotRestartPathLock(
|
||||
instancePath,
|
||||
() => removeMatchingHotRestartIntent(instancePath, expected),
|
||||
);
|
||||
const legacyPath = resolveLegacyHotRestartIntentPath(homeDir);
|
||||
await withHotRestartPathLock(
|
||||
legacyPath,
|
||||
() => removeMatchingHotRestartIntent(legacyPath, expected),
|
||||
);
|
||||
}
|
||||
|
||||
export function shouldHonorHotRestartIntentForProcess(
|
||||
intent: HotRestartIntent,
|
||||
pid = process.pid,
|
||||
|
|
|
|||
Loading…
Reference in New Issue