diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index d4266b9395..929f5a8023 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -307,6 +307,10 @@ A workspace-coherent adapter path means: - the effective cwd exists or is provider-reachable, according to the workspace provider - when the adapter or workspace strategy relies on git state, the cwd is git-valid for the selected workspace: it resolves to the expected repository root, required base refs or branch metadata can be resolved, and runtime-created worktrees are still registered or explicitly recoverable +Adapter-backed liveness also requires control-plane reachability from the agent's actual mutation surface, not just from the host adapter process. If the agent is expected to use Bash, shell tools, runtime helpers, or in-sandbox command execution to update issues, create comments, upload artifacts, or submit review decisions, the `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY` visible to that surface must route to Paperclip successfully. + +For sandbox-backed local adapters, Paperclip may satisfy that contract with a run-scoped in-sandbox bridge. The host adapter keeps the real run JWT on the host side, injects only the bridge URL/token into the sandbox tool environment, and forwards allowed Paperclip API requests with the run id attached. The bridge credentials are execution plumbing, not user-facing context: they must not be written into prompts, visible comments, issue documents, restored workspace files, or durable logs. Agents and skills must use the env vars available in Bash/curl rather than assuming that the host's localhost API URL is reachable from browser or web-extraction tools inside the sandbox. + The state `projectWorkspaceId` plus `executionWorkspaceId` without `projectId` is invalid for project-scoped execution. Paperclip may treat it as recoverable only when it can derive exactly one owning project from the execution workspace, project workspace, or source issue in the same company and then repair the persisted state before delivery. If the owning project is missing, ambiguous, or cross-company, the queued adapter run must not be counted as a live path. Workspace incoherence feeds into the same non-terminal liveness and stranded assigned-work model as a disappeared run. The recovery path should first fail or reject the incoherent wake, then either repair and requeue one bounded continuation for the same assignee or surface an explicit recovery action. It must not leave an agent-owned `in_progress` issue healthy solely because a wake record exists that would invoke the adapter in the wrong cwd, a non-git directory where git is required, an unrelated project workspace, or an unrecoverable missing worktree. @@ -398,6 +402,8 @@ Agent-assigned `in_review` with no typed participant is only healthy when one of An `in_review` issue is stalled when it has no typed participant, no pending interaction or approval, no user owner, no active monitor, no active run, no queued wake, and no explicit recovery action. Paperclip should surface that state as recovery work rather than silently completing the issue or leaving blocker chains parked indefinitely. +When an execution-policy review stage has a pending agent participant, the participant's run is part of the review path only while it is live or queued. If that participant run reaches a terminal state while `executionState.status` remains `pending`, no decision has been recorded. Paperclip should queue one bounded normal-model recovery wake for the same participant when the agent is invokable and no other review path exists. If that recovery run also finishes while the stage remains pending, or the participant cannot be invoked, Paperclip must move the source issue to an explicit blocked/recovery path instead of leaving `in_review` to drift silently. + ### Issue monitors An issue monitor is a one-shot deferred action path for agent-owned issues in `in_progress` or `in_review`. diff --git a/packages/adapter-utils/src/execution-target-sandbox.test.ts b/packages/adapter-utils/src/execution-target-sandbox.test.ts index 92f964bcd1..0e7fac90fb 100644 --- a/packages/adapter-utils/src/execution-target-sandbox.test.ts +++ b/packages/adapter-utils/src/execution-target-sandbox.test.ts @@ -1,5 +1,5 @@ import { createServer } from "node:http"; -import { mkdir, mkdtemp, rm } from "node:fs/promises"; +import { mkdir, mkdtemp, readdir, readFile, rm } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -17,6 +17,7 @@ import { type AdapterSandboxExecutionTarget, } from "./execution-target.js"; import { runChildProcess } from "./server-utils.js"; +import { shellQuote } from "./ssh.js"; describe("sandbox adapter execution targets", () => { const cleanupDirs: string[] = []; @@ -60,6 +61,20 @@ describe("sandbox adapter execution targets", () => { }; } + async function readRuntimeTextFiles(rootDir: string): Promise { + const entries = await readdir(rootDir, { withFileTypes: true }).catch(() => []); + const contents: string[] = []; + for (const entry of entries) { + const entryPath = path.join(rootDir, entry.name); + if (entry.isDirectory()) { + contents.push(...await readRuntimeTextFiles(entryPath)); + } else if (entry.isFile()) { + contents.push(await readFile(entryPath, "utf8").catch(() => "")); + } + } + return contents; + } + it("executes through the provider-neutral runner without a remote spec", async () => { const runner = { execute: vi.fn(async () => ({ @@ -449,6 +464,120 @@ describe("sandbox adapter execution targets", () => { } }); + it("exposes the Paperclip bridge to the sandbox shell surface", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-execution-target-bridge-shell-")); + cleanupDirs.push(rootDir); + const remoteCwd = path.join(rootDir, "workspace"); + const runtimeRootDir = path.join(remoteCwd, ".paperclip-runtime", "claude"); + await mkdir(runtimeRootDir, { recursive: true }); + + const requests: Array<{ method: string; url: string; auth: string | null; runId: string | null }> = []; + const apiServer = createServer((req, res) => { + requests.push({ + method: req.method ?? "GET", + url: req.url ?? "/", + auth: req.headers.authorization ?? null, + runId: typeof req.headers["x-paperclip-run-id"] === "string" ? req.headers["x-paperclip-run-id"] : null, + }); + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ ok: true })); + }); + await new Promise((resolve, reject) => { + apiServer.once("error", reject); + apiServer.listen(0, "127.0.0.1", () => resolve()); + }); + const address = apiServer.address(); + if (!address || typeof address === "string") { + throw new Error("Expected the bridge shell test API server to listen on a TCP port."); + } + + const delegateRunner = createLocalSandboxRunner(); + const runner = { + execute: vi.fn(async (input: Parameters[0]) => delegateRunner.execute(input)), + }; + const target: AdapterSandboxExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "daytona", + environmentId: "env-1", + leaseId: "lease-1", + remoteCwd, + runner, + timeoutMs: 30_000, + }; + + const bridge = await startAdapterExecutionTargetPaperclipBridge({ + runId: "run-bridge-shell", + target, + runtimeRootDir, + adapterKey: "claude", + hostApiToken: "real-run-jwt", + hostApiUrl: `http://127.0.0.1:${address.port}`, + }); + try { + expect(bridge).not.toBeNull(); + const shellProbe = [ + "const url = `${process.env.PAPERCLIP_API_URL}/api/agents/me`;", + "fetch(url, { headers: { authorization: `Bearer ${process.env.PAPERCLIP_API_KEY}`, accept: 'application/json' } })", + " .then(async (response) => {", + " const body = await response.json();", + " process.stdout.write(JSON.stringify({", + " status: response.status,", + " body,", + " bridgeMode: process.env.PAPERCLIP_API_BRIDGE_MODE,", + " }));", + " })", + " .catch((error) => {", + " console.error(error instanceof Error ? error.stack : String(error));", + " process.exit(1);", + " });", + ].join("\n"); + + const result = await runAdapterExecutionTargetShellCommand( + "run-bridge-shell", + target, + `${shellQuote(process.execPath)} -e ${shellQuote(shellProbe)}`, + { + cwd: remoteCwd, + env: bridge!.env, + timeoutSec: 15, + graceSec: 5, + onLog: async () => {}, + }, + ); + + expect(result.exitCode).toBe(0); + expect(result.stderr).toBe(""); + expect(JSON.parse(result.stdout)).toEqual({ + status: 200, + body: { ok: true }, + bridgeMode: "queue_v1", + }); + expect(`${result.stdout}\n${result.stderr}`).not.toContain("real-run-jwt"); + expect(`${result.stdout}\n${result.stderr}`).not.toContain(bridge!.env.PAPERCLIP_API_KEY); + const runnerCommandText = JSON.stringify( + runner.execute.mock.calls.map(([call]) => ({ + command: call.command, + args: call.args, + })), + ); + expect(runnerCommandText).not.toContain("real-run-jwt"); + expect(runnerCommandText).not.toContain(bridge!.env.PAPERCLIP_API_KEY); + const runtimeFiles = (await readRuntimeTextFiles(runtimeRootDir)).join("\n"); + expect(runtimeFiles).not.toContain("real-run-jwt"); + expect(runtimeFiles).not.toContain(bridge!.env.PAPERCLIP_API_KEY); + expect(requests).toEqual([{ + method: "GET", + url: "/api/agents/me", + auth: "Bearer real-run-jwt", + runId: "run-bridge-shell", + }]); + } finally { + await bridge?.stop(); + await new Promise((resolve) => apiServer.close(() => resolve())); + } + }); + it("uses the effective adapter timeout when starting the sandbox callback bridge", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-execution-target-bridge-timeout-")); cleanupDirs.push(rootDir); diff --git a/packages/adapter-utils/src/sandbox-callback-bridge.ts b/packages/adapter-utils/src/sandbox-callback-bridge.ts index c673e80efd..6531aacfe9 100644 --- a/packages/adapter-utils/src/sandbox-callback-bridge.ts +++ b/packages/adapter-utils/src/sandbox-callback-bridge.ts @@ -914,8 +914,7 @@ export async function startSandboxCallbackBridgeServer(input: { [ `mkdir -p ${shellQuote(directories.requestsDir)} ${shellQuote(directories.responsesDir)} ${shellQuote(directories.logsDir)}`, `rm -f ${shellQuote(directories.readyFile)} ${shellQuote(directories.pidFile)}`, - `nohup env ${Object.entries(env).map(([key, value]) => `${key}=${shellQuote(value)}`).join(" ")} ` + - `${shellQuote(nodeCommand)} ${shellQuote(remoteEntrypoint)} ` + + `nohup ${shellQuote(nodeCommand)} ${shellQuote(remoteEntrypoint)} ` + `>> ${shellQuote(directories.logFile)} 2>&1 < /dev/null &`, "pid=$!", `printf '%s\\n' \"$pid\" > ${shellQuote(directories.pidFile)}`, @@ -925,6 +924,7 @@ export async function startSandboxCallbackBridgeServer(input: { cwd: input.remoteCwd, env: { [SANDBOX_EXEC_CHANNEL_ENV]: SANDBOX_EXEC_CHANNEL_BRIDGE, + ...env, }, timeoutMs, }); diff --git a/server/src/__tests__/file-resources.test.ts b/server/src/__tests__/file-resources.test.ts index 8064f4bcf0..3eddcf8d7b 100644 --- a/server/src/__tests__/file-resources.test.ts +++ b/server/src/__tests__/file-resources.test.ts @@ -63,7 +63,8 @@ async function seedGraph(db: Db, input: { projectSourceType?: string; targetProjectSourceType?: string; }): Promise { - const suffix = crypto.randomUUID().slice(0, 8); + const suffix = crypto.randomUUID().replace(/-/g, "").slice(0, 12); + const prefixSuffix = suffix.toUpperCase(); const companyId = crypto.randomUUID(); const otherCompanyId = crypto.randomUUID(); const goalId = crypto.randomUUID(); @@ -79,8 +80,8 @@ async function seedGraph(db: Db, input: { const otherIssueId = crypto.randomUUID(); await db.insert(companies).values([ - { id: companyId, name: `Company ${suffix}`, issuePrefix: `F${suffix.slice(0, 4).toUpperCase()}` }, - { id: otherCompanyId, name: `Other ${suffix}`, issuePrefix: `G${suffix.slice(0, 4).toUpperCase()}` }, + { id: companyId, name: `Company ${suffix}`, issuePrefix: `F${prefixSuffix}` }, + { id: otherCompanyId, name: `Other ${suffix}`, issuePrefix: `G${prefixSuffix}` }, ]); await db.insert(goals).values([ { id: goalId, companyId, title: "Goal", level: "company", status: "active" }, diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index af0c89a8bd..2d7b3b016e 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -644,7 +644,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { async function seedStrandedIssueFixture(input: { status: "todo" | "in_progress"; runStatus: "failed" | "timed_out" | "cancelled" | "succeeded"; - retryReason?: "assignment_recovery" | "issue_continuation_needed" | null; + retryReason?: "assignment_recovery" | "issue_continuation_needed" | "execution_review_participant_recovery" | null; runSource?: string | null; assignToUser?: boolean; activePauseHold?: boolean; @@ -769,6 +769,103 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { return { companyId, agentId, runId, wakeupRequestId, issueId, rootIssueId }; } + async function seedInReviewParticipantRunFixture(input?: { + wakeReason?: string; + retryReason?: string | null; + }) { + const companyId = randomUUID(); + const agentId = randomUUID(); + const runId = randomUUID(); + const wakeupRequestId = randomUUID(); + const issueId = randomUUID(); + const stageId = randomUUID(); + const now = new Date("2026-03-19T00:00:00.000Z"); + const issuePrefix = `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`; + const wakeReason = input?.wakeReason ?? "execution_review_requested"; + + await db.insert(companies).values({ + id: companyId, + name: "Paperclip", + issuePrefix, + requireBoardApprovalForNewAgents: false, + }); + + await db.insert(agents).values({ + id: agentId, + companyId, + name: "CodexReviewer", + role: "engineer", + status: "idle", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + + await db.insert(agentWakeupRequests).values({ + id: wakeupRequestId, + companyId, + agentId, + source: "automation", + triggerDetail: "system", + reason: wakeReason, + payload: { + issueId, + ...(input?.retryReason ? { retryReason: input.retryReason } : {}), + }, + status: "queued", + runId, + requestedAt: now, + updatedAt: now, + }); + + await db.insert(heartbeatRuns).values({ + id: runId, + companyId, + agentId, + invocationSource: "automation", + triggerDetail: "system", + status: "queued", + wakeupRequestId, + contextSnapshot: { + issueId, + taskId: issueId, + wakeReason, + ...(input?.retryReason ? { retryReason: input.retryReason } : {}), + }, + updatedAt: now, + }); + + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Review participant stayed pending", + status: "in_review", + priority: "medium", + assigneeAgentId: agentId, + assigneeUserId: null, + executionRunId: runId, + executionAgentNameKey: "codexreviewer", + executionLockedAt: now, + issueNumber: 1, + identifier: `${issuePrefix}-1`, + executionState: { + status: "pending", + currentStageId: stageId, + currentStageIndex: 0, + currentStageType: "review", + currentParticipant: { type: "agent", agentId, userId: null }, + returnAssignee: { type: "agent", agentId, userId: null }, + reviewRequest: null, + completedStageIds: [], + lastDecisionId: null, + lastDecisionOutcome: null, + }, + }); + + return { companyId, agentId, runId, wakeupRequestId, issueId, stageId }; + } + async function seedAssignedTodoNoRunFixture(input?: { agentStatus?: "paused" | "idle" | "running"; }) { @@ -816,10 +913,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { agentId: string; issueId: string; runId: string; - previousStatus: "todo" | "in_progress"; - retryReason?: "assignment_recovery" | "issue_continuation_needed" | null; + previousStatus: "todo" | "in_progress" | "in_review"; + retryReason?: "assignment_recovery" | "issue_continuation_needed" | "execution_review_participant_recovery" | null; cause?: string; kind?: string; + previousOwnerAgentId?: string | null; + returnOwnerAgentId?: string | null; }) { const action = await waitForValue(async () => db.select().from(issueRecoveryActions).where( @@ -839,8 +938,8 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { status: "active", ownerType: "agent", ownerAgentId: input.agentId, - previousOwnerAgentId: input.agentId, - returnOwnerAgentId: input.agentId, + previousOwnerAgentId: input.previousOwnerAgentId ?? input.agentId, + returnOwnerAgentId: input.returnOwnerAgentId ?? input.agentId, cause: input.cause ?? "stranded_assigned_issue", attemptCount: 1, maxAttempts: null, @@ -851,9 +950,13 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { latestRunId: input.runId, retryReason: input.retryReason ?? null, }); - expect(action.nextAction).toContain( - input.kind === "missing_disposition" ? "valid issue disposition" : "Restore a live execution path", - ); + if (input.cause === "execution_review_participant_recovery") { + expect(action.nextAction).toContain("failed review participant path"); + } else { + expect(action.nextAction).toContain( + input.kind === "missing_disposition" ? "valid issue disposition" : "Restore a live execution path", + ); + } const recoveryIssues = await db .select() @@ -2554,6 +2657,458 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { } }); + it("re-enqueues an already stranded execution-review participant during reconciliation", async () => { + const { agentId, issueId, runId, wakeupRequestId, stageId } = await seedInReviewParticipantRunFixture(); + const finishedAt = new Date("2026-03-19T00:05:00.000Z"); + await db + .update(heartbeatRuns) + .set({ + status: "succeeded", + startedAt: new Date("2026-03-19T00:00:00.000Z"), + finishedAt, + updatedAt: finishedAt, + }) + .where(eq(heartbeatRuns.id, runId)); + await db + .update(agentWakeupRequests) + .set({ + status: "completed", + finishedAt, + updatedAt: finishedAt, + }) + .where(eq(agentWakeupRequests.id, wakeupRequestId)); + const heartbeat = heartbeatService(db); + + const result = await heartbeat.reconcileStrandedAssignedIssues(); + expect(result.reviewParticipantRequeued).toBe(1); + expect(result.escalated).toBe(0); + expect(result.issueIds).toEqual([issueId]); + + const runs = await db + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.agentId, agentId)); + const retryRun = runs.find((row) => row.id !== runId); + expect(["queued", "running"]).toContain(retryRun?.status); + expect(retryRun).toMatchObject({ + retryOfRunId: runId, + }); + expect(retryRun?.contextSnapshot).toMatchObject({ + issueId, + taskId: issueId, + wakeReason: "execution_review_participant_recovery", + retryReason: "execution_review_participant_recovery", + source: "issue.execution_review_recovery", + retryOfRunId: runId, + currentStageId: stageId, + currentStageType: "review", + reviewRecoveryInstruction: expect.stringContaining("Submit the review decision now"), + }); + expect(retryRun?.contextSnapshot as Record).not.toHaveProperty("modelProfile"); + }); + + it("re-enqueues a stranded execution-review participant when another agent has the latest issue run", async () => { + const { companyId, agentId, issueId, runId, wakeupRequestId, stageId } = + await seedInReviewParticipantRunFixture(); + const otherAgentId = randomUUID(); + const otherRunId = randomUUID(); + const finishedAt = new Date("2026-03-19T00:05:00.000Z"); + + await db + .update(heartbeatRuns) + .set({ + status: "succeeded", + startedAt: new Date("2026-03-19T00:00:00.000Z"), + finishedAt, + updatedAt: finishedAt, + }) + .where(eq(heartbeatRuns.id, runId)); + await db + .update(agentWakeupRequests) + .set({ + status: "completed", + finishedAt, + updatedAt: finishedAt, + }) + .where(eq(agentWakeupRequests.id, wakeupRequestId)); + + await db.insert(agents).values({ + id: otherAgentId, + companyId, + name: "CodexImplementor", + role: "engineer", + status: "idle", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + await db.insert(heartbeatRuns).values({ + id: otherRunId, + companyId, + agentId: otherAgentId, + invocationSource: "automation", + triggerDetail: "system", + status: "succeeded", + contextSnapshot: { + issueId, + taskId: issueId, + wakeReason: "issue_commented", + }, + startedAt: new Date("2026-03-19T00:10:00.000Z"), + finishedAt: new Date("2026-03-19T00:15:00.000Z"), + createdAt: new Date(Date.now() + 1_000), + updatedAt: new Date("2026-03-19T00:15:00.000Z"), + }); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.reconcileStrandedAssignedIssues(); + expect(result.reviewParticipantRequeued).toBe(1); + expect(result.escalated).toBe(0); + expect(result.issueIds).toEqual([issueId]); + + const retryRun = await db + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.agentId, agentId)) + .then((runs) => + runs.find((row) => + row.id !== runId && + (row.contextSnapshot as Record | null)?.retryReason === + "execution_review_participant_recovery" + ) ?? null + ); + expect(retryRun).toMatchObject({ + retryOfRunId: runId, + }); + expect(retryRun?.contextSnapshot).toMatchObject({ + issueId, + currentStageId: stageId, + currentStageType: "review", + }); + }); + + it("re-enqueues a stranded execution-review participant when another agent has a queued issue wake", async () => { + const { companyId, agentId, issueId, runId, wakeupRequestId } = + await seedInReviewParticipantRunFixture(); + const otherAgentId = randomUUID(); + const otherWakeId = randomUUID(); + const finishedAt = new Date("2026-03-19T00:05:00.000Z"); + + await db + .update(heartbeatRuns) + .set({ + status: "succeeded", + startedAt: new Date("2026-03-19T00:00:00.000Z"), + finishedAt, + updatedAt: finishedAt, + }) + .where(eq(heartbeatRuns.id, runId)); + await db + .update(agentWakeupRequests) + .set({ + status: "completed", + finishedAt, + updatedAt: finishedAt, + }) + .where(eq(agentWakeupRequests.id, wakeupRequestId)); + + await db.insert(agents).values({ + id: otherAgentId, + companyId, + name: "CodexImplementor", + role: "engineer", + status: "idle", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + await db.insert(agentWakeupRequests).values({ + id: otherWakeId, + companyId, + agentId: otherAgentId, + source: "automation", + triggerDetail: "system", + reason: "issue_commented", + payload: { issueId }, + status: "queued", + }); + + const heartbeat = heartbeatService(db); + const result = await heartbeat.reconcileStrandedAssignedIssues(); + expect(result.reviewParticipantRequeued).toBe(1); + expect(result.escalated).toBe(0); + expect(result.issueIds).toEqual([issueId]); + + const wakeups = await db + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.agentId, agentId)); + expect(wakeups.some((wakeup) => + wakeup.reason === "execution_review_participant_recovery" && + wakeup.status !== "skipped" + )).toBe(true); + }); + + it("retries a pending execution-review participant when another agent has an active issue run", async () => { + const { companyId, agentId, issueId, runId } = await seedInReviewParticipantRunFixture(); + const otherAgentId = randomUUID(); + const otherRunId = randomUUID(); + await db.insert(agents).values({ + id: otherAgentId, + companyId, + name: "CodexImplementor", + role: "engineer", + status: "idle", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + await db.insert(heartbeatRuns).values({ + id: otherRunId, + companyId, + agentId: otherAgentId, + invocationSource: "automation", + triggerDetail: "system", + status: "running", + contextSnapshot: { + issueId, + taskId: issueId, + wakeReason: "issue_commented", + }, + startedAt: new Date("2026-03-19T00:01:00.000Z"), + updatedAt: new Date("2026-03-19T00:01:00.000Z"), + }); + + const heartbeat = heartbeatService(db); + await heartbeat.resumeQueuedRuns(); + const reviewRecoveryRun = await waitForValue(async () => { + const runs = await db + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.agentId, agentId)); + return runs.find((row) => + (row.contextSnapshot as Record | null)?.retryReason === + "execution_review_participant_recovery" + ) ?? null; + }, 8_000); + + expect(reviewRecoveryRun).toMatchObject({ + companyId, + agentId, + retryOfRunId: runId, + }); + }); + + it("does not immediately recover a generic on-demand run used for an in-review agent API update", async () => { + const { agentId, issueId, runId } = await seedInReviewParticipantRunFixture({ + wakeReason: "manual", + }); + const heartbeat = heartbeatService(db); + + await heartbeat.resumeQueuedRuns(); + const settledRun = await waitForRunToSettle(heartbeat, runId, 8_000); + expect(settledRun?.status).toBe("succeeded"); + + const runs = await db + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.agentId, agentId)); + expect(runs.some((row) => + (row.contextSnapshot as Record | null)?.retryReason === + "execution_review_participant_recovery" + )).toBe(false); + + const issue = await db + .select() + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0] ?? null); + expect(issue?.status).toBe("in_review"); + expect(issue?.assigneeAgentId).toBe(agentId); + }); + + it("retries a pending execution-review participant once before blocking with a recovery action", async () => { + const { companyId, agentId, issueId, runId, stageId } = await seedInReviewParticipantRunFixture(); + const heartbeat = heartbeatService(db); + + await heartbeat.resumeQueuedRuns(); + const reviewRecoveryRun = await waitForValue(async () => { + const runs = await db + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.agentId, agentId)); + return runs.find((row) => + (row.contextSnapshot as Record | null)?.retryReason === + "execution_review_participant_recovery" && + row.status !== "queued" && + row.status !== "running" + ) ?? null; + }, 8_000); + expect(reviewRecoveryRun).toBeTruthy(); + expect(reviewRecoveryRun).toMatchObject({ + companyId, + agentId, + retryOfRunId: runId, + status: "succeeded", + }); + expect(reviewRecoveryRun?.contextSnapshot).toMatchObject({ + issueId, + taskId: issueId, + wakeReason: "execution_review_participant_recovery", + retryReason: "execution_review_participant_recovery", + source: "issue.execution_review_recovery", + retryOfRunId: runId, + currentStageId: stageId, + currentStageType: "review", + reviewRecoveryInstruction: expect.stringContaining("Submit the review decision now"), + }); + expect(reviewRecoveryRun?.contextSnapshot as Record).not.toHaveProperty("modelProfile"); + + const sourceIssue = await waitForValue(async () => { + const row = await db + .select() + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0] ?? null); + return row?.status === "blocked" ? row : null; + }, 8_000); + expect(sourceIssue).toMatchObject({ + status: "blocked", + assigneeAgentId: agentId, + executionRunId: null, + }); + + const recoveryAction = await expectSourceScopedStrandedRecoveryAction({ + companyId, + agentId, + issueId, + runId: reviewRecoveryRun!.id, + previousStatus: "in_review", + retryReason: "execution_review_participant_recovery", + cause: "execution_review_participant_recovery", + }); + expect(recoveryAction.evidence).toMatchObject({ + latestRunId: reviewRecoveryRun?.id, + latestRunStatus: "succeeded", + latestRunErrorCode: null, + recoveryCause: "execution_review_participant_recovery", + }); + + const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, issueId)); + const recoveryComment = comments.find((comment) => + comment.body.includes("pending execution-review participant once") && + comment.body.includes(`Recovery action: \`${recoveryAction.id}\``), + ); + expect(recoveryComment).toBeTruthy(); + + const activity = await db.select().from(activityLog).where(eq(activityLog.entityId, issueId)); + expect(activity.some((event) => + (event.details as Record | null)?.source === + "recovery.reconcile_execution_review_participant", + )).toBe(true); + }); + + it("blocks failed execution-review recovery under the reviewer when the source assignee differs", async () => { + const { companyId, agentId, issueId, runId, wakeupRequestId, stageId } = + await seedInReviewParticipantRunFixture({ + wakeReason: "execution_review_participant_recovery", + retryReason: "execution_review_participant_recovery", + }); + const sourceAssigneeAgentId = randomUUID(); + const finishedAt = new Date("2026-03-19T00:05:00.000Z"); + + await db.insert(agents).values({ + id: sourceAssigneeAgentId, + companyId, + name: "CodexImplementor", + role: "engineer", + status: "idle", + adapterType: "codex_local", + adapterConfig: {}, + runtimeConfig: {}, + permissions: {}, + }); + await db + .update(issues) + .set({ + assigneeAgentId: sourceAssigneeAgentId, + executionState: { + status: "pending", + currentStageId: stageId, + currentStageIndex: 0, + currentStageType: "review", + currentParticipant: { type: "agent", agentId, userId: null }, + returnAssignee: { type: "agent", agentId: sourceAssigneeAgentId, userId: null }, + reviewRequest: null, + completedStageIds: [], + lastDecisionId: null, + lastDecisionOutcome: null, + }, + }) + .where(eq(issues.id, issueId)); + await db + .update(heartbeatRuns) + .set({ + status: "failed", + startedAt: new Date("2026-03-19T00:00:00.000Z"), + finishedAt, + updatedAt: finishedAt, + errorCode: "adapter_failed", + error: "review recovery failed before submitting a decision", + }) + .where(eq(heartbeatRuns.id, runId)); + await db + .update(agentWakeupRequests) + .set({ + status: "failed", + claimedAt: new Date("2026-03-19T00:00:00.000Z"), + finishedAt, + updatedAt: finishedAt, + error: "review recovery failed before submitting a decision", + }) + .where(eq(agentWakeupRequests.id, wakeupRequestId)); + const heartbeat = heartbeatService(db); + + const result = await heartbeat.reconcileStrandedAssignedIssues(); + expect(result.reviewParticipantRequeued).toBe(0); + expect(result.escalated).toBe(1); + expect(result.issueIds).toEqual([issueId]); + + const sourceIssue = await waitForValue(async () => { + const row = await db + .select() + .from(issues) + .where(eq(issues.id, issueId)) + .then((rows) => rows[0] ?? null); + return row?.status === "blocked" ? row : null; + }); + expect(sourceIssue).toMatchObject({ + status: "blocked", + assigneeAgentId: agentId, + }); + + const recoveryAction = await expectSourceScopedStrandedRecoveryAction({ + companyId, + agentId, + issueId, + runId, + previousStatus: "in_review", + retryReason: "execution_review_participant_recovery", + cause: "execution_review_participant_recovery", + previousOwnerAgentId: sourceAssigneeAgentId, + returnOwnerAgentId: sourceAssigneeAgentId, + }); + expect(recoveryAction.evidence).toMatchObject({ + latestRunId: runId, + latestRunStatus: "failed", + latestRunErrorCode: "adapter_failed", + recoveryCause: "execution_review_participant_recovery", + }); + }); + it.each([ ["failed", "adapter_failed"], ["failed", "process_lost"], diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 63afaca38c..70e5465494 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -279,6 +279,9 @@ const WORKSPACE_VALIDATION_FAILURE_CODE = "workspace_validation_failed"; const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed"; const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete"; const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete"; +const EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON = "execution_review_participant_recovery"; +const EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON = "execution_review_participant_recovery"; +const EXECUTION_REVIEW_PARTICIPANT_RECOVERY_CAUSE = "execution_review_participant_recovery"; const GITHUB_PR_WORKFLOW_SKILL_KEY = "paperclipai/bundled/software-development/github-pr-workflow"; const GITHUB_PR_WORKFLOW_SKILL_SLUG = "github-pr-workflow"; const PUSH_CAPABILITY_ENV_KEYS = ["GH_TOKEN", "GITHUB_TOKEN"] as const; @@ -2035,7 +2038,10 @@ function summarizeRunFailureForIssueComment( function didAutomaticRecoveryFail( latestRun: Pick | null, - expectedRetryReason: "assignment_recovery" | "issue_continuation_needed", + expectedRetryReason: + | "assignment_recovery" + | "issue_continuation_needed" + | typeof EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON, ) { if (!latestRun) return false; @@ -2049,6 +2055,27 @@ function didAutomaticRecoveryFail( ); } +function isExecutionReviewParticipantRecoveryRun( + run: Pick | null, +) { + if (!run) return false; + const context = parseObject(run.contextSnapshot); + return readNonEmptyString(context.retryReason) === EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON; +} + +function isExecutionReviewParticipantRecoveryEligibleRun( + run: Pick | null, +) { + if (!run) return false; + const context = parseObject(run.contextSnapshot); + const wakeReason = readNonEmptyString(context.wakeReason); + return ( + wakeReason === "execution_review_requested" || + wakeReason === "execution_approval_requested" || + isExecutionReviewParticipantRecoveryRun(run) + ); +} + function normalizeLedgerBillingType(value: unknown): BillingType { const raw = readNonEmptyString(value); switch (raw) { @@ -2397,6 +2424,7 @@ export function shouldResetTaskSessionForWake( if ( wakeReason === "issue_assigned" || wakeReason === "execution_review_requested" || + wakeReason === EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON || wakeReason === "execution_approval_requested" || wakeReason === "execution_changes_requested" || // PF-4: timer-driven wakes are exploratory ("any new work?"). They do not @@ -2509,6 +2537,9 @@ export function describeSessionResetReason( const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason); if (wakeReason === "issue_assigned") return "wake reason is issue_assigned"; if (wakeReason === "execution_review_requested") return "wake reason is execution_review_requested"; + if (wakeReason === EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON) { + return `wake reason is ${EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON}`; + } if (wakeReason === "execution_approval_requested") return "wake reason is execution_approval_requested"; if (wakeReason === "execution_changes_requested") return "wake reason is execution_changes_requested"; // PF-4: paired with shouldResetTaskSessionForWake — keep the reason wording @@ -11458,6 +11489,18 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) ); } + function buildExecutionReviewParticipantRecoveryComment(input: { + latestRun: Pick | null | undefined; + }) { + const failureSummary = summarizeRunFailureForIssueComment(input.latestRun); + return ( + "Paperclip retried the pending execution-review participant once, but the review stage still has no completed decision " + + `or live reviewer run.${failureSummary ?? ""} ` + + "Moving it to `blocked` with a source-scoped recovery action so the recovery owner can repair the reviewer runtime, " + + "restore the review stage, or record an intentional manual resolution." + ); + } + async function releaseIssueExecutionAndPromote( run: typeof heartbeatRuns.$inferSelect, options: { suppressImmediateRecovery?: boolean } = {}, @@ -11839,6 +11882,149 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) }; } + const findExistingExecutionPath = (agentId?: string | null) => + tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, issue.companyId), + inArray(heartbeatRuns.status, [...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES]), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`, + sql`${heartbeatRuns.id} <> ${run.id}`, + agentId ? eq(heartbeatRuns.agentId, agentId) : sql`true`, + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + + const issueHasScheduledMonitor = + issue.monitorNextCheckAt instanceof Date && + issue.monitorNextCheckAt.getTime() > Date.now(); + const executionState = parseIssueExecutionState(issue.executionState); + const currentParticipant = executionState?.status === "pending" + ? executionState.currentParticipant + : null; + const issueNeedsReviewParticipantRecovery = + issue.status === "in_review" && + !issue.assigneeUserId && + currentParticipant?.type === "agent" && + currentParticipant.agentId === run.agentId && + isExecutionReviewParticipantRecoveryEligibleRun(run) && + HEARTBEAT_RUN_TERMINAL_STATUSES.includes( + run.status as (typeof HEARTBEAT_RUN_TERMINAL_STATUSES)[number], + ); + + if (issueNeedsReviewParticipantRecovery) { + const existingReviewParticipantExecutionPath = await findExistingExecutionPath(currentParticipant.agentId); + if ( + options.suppressImmediateRecovery || + existingReviewParticipantExecutionPath || + issueHasScheduledMonitor || + await isAutomaticRecoverySuppressedByPauseHold(db, issue.companyId, issue.id, treeControlSvc) + ) { + return { kind: "released" as const }; + } + + if (issue.originKind === RECOVERY_ORIGIN_KINDS.strandedIssueRecovery) { + return { + kind: "blocked_recovery_in_place" as const, + issue, + previousStatus: issue.status, + }; + } + + const shouldBlockReviewRecovery = + !recoveryAgentInvokable || + !recoveryAgent || + isExecutionReviewParticipantRecoveryRun(run); + if (shouldBlockReviewRecovery) { + return { + kind: "blocked" as const, + issue, + previousStatus: issue.status, + comment: buildExecutionReviewParticipantRecoveryComment({ latestRun: run }), + recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_CAUSE, + recoveryOwnerAgentId: currentParticipant.agentId, + }; + } + + const now = new Date(); + const wakeupRequest = await tx + .insert(agentWakeupRequests) + .values({ + companyId: issue.companyId, + agentId: recoveryAgent.id, + source: "automation", + triggerDetail: "system", + reason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON, + payload: withRecoveryModelProfileHint({ + issueId: issue.id, + retryOfRunId: run.id, + retryReason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON, + currentStageId: executionState?.currentStageId ?? null, + currentStageType: executionState?.currentStageType ?? null, + }, "normal_model"), + status: "queued", + requestedByActorType: "system", + requestedByActorId: null, + updatedAt: now, + }) + .returning() + .then((rows) => rows[0]); + + const queuedRun = await tx + .insert(heartbeatRuns) + .values({ + companyId: issue.companyId, + agentId: recoveryAgent.id, + invocationSource: "automation", + triggerDetail: "system", + status: "queued", + wakeupRequestId: wakeupRequest.id, + contextSnapshot: withRecoveryModelProfileHint({ + issueId: issue.id, + taskId: issue.id, + wakeReason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_WAKE_REASON, + retryReason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_RETRY_REASON, + source: "issue.execution_review_recovery", + retryOfRunId: run.id, + currentStageId: executionState?.currentStageId ?? null, + currentStageType: executionState?.currentStageType ?? null, + reviewRecoveryInstruction: + "The previous reviewer run ended while this execution-review stage was still pending. Submit the review decision now, or mark the issue blocked with the exact unblock action.", + }, "normal_model"), + sessionIdBefore: recoverySessionBefore, + retryOfRunId: run.id, + updatedAt: now, + }) + .returning() + .then((rows) => rows[0]); + + await tx + .update(agentWakeupRequests) + .set({ + runId: queuedRun.id, + updatedAt: now, + }) + .where(eq(agentWakeupRequests.id, wakeupRequest.id)); + + await tx + .update(issues) + .set({ + executionRunId: queuedRun.id, + executionAgentNameKey: recoveryAgentNameKey, + executionLockedAt: now, + updatedAt: now, + }) + .where(eq(issues.id, issue.id)); + + return { + kind: "queued_recovery" as const, + run: queuedRun, + }; + } + const issueNeedsImmediateRecovery = (issue.status === "todo" || issue.status === "in_progress") && !issue.assigneeUserId && @@ -11852,19 +12038,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) return { kind: "released" as const }; } - const existingExecutionPath = await tx - .select({ id: heartbeatRuns.id }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, issue.companyId), - inArray(heartbeatRuns.status, [...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES]), - sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`, - sql`${heartbeatRuns.id} <> ${run.id}`, - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); + const existingExecutionPath = await findExistingExecutionPath(); if (existingExecutionPath) { return { kind: "released" as const }; } @@ -11987,7 +12161,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) if (promotionResult?.kind === "blocked") { await recovery.escalateStrandedAssignedIssue({ issue: promotionResult.issue, - previousStatus: promotionResult.previousStatus as "todo" | "in_progress", + previousStatus: promotionResult.previousStatus as "todo" | "in_progress" | "in_review", latestRun: run, comment: promotionResult.comment, recoveryCause: @@ -11995,7 +12169,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) ? WORKSPACE_VALIDATION_RECOVERY_CAUSE : promotionResult.recoveryCause === CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE ? CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE + : promotionResult.recoveryCause === EXECUTION_REVIEW_PARTICIPANT_RECOVERY_CAUSE + ? EXECUTION_REVIEW_PARTICIPANT_RECOVERY_CAUSE : undefined, + recoveryOwnerAgentId: promotionResult.recoveryOwnerAgentId, }); return; } @@ -12003,7 +12180,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) if (promotionResult?.kind === "blocked_recovery_in_place") { await recovery.escalateStrandedRecoveryIssueInPlace({ issue: promotionResult.issue, - previousStatus: promotionResult.previousStatus as "todo" | "in_progress", + previousStatus: promotionResult.previousStatus as "todo" | "in_progress" | "in_review", latestRun: run, }); return; diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index fa577ee791..607c8e8f7c 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1,4 +1,4 @@ -import { and, asc, desc, eq, gt, gte, inArray, isNull, notInArray, sql } from "drizzle-orm"; +import { and, asc, desc, eq, gt, gte, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { DEFAULT_ISSUE_GRAPH_LIVENESS_AUTO_RECOVERY_LOOKBACK_HOURS, @@ -37,6 +37,7 @@ import { instanceSettingsService } from "../instance-settings.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { issueTreeControlService } from "../issue-tree-control.js"; import { TERMINAL_HEARTBEAT_RUN_STATUSES, issueService } from "../issues.js"; +import { parseIssueExecutionState } from "../issue-execution-policy.js"; import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; import { getRunLogStore } from "../run-log-store.js"; import { @@ -72,6 +73,7 @@ const ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES = 8 * 1024; const STRANDED_ISSUE_RECOVERY_ORIGIN_KIND = RECOVERY_ORIGIN_KINDS.strandedIssueRecovery; const STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND = RECOVERY_ORIGIN_KINDS.staleActiveRunEvaluation; const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext"; +const EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON = "execution_review_participant_recovery"; const SESSIONED_LOCAL_ADAPTERS = new Set([ "claude_local", "codex_local", @@ -122,8 +124,11 @@ type StrandedRecoveryCause = | "stranded_assigned_issue" | "workspace_validation_failed" | "configuration_incomplete" + | "execution_review_participant_recovery" | typeof SUCCESSFUL_RUN_MISSING_STATE_REASON; +type StrandedPreviousStatus = "todo" | "in_progress" | "in_review"; + type SuccessfulRunHandoffRecoveryEvidence = { sourceRunId: string | null; correctiveRunId: string; @@ -175,9 +180,29 @@ function summarizeRunFailureForIssueComment(run: LatestIssueRun) { return null; } +function buildExecutionReviewParticipantRecoveryComment(latestRun: LatestIssueRun) { + const failureSummary = summarizeRunFailureForIssueComment(latestRun); + return ( + "Paperclip retried the pending execution-review participant once, but the review stage still has no completed decision " + + `or live reviewer run.${failureSummary ?? ""} ` + + "Moving it to `blocked` with a source-scoped recovery action so the recovery owner can repair the reviewer runtime, " + + "restore the review stage, or record an intentional manual resolution." + ); +} + +function buildExecutionReviewParticipantUnavailableComment(latestRun: LatestIssueRun) { + const failureSummary = summarizeRunFailureForIssueComment(latestRun); + return ( + "Paperclip cannot continue the pending execution-review participant because the participant is not invokable " + + `and the review stage has no completed decision or live reviewer run.${failureSummary ?? ""} ` + + "Moving it to `blocked` with a source-scoped recovery action so the recovery owner can repair the reviewer runtime, " + + "restore the review stage, or record an intentional manual resolution." + ); +} + function didAutomaticRecoveryFail( latestRun: LatestIssueRun, - expectedRetryReason: "assignment_recovery" | "issue_continuation_needed", + expectedRetryReason: "assignment_recovery" | "issue_continuation_needed" | typeof EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON, ) { if (!latestRun) return false; @@ -189,6 +214,11 @@ function didAutomaticRecoveryFail( ); } +function isTerminalIssueRun(latestRun: LatestIssueRun) { + if (!latestRun) return false; + return TERMINAL_HEARTBEAT_RUN_STATUSES.has(latestRun.status); +} + const TRANSIENT_INFRA_CONTINUATION_ERROR_CODES = new Set([ "adapter_failed", "codex_transient_upstream", @@ -523,6 +553,35 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) .then((rows) => rows[0] ?? null); } + async function getLatestIssueRunForAgent( + companyId: string, + issueId: string, + agentId: string, + ): Promise { + return db + .select({ + id: heartbeatRuns.id, + agentId: heartbeatRuns.agentId, + status: heartbeatRuns.status, + error: heartbeatRuns.error, + errorCode: heartbeatRuns.errorCode, + contextSnapshot: heartbeatRuns.contextSnapshot, + livenessState: heartbeatRuns.livenessState, + resultJson: heartbeatRuns.resultJson, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, companyId), + eq(heartbeatRuns.agentId, agentId), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issueId}`, + ), + ) + .orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)) + .limit(1) + .then((rows) => rows[0] ?? null); + } + async function summarizeRecentContinuationRetries( companyId: string, issueId: string, @@ -571,7 +630,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) return { consecutive, latestFinishedAt }; } - async function hasActiveExecutionPath(companyId: string, issueId: string) { + async function hasActiveExecutionPath(companyId: string, issueId: string, agentId?: string | null) { const [run, deferredWake] = await Promise.all([ db .select({ id: heartbeatRuns.id }) @@ -581,6 +640,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.status, [...EXECUTION_PATH_HEARTBEAT_RUN_STATUSES]), sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issueId}`, + agentId ? eq(heartbeatRuns.agentId, agentId) : sql`true`, ), ) .limit(1) @@ -593,6 +653,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.payload} ->> 'issueId' = ${issueId}`, + agentId ? eq(agentWakeupRequests.agentId, agentId) : sql`true`, ), ) .limit(1) @@ -618,7 +679,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) .then((rows) => Boolean(rows[0])); } - async function hasQueuedIssueWake(companyId: string, issueId: string) { + async function hasQueuedIssueWake(companyId: string, issueId: string, agentId?: string | null) { return db .select({ id: agentWakeupRequests.id }) .from(agentWakeupRequests) @@ -627,6 +688,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "queued"), sql`${agentWakeupRequests.payload} ->> 'issueId' = ${issueId}`, + agentId ? eq(agentWakeupRequests.agentId, agentId) : sql`true`, ), ) .limit(1) @@ -677,10 +739,11 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) async function enqueueStrandedIssueRecovery(input: { issueId: string; agentId: string; - reason: "issue_assignment_recovery" | "issue_continuation_needed"; - retryReason: "assignment_recovery" | "issue_continuation_needed"; + reason: "issue_assignment_recovery" | "issue_continuation_needed" | typeof EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON; + retryReason: "assignment_recovery" | "issue_continuation_needed" | typeof EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON; source: string; retryOfRunId?: string | null; + extraContext?: Record; }) { const queued = await deps.enqueueWakeup(input.agentId, { source: "automation", @@ -689,6 +752,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) payload: withRecoveryModelProfileHint({ issueId: input.issueId, ...(input.retryOfRunId ? { retryOfRunId: input.retryOfRunId } : {}), + ...(input.extraContext ?? {}), }, "normal_model"), requestedByActorType: "system", requestedByActorId: null, @@ -699,6 +763,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) retryReason: input.retryReason, source: input.source, ...(input.retryOfRunId ? { retryOfRunId: input.retryOfRunId } : {}), + ...(input.extraContext ?? {}), }, "normal_model"), }); @@ -2052,8 +2117,12 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) ].join("\n"); } - async function resolveStrandedIssueRecoveryOwnerAgentId(issue: typeof issues.$inferSelect) { + async function resolveStrandedIssueRecoveryOwnerAgentId( + issue: typeof issues.$inferSelect, + preferredOwnerAgentId?: string | null, + ) { const candidateIds: string[] = []; + if (preferredOwnerAgentId) candidateIds.push(preferredOwnerAgentId); if (issue.assigneeAgentId) { const assignee = await getAgent(issue.assigneeAgentId); if (assignee?.reportsTo) candidateIds.push(assignee.reportsTo); @@ -2091,7 +2160,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) function buildStrandedIssueRecoveryDescription(input: { issue: typeof issues.$inferSelect; latestRun: LatestIssueRun; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; prefix: string; recoveryCause?: StrandedRecoveryCause; successfulRunHandoffEvidence?: SuccessfulRunHandoffRecoveryEvidence | null; @@ -2134,9 +2203,26 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) const retryReason = readNonEmptyString(parseObject(input.latestRun?.contextSnapshot)?.retryReason) ?? "unknown"; const failureSummary = summarizeRunFailureForIssueComment(input.latestRun); + const isReviewParticipantRecovery = input.recoveryCause === "execution_review_participant_recovery"; + const detectedInvariant = isReviewParticipantRecovery + ? "execution_review_participant_recovery" + : "stranded_assigned_issue"; + const requiredAction = isReviewParticipantRecovery + ? [ + "- Inspect the latest reviewer run and the pending execution-review stage.", + "- Fix the reviewer runtime, restore the source issue to `in_review` with a live participant, or record an intentional manual resolution.", + "- When the source issue has a live review path or has been intentionally resolved, mark this recovery issue done.", + ] + : [ + "- Inspect the latest run and source issue state.", + "- Fix the runtime/adapter problem, reassign the source issue, or convert the source issue into a clear manual-review state.", + "- When the source issue has a live execution path or has been intentionally resolved, mark this recovery issue done.", + ]; return [ - "Paperclip exhausted automatic recovery for an assigned issue and created this explicit recovery task.", + isReviewParticipantRecovery + ? "Paperclip exhausted automatic recovery for a pending execution-review participant and created this explicit recovery task." + : "Paperclip exhausted automatic recovery for an assigned issue and created this explicit recovery task.", "", "## Source", "", @@ -2144,7 +2230,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) `- Previous source status: \`${input.previousStatus}\``, `- Latest retry run: ${runLink}`, `- Latest retry status: \`${input.latestRun?.status ?? "unknown"}\``, - `- Detected invariant: \`stranded_assigned_issue\``, + `- Detected invariant: \`${detectedInvariant}\``, `- Retry reason: \`${retryReason}\``, failureSummary ? `- Failure: ${failureSummary.trim()}` : "- Failure: none recorded", "", @@ -2154,16 +2240,14 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) "", "## Required Action", "", - "- Inspect the latest run and source issue state.", - "- Fix the runtime/adapter problem, reassign the source issue, or convert the source issue into a clear manual-review state.", - "- When the source issue has a live execution path or has been intentionally resolved, mark this recovery issue done.", + ...requiredAction, ].join("\n"); } async function ensureStrandedIssueRecoveryIssue(input: { issue: typeof issues.$inferSelect; latestRun: LatestIssueRun; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; recoveryCause?: StrandedRecoveryCause; successfulRunHandoffEvidence?: SuccessfulRunHandoffRecoveryEvidence | null; }) { @@ -2284,7 +2368,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) function buildStrandedRecoveryActionEvidence(input: { issue: typeof issues.$inferSelect; latestRun: LatestIssueRun; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; recoveryCause: StrandedRecoveryCause; successfulRunHandoffEvidence?: SuccessfulRunHandoffRecoveryEvidence | null; }) { @@ -2314,12 +2398,16 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) async function ensureSourceScopedStrandedRecoveryAction(input: { issue: typeof issues.$inferSelect; latestRun: LatestIssueRun; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; recoveryCause?: StrandedRecoveryCause; + recoveryOwnerAgentId?: string | null; successfulRunHandoffEvidence?: SuccessfulRunHandoffRecoveryEvidence | null; }) { const recoveryCause = input.recoveryCause ?? "stranded_assigned_issue"; - const ownerAgentId = await resolveStrandedIssueRecoveryOwnerAgentId(input.issue); + const ownerAgentId = await resolveStrandedIssueRecoveryOwnerAgentId( + input.issue, + input.recoveryOwnerAgentId, + ); const now = new Date(); const action = await recoveryActionsSvc.upsertSourceScoped({ companyId: input.issue.companyId, @@ -2350,6 +2438,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) : "Repair the source issue workspace link, project workspace cwd, or git checkout before resuming adapter execution." : recoveryCause === "configuration_incomplete" ? "Bind the missing secret(s) named in the run failure to the agent/project/routine env before resuming adapter execution." + : recoveryCause === "execution_review_participant_recovery" + ? "Repair the failed review participant path, restore the source issue to in_review with a live reviewer, or record an intentional manual resolution." : "Restore a live execution path, fix the runtime/adapter failure, or record an intentional manual resolution.", wakePolicy: recoveryCause === "workspace_validation_failed" || recoveryCause === "configuration_incomplete" ? { @@ -2414,7 +2504,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) function buildRecoveryIssueInPlaceEscalationComment(input: { issue: typeof issues.$inferSelect; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; latestRun: LatestIssueRun; prefix: string; }) { @@ -2441,7 +2531,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) async function escalateStrandedRecoveryIssueInPlace(input: { issue: typeof issues.$inferSelect; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; latestRun: LatestIssueRun; }) { const updated = await issuesSvc.update(input.issue.id, { status: "blocked" }); @@ -2574,10 +2664,11 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) async function escalateStrandedAssignedIssue(input: { issue: typeof issues.$inferSelect; - previousStatus: "todo" | "in_progress"; + previousStatus: StrandedPreviousStatus; latestRun: LatestIssueRun; comment?: string; recoveryCause?: StrandedRecoveryCause; + recoveryOwnerAgentId?: string | null; successfulRunHandoffEvidence?: SuccessfulRunHandoffRecoveryEvidence | null; }) { if (isStrandedIssueRecoveryIssue(input.issue)) { @@ -2594,6 +2685,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) previousStatus: input.previousStatus, latestRun: input.latestRun, recoveryCause, + recoveryOwnerAgentId: input.recoveryOwnerAgentId, successfulRunHandoffEvidence: input.successfulRunHandoffEvidence, }); const blockerIds = await existingUnresolvedBlockerIssueIds(input.issue.companyId, input.issue.id); @@ -2697,6 +2789,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) ? "recovery.reconcile_workspace_validation_failed" : input.recoveryCause === "configuration_incomplete" ? "recovery.reconcile_configuration_incomplete" + : input.recoveryCause === "execution_review_participant_recovery" + ? "recovery.reconcile_execution_review_participant" : "recovery.reconcile_stranded_assigned_issue", recoveryCause: input.recoveryCause ?? "stranded_assigned_issue", latestRunId: input.latestRun?.id ?? null, @@ -2750,8 +2844,11 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) .where( and( isNull(issues.assigneeUserId), - inArray(issues.status, ["todo", "in_progress"]), - sql`${issues.assigneeAgentId} is not null`, + inArray(issues.status, ["todo", "in_progress", "in_review"]), + or( + sql`${issues.assigneeAgentId} is not null`, + eq(issues.status, "in_review"), + ), ), ); @@ -2763,6 +2860,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) successfulContinuationObserved: 0, orphanBlockersAssigned: 0, successfulRunHandoffEscalated: 0, + reviewParticipantRequeued: 0, escalated: 0, waitingOnReviewResolved: 0, recentProgressExempted: 0, @@ -2771,19 +2869,36 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) }; for (const issue of candidates) { - const agentId = issue.assigneeAgentId; + const executionState = issue.status === "in_review" + ? parseIssueExecutionState(issue.executionState) + : null; + const pendingExecutionState = executionState?.status === "pending" ? executionState : null; + const currentParticipant = pendingExecutionState + ? pendingExecutionState.currentParticipant + : null; + const participantAgentId = currentParticipant?.type === "agent" ? currentParticipant.agentId : null; + const agentId = issue.status === "in_review" && participantAgentId + ? participantAgentId + : issue.assigneeAgentId; if (!agentId) { result.skipped += 1; continue; } const agent = await getAgent(agentId); - if (!agent || agent.companyId !== issue.companyId || !(await isAgentInvokable(agent))) { + const agentInvokable = agent && agent.companyId === issue.companyId + ? await isAgentInvokable(agent) + : false; + if (issue.status !== "in_review" && !agentInvokable) { result.skipped += 1; continue; } - if (await hasActiveExecutionPath(issue.companyId, issue.id)) { + if (await hasActiveExecutionPath( + issue.companyId, + issue.id, + issue.status === "in_review" ? agentId : null, + )) { result.skipped += 1; continue; } @@ -2802,7 +2917,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) if (isStrandedIssueRecoveryIssue(issue) && isUnsuccessfulTerminalIssueRun(latestRun)) { const updated = await escalateStrandedRecoveryIssueInPlace({ issue, - previousStatus: issue.status as "todo" | "in_progress", + previousStatus: issue.status as StrandedPreviousStatus, latestRun, }); if (updated) { @@ -2814,6 +2929,108 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) continue; } + if (issue.status === "in_review") { + if (!participantAgentId || !pendingExecutionState) { + result.skipped += 1; + continue; + } + const participantLatestRun = await getLatestIssueRunForAgent( + issue.companyId, + issue.id, + participantAgentId, + ); + + if (!participantLatestRun || !isTerminalIssueRun(participantLatestRun)) { + if (!agentInvokable) { + const updated = await escalateStrandedAssignedIssue({ + issue, + previousStatus: "in_review", + latestRun: participantLatestRun, + comment: buildExecutionReviewParticipantUnavailableComment(participantLatestRun), + recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON, + recoveryOwnerAgentId: participantAgentId, + }); + if (updated) { + result.escalated += 1; + result.issueIds.push(issue.id); + } else { + result.skipped += 1; + } + } else { + result.skipped += 1; + } + continue; + } + + if (!agentInvokable) { + const updated = await escalateStrandedAssignedIssue({ + issue, + previousStatus: "in_review", + latestRun: participantLatestRun, + comment: buildExecutionReviewParticipantUnavailableComment(participantLatestRun), + recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON, + recoveryOwnerAgentId: participantAgentId, + }); + if (updated) { + result.escalated += 1; + result.issueIds.push(issue.id); + } else { + result.skipped += 1; + } + continue; + } + + if (didAutomaticRecoveryFail(participantLatestRun, EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON)) { + const updated = await escalateStrandedAssignedIssue({ + issue, + previousStatus: "in_review", + latestRun: participantLatestRun, + comment: buildExecutionReviewParticipantRecoveryComment(participantLatestRun), + recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON, + recoveryOwnerAgentId: participantAgentId, + }); + if (updated) { + result.escalated += 1; + result.issueIds.push(issue.id); + } else { + result.skipped += 1; + } + continue; + } + + if (await hasQueuedIssueWake(issue.companyId, issue.id, participantAgentId)) { + result.skipped += 1; + continue; + } + + if (await isInvocationBudgetBlocked(issue, participantAgentId)) { + result.skipped += 1; + continue; + } + + const queued = await enqueueStrandedIssueRecovery({ + issueId: issue.id, + agentId: participantAgentId, + reason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON, + retryReason: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON, + source: "issue.execution_review_recovery", + retryOfRunId: participantLatestRun.id, + extraContext: { + currentStageId: pendingExecutionState.currentStageId ?? null, + currentStageType: pendingExecutionState.currentStageType ?? null, + reviewRecoveryInstruction: + "The previous reviewer run ended while this execution-review stage was still pending. Submit the review decision now, or mark the issue blocked with the exact unblock action.", + }, + }); + if (queued) { + result.reviewParticipantRequeued += 1; + result.issueIds.push(issue.id); + } else { + result.skipped += 1; + } + continue; + } + if (issue.status === "todo") { if (!latestRun) { if (await hasQueuedIssueWake(issue.companyId, issue.id)) { diff --git a/skills/paperclip/SKILL.md b/skills/paperclip/SKILL.md index 7f9e2d6bc5..86bd26a620 100644 --- a/skills/paperclip/SKILL.md +++ b/skills/paperclip/SKILL.md @@ -19,7 +19,7 @@ In Paperclip, **task** and **issue** refer to the same work item. The UI may use ## Authentication -Env vars auto-injected: `PAPERCLIP_AGENT_ID`, `PAPERCLIP_COMPANY_ID`, `PAPERCLIP_API_URL`, `PAPERCLIP_RUN_ID`. Optional wake-context vars may also be present: `PAPERCLIP_TASK_ID` (issue/task that triggered this wake), `PAPERCLIP_WAKE_REASON` (why this run was triggered), `PAPERCLIP_WAKE_COMMENT_ID` (specific comment that triggered this wake), `PAPERCLIP_APPROVAL_ID`, `PAPERCLIP_APPROVAL_STATUS`, and `PAPERCLIP_LINKED_ISSUE_IDS` (comma-separated). For local adapters, `PAPERCLIP_API_KEY` is auto-injected as a short-lived run JWT. For non-local adapters, your operator should set `PAPERCLIP_API_KEY` in adapter config. All requests use `Authorization: Bearer $PAPERCLIP_API_KEY`. All endpoints under `/api`, all JSON. Never hard-code the API URL. +Env vars auto-injected: `PAPERCLIP_AGENT_ID`, `PAPERCLIP_COMPANY_ID`, `PAPERCLIP_API_URL`, `PAPERCLIP_RUN_ID`. Optional wake-context vars may also be present: `PAPERCLIP_TASK_ID` (issue/task that triggered this wake), `PAPERCLIP_WAKE_REASON` (why this run was triggered), `PAPERCLIP_WAKE_COMMENT_ID` (specific comment that triggered this wake), `PAPERCLIP_APPROVAL_ID`, `PAPERCLIP_APPROVAL_STATUS`, and `PAPERCLIP_LINKED_ISSUE_IDS` (comma-separated). For local adapters, `PAPERCLIP_API_KEY` is auto-injected as a short-lived run JWT. For sandbox-backed local adapters, the Bash/tool environment may receive `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY` for a run-scoped bridge instead of the host API directly; use those exact env vars from Bash/curl and do not assume the host port is reachable from browser or web tools. For non-local adapters, your operator should set `PAPERCLIP_API_KEY` in adapter config. All requests use `Authorization: Bearer $PAPERCLIP_API_KEY`. All endpoints under `/api`, all JSON. Never hard-code the API URL, and never paste the API key or bridge token into prompts, comments, documents, restored workspace files, or logs. Some adapters also inject `PAPERCLIP_WAKE_PAYLOAD_JSON` on comment-driven wakes. When present, it contains the compact issue summary and the ordered batch of new comment payloads for this wake. Use it first. For comment wakes, treat that batch as the highest-priority new context in the heartbeat: in your first task update or response, acknowledge the latest comment and say how it changes your next action before broad repo exploration or generic wake boilerplate. Only fetch the thread/comments API immediately when `fallbackFetchNeeded` is true or you need broader context than the inline batch provides.