fix(recovery): preserve hand-back wake liveness (#10562)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service delivers issue work to assigned agents. > - Recovery can hand an issue back to its agent while the recovery run is still active. > - The hand-back wake can merge into that active run and disappear when the run exits. > - The stranded-work scan also treats the successful recovery run as proof that the handed-back issue is live. > - This pull request keeps the hand-back wake for follow-up delivery and lets the scan repair a lost wake. > - The benefit is that an assigned issue continues after recovery without manual operator action. ## Linked Issues or Issue Description No public issue exists. This is related to the wake reconciliation work in #8943. **What happened?** A recovery action could hand an assigned issue back from `blocked` to `todo`. The `issue_recovery_action_restored` wake then merged into the recovery run that made the change. The wake disappeared when that run exited. The stranded-work scan did not repair the issue because it treated the successful recovery run as current liveness. **Expected behavior** Paperclip must dispatch the hand-back wake after the recovery run exits. If that delivery is lost, the stranded-work scan must enqueue the assigned `todo` issue again. **Steps to reproduce** 1. Start a recovery run for an assigned blocked issue. 2. Resolve a recovery action with the `handed_back` outcome. 3. Move the issue to `todo` while the recovery run is still active. 4. Observe that the wake merges into the active run and no new run starts after it exits. 5. Run the stranded-work scan and observe that the successful latest run prevents repair. **Paperclip version or commit** `131d476a7e` **Deployment mode** Local dev (`pnpm dev`). The defect is in the core server and is not deployment-specific. **Agent adapter(s) involved** Not adapter-specific. This is a core heartbeat and recovery defect. ## What Changed - Added `issue_recovery_action_restored` to the wake reasons that require follow-up delivery when an issue run is active. - Made the stranded-work scan detect a resolved hand-back that occurred during or after the latest successful run. - Added focused regression tests for the heartbeat coalescing seam and the stranded-work repair shape. - Documented the hand-back liveness guarantee in execution semantics section 9.1. ## Verification - `pnpm exec vitest run server/src/__tests__/heartbeat-comment-wake-batching.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts` passed: 104 tests. - `pnpm --filter @paperclipai/server typecheck` passed. - `pnpm -r typecheck` passed. - `pnpm build` passed. - `pnpm test:run` passed the server shard (3,095 passed, 2 skipped) and UI shard (3,182 passed). One unrelated CLI test failed because the agent environment exports static AWS credentials. `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN pnpm exec vitest run cli/src/__tests__/secrets.test.ts` passed all 8 tests. - `git diff --check` passed. - All GitHub checks passed on commit `8b380e67e6`. - Greptile gave 5/5 confidence with no comments or unresolved threads. ## Risks - Low risk. The follow-up rule affects only a recovery hand-back wake that arrives while the same issue already has an active run. - The backstop adds one indexed recovery-action lookup for an assigned `todo` issue whose latest run succeeded. - The timestamp check uses the latest run start time. This includes hand-backs made by that run and later hand-backs, but excludes older resolved actions. ## Model Used - OpenAI Codex with GPT-5 (`gpt-5`), agentic reasoning, tool use, and code execution. The serving context-window size is not exposed to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-31 11:03:30 -07:00 · 2026-07-31 11:03:30 -07:00 · cd7f84965c
parent bd53b99686
commit cd7f84965c
5 changed files with 221 additions and 2 deletions
--- a/doc/execution-semantics.md
+++ b/doc/execution-semantics.md
@ -519,6 +519,11 @@ Recovery rule:

 This is a dispatch recovery, not a continuation recovery.

+Recovery hand-back is covered by the same liveness guarantee:
+
+- an `issue_recovery_action_restored` wake requested while the resolving recovery run is still active is persisted as a follow-up and dispatched only after that run exits, so it cannot be coalesced into the run that requested it
+- if that follow-up is nevertheless lost, the stranded-work backstop treats an assigned `todo` issue with a resolved `handed_back` recovery action from during or after its latest successful run as stranded and queues the bounded assignment recovery wake; the successful resolving run is not, by itself, evidence that the handed-back source work is live
+
 ### 9.2 Stranded assigned `in_progress`

 Example:
--- a/server/src/tests/heartbeat-comment-wake-batching.test.ts
+++ b/server/src/tests/heartbeat-comment-wake-batching.test.ts
@ -295,6 +295,125 @@ describeEmbeddedPostgres("heartbeat comment wake batching", () => {
    expect(runs[0]?.id).toBe(runId);
  });

+  it("defers recovery hand-back wakes until the resolving run exits", async () => {
+    const companyId = randomUUID();
+    const agentId = randomUUID();
+    const issueId = randomUUID();
+    const runId = randomUUID();
+    const recoveryActionId = randomUUID();
+    const issuePrefix = `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`;
+    const heartbeat = heartbeatService(db);
+
+    await db.insert(companies).values({
+      id: companyId,
+      name: "Paperclip",
+      issuePrefix,
+      requireBoardApprovalForNewAgents: false,
+      defaultResponsibleUserId: "responsible-user",
+    });
+
+    await db.insert(agents).values({
+      id: agentId,
+      companyId,
+      name: "Recovery owner",
+      role: "engineer",
+      status: "running",
+      adapterType: "process",
+      adapterConfig: {},
+      runtimeConfig: {},
+      permissions: {},
+    });
+
+    await db.insert(heartbeatRuns).values({
+      id: runId,
+      companyId,
+      agentId,
+      invocationSource: "assignment",
+      triggerDetail: "system",
+      status: "running",
+      contextSnapshot: {
+        issueId,
+        taskId: issueId,
+        wakeReason: "source_scoped_recovery_action",
+      },
+    });
+    runningProcesses.set(runId, {
+      child: {} as never,
+      graceSec: 0,
+      processGroupId: null,
+    });
+
+    await db.insert(issues).values({
+      id: issueId,
+      companyId,
+      title: "Resume handed-back work",
+      status: "todo",
+      priority: "medium",
+      responsibleUserId: "responsible-user",
+      assigneeAgentId: agentId,
+      executionRunId: runId,
+      executionAgentNameKey: "recovery-owner",
+      executionLockedAt: new Date(),
+      issueNumber: 1,
+      identifier: `${issuePrefix}-1`,
+    });
+
+    const followupRun = await heartbeat.wakeup(agentId, {
+      source: "automation",
+      triggerDetail: "system",
+      reason: "issue_recovery_action_restored",
+      payload: {
+        issueId,
+        recoveryActionId,
+        mutation: "recovery_action_resolution",
+      },
+      contextSnapshot: {
+        issueId,
+        taskId: issueId,
+        recoveryActionId,
+        wakeReason: "issue_recovery_action_restored",
+        source: "issue.recovery_action_resolution",
+      },
+      requestedByActorType: "agent",
+      requestedByActorId: agentId,
+    });
+
+    expect(followupRun).toBeNull();
+
+    const deferred = await db
+      .select()
+      .from(agentWakeupRequests)
+      .where(
+        and(
+          eq(agentWakeupRequests.companyId, companyId),
+          eq(agentWakeupRequests.agentId, agentId),
+          eq(agentWakeupRequests.status, "deferred_issue_execution"),
+        ),
+      )
+      .then((rows) => rows[0] ?? null);
+
+    expect(deferred).toMatchObject({
+      reason: "issue_execution_deferred",
+      runId: null,
+      payload: expect.objectContaining({
+        issueId,
+        recoveryActionId,
+        mutation: "recovery_action_resolution",
+      }),
+    });
+    expect((deferred?.payload as Record<string, unknown>)._paperclipWakeContext).toMatchObject({
+      issueId,
+      taskId: issueId,
+      recoveryActionId,
+      wakeReason: "issue_recovery_action_restored",
+      source: "issue.recovery_action_resolution",
+    });
+
+    const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId));
+    expect(runs).toHaveLength(1);
+    expect(runs[0]?.id).toBe(runId);
+  });
+
  it("batches deferred comment wakes and forwards the ordered batch to the next run", async () => {
    const gateway = await createControlledGatewayServer();
    const companyId = randomUUID();
--- a/server/src/tests/heartbeat-process-recovery.test.ts
+++ b/server/src/tests/heartbeat-process-recovery.test.ts
@ -3751,6 +3751,60 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
    }
  });

+  it("re-enqueues handed-back todo work when its resolving run succeeded but the wake was lost", async () => {
+    const { companyId, agentId, issueId, runId } = await seedStrandedIssueFixture({
+      status: "todo",
+      runStatus: "succeeded",
+    });
+    const resolvedAt = new Date("2026-03-19T00:04:00.000Z");
+    await db.insert(issueRecoveryActions).values({
+      companyId,
+      sourceIssueId: issueId,
+      kind: "stranded_assigned_issue",
+      status: "resolved",
+      ownerType: "agent",
+      ownerAgentId: agentId,
+      previousOwnerAgentId: agentId,
+      returnOwnerAgentId: agentId,
+      cause: "stranded_assigned_issue",
+      fingerprint: `handed-back:${issueId}`,
+      nextAction: "Resume source work",
+      outcome: "handed_back",
+      resolutionNote: "Returned source work to the original owner",
+      resolvedAt,
+      createdAt: new Date("2026-03-19T00:01:00.000Z"),
+      updatedAt: resolvedAt,
+    });
+    const heartbeat = heartbeatService(db);
+
+    const result = await heartbeat.reconcileStrandedAssignedIssues();
+    expect(result.assignmentDispatched).toBe(0);
+    expect(result.dispatchRequeued).toBe(1);
+    expect(result.continuationRequeued).toBe(0);
+    expect(result.escalated).toBe(0);
+    expect(result.issueIds).toEqual([issueId]);
+
+    const runs = await db
+      .select()
+      .from(heartbeatRuns)
+      .where(eq(heartbeatRuns.agentId, agentId));
+    expect(runs).toHaveLength(2);
+
+    const retryRun = runs.find((row) => row.id !== runId);
+    expect(retryRun?.contextSnapshot).toMatchObject({
+      issueId,
+      taskId: issueId,
+      wakeReason: "issue_assignment_recovery",
+      retryReason: "assignment_recovery",
+      source: "issue.assignment_recovery",
+      retryOfRunId: runId,
+    });
+    expect(retryRun?.contextSnapshot as Record<string, unknown>).not.toHaveProperty("modelProfile");
+    if (retryRun) {
+      await waitForRunToSettle(heartbeat, retryRun.id);
+    }
+  });
+
  it("re-enqueues an already stranded execution-review participant during reconciliation", async () => {
    const { agentId, issueId, runId, wakeupRequestId, stageId } = await seedInReviewParticipantRunFixture();
    const finishedAt = new Date("2026-03-19T00:05:00.000Z");
--- a/server/src/services/heartbeat.ts
+++ b/server/src/services/heartbeat.ts
@ -544,6 +544,7 @@ function mergeAdapterRecoveryMetadata(input: {
 const RUNNING_ISSUE_WAKE_REASONS_REQUIRING_FOLLOWUP = new Set([
  "approval_approved",
  ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
+  "issue_recovery_action_restored",
 ]);
 const ISSUE_RESPONSIBLE_USER_WAKE_REASONS = new Set([
  "issue_assigned",
--- a/server/src/services/recovery/service.ts
+++ b/server/src/services/recovery/service.ts
@ -140,7 +140,15 @@ type ResolvedDependencyWakeBackstopOptions = {

 type LatestIssueRun = Pick<
  typeof heartbeatRuns.$inferSelect,
-  "id" | "agentId" | "status" | "error" | "errorCode" | "contextSnapshot" | "livenessState"
+  | "id"
+  | "agentId"
+  | "status"
+  | "error"
+  | "errorCode"
+  | "contextSnapshot"
+  | "livenessState"
+  | "startedAt"
+  | "createdAt"
 > & {
  resultJson?: unknown;
 } | null;
@ -784,6 +792,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
        contextSnapshot: heartbeatRuns.contextSnapshot,
        livenessState: heartbeatRuns.livenessState,
        resultJson: heartbeatRuns.resultJson,
+        startedAt: heartbeatRuns.startedAt,
+        createdAt: heartbeatRuns.createdAt,
      })
      .from(heartbeatRuns)
      .where(
@ -812,6 +822,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
        contextSnapshot: heartbeatRuns.contextSnapshot,
        livenessState: heartbeatRuns.livenessState,
        resultJson: heartbeatRuns.resultJson,
+        startedAt: heartbeatRuns.startedAt,
+        createdAt: heartbeatRuns.createdAt,
      })
      .from(heartbeatRuns)
      .where(
@ -948,6 +960,29 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
      .then((rows) => Boolean(rows[0]));
  }

+  async function wasTodoHandedBackDuringOrAfterLatestRun(
+    issue: typeof issues.$inferSelect,
+    latestRun: LatestIssueRun,
+  ) {
+    if (issue.status !== "todo" || latestRun?.status !== "succeeded") return false;
+    const runBeganAt = latestRun.startedAt ?? latestRun.createdAt;
+
+    return db
+      .select({ id: issueRecoveryActions.id })
+      .from(issueRecoveryActions)
+      .where(
+        and(
+          eq(issueRecoveryActions.companyId, issue.companyId),
+          eq(issueRecoveryActions.sourceIssueId, issue.id),
+          eq(issueRecoveryActions.status, "resolved"),
+          eq(issueRecoveryActions.outcome, "handed_back"),
+          gte(issueRecoveryActions.resolvedAt, runBeganAt),
+        ),
+      )
+      .limit(1)
+      .then((rows) => Boolean(rows[0]));
+  }
+
  async function hasQueuedIssueWake(companyId: string, issueId: string, agentId?: string | null) {
    return db
      .select({ id: agentWakeupRequests.id })
@ -1026,6 +1061,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
        contextSnapshot: heartbeatRuns.contextSnapshot,
        livenessState: heartbeatRuns.livenessState,
        resultJson: heartbeatRuns.resultJson,
+        startedAt: heartbeatRuns.startedAt,
+        createdAt: heartbeatRuns.createdAt,
      })
      .from(heartbeatRuns)
      .where(
@ -4001,7 +4038,10 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
          continue;
        }

-        if (latestRun.status === "succeeded") {
+        if (
+          latestRun.status === "succeeded" &&
+          !(await wasTodoHandedBackDuringOrAfterLatestRun(issue, latestRun))
+        ) {
          result.skipped += 1;
          continue;
        }