fix(server): stand down recovery while an operator-cancelled run is the latest activity (#10656)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Operators can cancel a running agent from the board when a run is
unwanted — most acutely while cleaning up a runaway loop
> - The recovery machinery treats a cancelled run like any other
unsuccessful terminal run: the stranded-issue sweep classifies the issue
as stranded, creates a recovery action, and wakes the agent again
> - So cancelling runs to stop a loop *fed* the loop: each operator
cancel spawned a recovery action that re-woke the agent the operator had
just stopped
> - This pull request stamps board-initiated cancellations with operator
attribution and makes the sweep stand down while such a run is the
issue's latest activity
> - The benefit is that an operator's cancel is final until something
new happens, instead of being fought by automation

## Linked Issues or Issue Description

Fixes #10646

## What Changed

- `POST /heartbeat-runs/:runId/cancel` (board-only) now cancels with an
explicit reason ("Cancelled by a board operator") and stamps
`resultJson.cancelledByActorType: "user"` / `cancelledByUserId`.
- `reconcileStrandedAssignedIssues` gains an early stand-down: when the
issue's latest run is operator-cancelled (the new stamp, or the existing
`operator_interrupted` error code from interrupt-by-comment), the issue
is skipped entirely — no recovery action, no wake — and counted in a new
`operatorCancelExempted` result field. The exemption is inherently
self-limiting: any newer run or wake supersedes it because the gate only
looks at the *latest* run.
- System cancellations without operator attribution (lease expiry,
assignee changes, terminal-status cancels, pause holds) keep today's
recovery behavior unchanged.

## Verification

- `pnpm vitest run server/src/__tests__/issue-recovery-actions.test.ts`
(embedded Postgres) — 3 new cases: a stamped operator cancel produces
zero recovery actions and zero wakes; an `operator_interrupted` cancel
likewise; an unattributed system cancel still flows into pre-existing
recovery (wake observed), proving the stand-down is scoped to operator
attribution.
- `pnpm vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/issue-scheduled-retry-routes.test.ts` — unchanged
(109 tests).
- `cd server && pnpm run typecheck`.

## Risks

- Low. The only suppressed behavior is recovery of runs a human
explicitly cancelled from the board; everything else is byte-identical.
If an operator cancels and walks away, the issue stays quiet until any
new activity — which is the intent (the operator owns the next step).

## Model Used

Claude Fable 5 (`claude-fable-5`, Anthropic) via Claude Code — extended
thinking, agentic tool use. No other models involved.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Devin Foley 2026-08-01 16:51:13 -07:00 committed by GitHub
parent 0f12721ee9
commit a0dbe21045
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 103 additions and 1 deletions

View File

@ -380,6 +380,80 @@ describeEmbeddedPostgres("issue recovery actions", () => {
},
);
it("stands down while the latest run was cancelled by a board operator", async () => {
const { companyId, coderId, sourceIssueId } = await seedCompany();
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId: coderId,
invocationSource: "manual",
status: "cancelled",
error: "Cancelled by a board operator",
errorCode: "cancelled",
resultJson: { cancelledByActorType: "user", cancelledByUserId: "board-user" },
startedAt: new Date("2026-07-15T20:00:00.000Z"),
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
contextSnapshot: { issueId: sourceIssueId },
});
const enqueueWakeup = vi.fn(async () => null);
const recovery = recoveryService(db, { enqueueWakeup });
const result = await recovery.reconcileStrandedAssignedIssues();
expect(result.operatorCancelExempted).toBe(1);
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
expect(enqueueWakeup).not.toHaveBeenCalled();
});
it("stands down after an operator interrupt cancellation", async () => {
const { companyId, coderId, sourceIssueId } = await seedCompany();
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId: coderId,
invocationSource: "manual",
status: "cancelled",
error: "Interrupted by board comment",
errorCode: "operator_interrupted",
startedAt: new Date("2026-07-15T20:00:00.000Z"),
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
contextSnapshot: { issueId: sourceIssueId },
});
const enqueueWakeup = vi.fn(async () => null);
const recovery = recoveryService(db, { enqueueWakeup });
const result = await recovery.reconcileStrandedAssignedIssues();
expect(result.operatorCancelExempted).toBe(1);
expect(enqueueWakeup).not.toHaveBeenCalled();
});
it("still recovers system-cancelled runs with no operator attribution", async () => {
const { companyId, coderId, sourceIssueId } = await seedCompany();
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId: coderId,
invocationSource: "manual",
status: "cancelled",
error: "Cancelled because the workspace lease expired",
errorCode: "cancelled",
startedAt: new Date("2026-07-15T20:00:00.000Z"),
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
contextSnapshot: { issueId: sourceIssueId },
});
const enqueueWakeup = vi.fn(async () => null);
const recovery = recoveryService(db, { enqueueWakeup });
const result = await recovery.reconcileStrandedAssignedIssues();
expect(result.operatorCancelExempted).toBe(0);
// The system-cancelled run still flows into the pre-existing recovery
// behavior (a continuation requeue or escalation — either produces a
// wake), proving the stand-down is scoped to operator attribution.
expect(enqueueWakeup).toHaveBeenCalled();
});
it("schedules a provider-quota monitor for the original assignee without creating recovery work", async () => {
const { companyId, coderId, sourceIssueId } = await seedCompany();
const runId = randomUUID();

View File

@ -3756,7 +3756,15 @@ export function agentRoutes(
const runId = req.params.runId as string;
const existing = await getAccessibleResource(req, res, heartbeat.getRun(runId), "Heartbeat run not found");
if (!existing) return;
const run = await heartbeat.cancelRun(runId);
// Stamp the cancellation as operator-initiated (this route is board-only).
// Recovery reads this to stand down instead of classifying the cancelled
// run as agent stranding and re-waking the agent the operator just stopped.
const run = await heartbeat.cancelRun(runId, "Cancelled by a board operator", {
resultJson: {
cancelledByActorType: "user",
cancelledByUserId: req.actor.userId ?? null,
},
});
if (run) {
await logActivity(db, {

View File

@ -645,6 +645,21 @@ function isStrandedIssueRecoveryIssue(issue: Pick<typeof issues.$inferSelect, "o
return isStrandedIssueRecoveryOriginKind(issue.originKind);
}
/**
* True when the issue's latest run was cancelled by a board operator (the
* board cancel route stamps the attribution; interrupt-by-comment uses the
* operator_interrupted error code). While such a run is the latest activity
* on an issue, recovery stands down entirely: the operator deliberately
* stopped the agent, and re-waking it or escalating "stranding" would
* fight the human. Any newer run or wake supersedes the exemption.
*/
function isOperatorCancelledRun(latestRun: LatestIssueRun): boolean {
if (!latestRun || latestRun.status !== "cancelled") return false;
if (latestRun.errorCode === "operator_interrupted") return true;
const result = parseObject(latestRun.resultJson);
return result.cancelledByActorType === "user" || result.cancelledByActorType === "board";
}
function isUnsuccessfulTerminalIssueRun(latestRun: LatestIssueRun) {
return Boolean(
latestRun &&
@ -3653,6 +3668,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
waitingOnReviewResolved: 0,
providerQuotaMonitored: 0,
recentProgressExempted: 0,
operatorCancelExempted: 0,
skipped: 0,
issueIds: [] as string[],
};
@ -3703,6 +3719,10 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
}
let latestRun = await getLatestIssueRun(issue.companyId, issue.id);
if (isOperatorCancelledRun(latestRun)) {
result.operatorCancelExempted += 1;
continue;
}
if (latestRun?.status === "succeeded" && await hasPersistedDurableWaitPath(issue)) {
result.skipped += 1;
continue;