RomanNum3ral
/
paperclip
mirror of https://github.com/paperclipai/paperclip.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fix: preserve recovery retries across restarts (#11817) 

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The control plane must keep each active issue on a clear execution
or recovery path.
> - A missing issue disposition can require more than one bounded repair
attempt.
> - A server restart could lose that repair path or move source
ownership to the recovery owner.
> - A parked or expired retry could also make the user interface show a
false healthy state.
> - Concurrent recovery loops must not schedule the same repair attempt
twice.
> - This pull request keeps retry state durable, makes scheduling
atomic, and keeps source ownership stable.
> - The benefit is that recovery continues after a restart and operators
see the correct state.

## Linked Issues or Issue Description

**What happened?**

A run that ended without a valid issue disposition could lose its repair
path after a server restart. Manager recovery could also change the
source owner. In addition, a parked or expired retry could make the
issue look healthy when no active work existed. Concurrent
reconciliation could also schedule the same repair attempt twice.

**Expected behavior**

Paperclip must keep bounded source and manager repair attempts across
restarts. Recovery ownership must stay separate from source issue
ownership. The server and user interface must report only a live retry
as active work. Each repair attempt must be scheduled at most once per
company.

**Steps to reproduce**

1. Start an agent run on an issue.
2. End the run without a valid issue disposition.
3. Let the first repair attempt schedule a retry.
4. Restart the server, let the retry time pass without a live run, or
start two reconciliation loops together.
5. Observe that the repair path can stop, the issue can show a false
healthy state, or duplicate retries can be created.

**Paperclip version or commit**

The problem existed on `master` before candidate head
`d8e620fe86bade7df18decac332007f5821ae04f`.

**Deployment mode**

The problem affects self-hosted servers and local builds that use
automatic recovery.

## What Changed

- Persist bounded source-owner and manager repair lineages with stable
fingerprints and retry limits.
- Resume incomplete disposition repairs after a server restart.
- Keep recovery ownership separate from source issue ownership and
enforce source mutation authority.
- Project live retry evidence into issue and blocker summaries.
- Show recovery owner, return owner, attempt count, and retry state in
the board user interface.
- Treat expired or parked retries as attention states unless a queued or
running attempt exists.
- Atomically deduplicate disposition-repair wake requests with a
company-scoped partial unique index.
- Reuse the winning run when concurrent reconciliation loses the
uniqueness race, without duplicate scheduling activity.
- Honor disabled on-demand wake policy before recovery scheduling and
again before delayed retry promotion.
- Keep the new index migration safe for lagging seeded databases that
already contain the index.
- Add server and user interface tests for recovery, restart, ownership,
retry, concurrency, and blocker states.
- Update the implementation and execution semantics documents.

## Verification

- Focused server recovery and ownership suites: 282 tests passed on the
repaired base candidate.
- Focused user interface recovery suites: 128 tests passed on the
repaired base candidate.
- Atomic-deduplication schema and recovery suites: 111 tests passed on
the first Greptile repair.
- Recovery and scheduled-retry wake-policy suites: 126 tests passed at
`d8e620fe86bade7df18decac332007f5821ae04f`.
- The exact lagging-source migration-order test passed after the index
migration became idempotent: 1 test passed and 62 unrelated tests were
skipped.
- `@paperclipai/db` and `@paperclipai/server` typechecks passed at the
current head.
- Migration generation and migration safety checks passed for migration
`0226_tan_colossus.sql`.
- `pnpm check:token-gates` passed on the repaired base candidate.
- `pnpm -r typecheck` passed on the repaired base candidate.
- `pnpm build` passed on the repaired base candidate.
- `pnpm test:run` passed 4,540 tests on the repaired base candidate.
Four fixed-port cases met listeners that already existed on the host.
- The two unchanged fixed-port files passed in an isolated network
namespace: 129 tests passed and 27 tests were skipped.
- Independent Security and QA reviews approved
`63c0423aab54c66f2293a20b0fb3f3b013ee3ba8`; exact-head re-review is
required after automated checks settle on
`d8e620fe86bade7df18decac332007f5821ae04f`.

## Risks

- Recovery orchestration affects issue liveness and ownership. The new
paths use bounded attempts, stable fingerprints, row locks, authority
checks, and database uniqueness.
- A conservative attention state can show more warnings when a scheduled
retry has no queued or running attempt. It does not hide stopped work.
- Migration `0226_tan_colossus.sql` creates a partial unique index on a
known-large table. Migrations run transactionally, so `CONCURRENTLY` is
unavailable. The matching disposition-repair key namespace is introduced
by this release, so deployed databases have no matching rows before the
index is added.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex from the GPT-5 model family used agentic reasoning, tool
use, and code execution. The runtime did not expose the exact model ID
or context window.
- Anthropic Claude Opus 5 used a 1M context window, tool use, and code
execution for part of the user interface repair, as recorded in the
commit history.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dotta 2026-08-20 17:09:42 -05:00 committed by GitHub
parent de9645ab73
commit cb0009b097
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
35 changed files with 44454 additions and 163 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
doc/SPEC-implementation.md
View File

@ -510,9 +510,15 @@ V1 non-terminal liveness rule:
- external waits are durable only when persisted as a bounded monitor/scheduled wake, a first-class blocker with a named owner and action, or healthy delegated child work connected by a blocker edge when the source must wait; parent/child structure alone is not a wait path
- unmanaged shell jobs, detached sessions, adapter child processes, local polling loops, PIDs, logs, and comments are evidence rather than liveness; a managed runtime service counts only when paired with a persisted monitor, wake, blocker, or delegated issue that owns the next check
- heartbeat finalization evaluates liveness from persisted Paperclip state; an issue cannot remain healthy `in_progress` solely because the exiting heartbeat started a local/background watcher
- invalid external-wait recovery queues at most one normal-model continuation per source-state fingerprint, then requires a real blocker or explicit recovery action instead of repeating equivalent recovery wakes; new durable source activity may establish a new fingerprint
- a continuation cancelled as `issue_continuation_waiting_on_review` first converts a current typed wait target into a first-class wait; without a current target it is classified as `deliberate_wait_without_target` and gives the invokable original owner five normal-model disposition-repair attempts (immediate, then after 60, 120, 240, and 480 seconds, with up to 10 percent jitter)
- disposition repair revalidates blockers, children, interactions, approvals, monitors, execution stages, queued wakes, active runs, work products, owner invokability, budgets, and governance before every attempt; the attempt bound is keyed by durable source state, so comments or equivalent parked prose do not reset it while durable source-state changes may establish a new fingerprint
- backwards-compatible upgrades count consecutive historical `issue_continuation_waiting_on_review` cancellations for the unchanged accepted-interaction source state against the same five-attempt disposition-repair ceiling; missing pre-upgrade recovery-action rows do not reset the budget
- the source fingerprint, source-attempt count, next due time, source owner, and return owner persist in the recovery action; startup and periodic reconciliation resume that exact lineage without duplicate wakes, fold it when a current typed wait appears, and reschedule or escalate an expired action that has no live scheduled run
- source-attempt exhaustion opens one source-scoped manager recovery action without changing the source assignee; manager runs get five separate persisted attempts on the same immediate, 60, 120, 240, and 480 second bounded-delay schedule, then escalate visibly to the board when exhausted
- an active recovery action counts as a live source or blocker-chain path only while its owner has a live run, queued wake, scheduled retry, typed wait, or explicit board escalation; source and blocker projections consume the same nested recovery-path result
- when Paperclip cannot safely infer the next action, it surfaces the problem through visible blocked/recovery work instead of silently completing or reassigning work
- explicit recovery actions are the liveness primitive; source-scoped actions are the default form, issue-backed recovery is a fallback for independent repair work or safety boundaries, and comments alone are evidence rather than a healthy liveness path
- recovery-action ownership is separate from source-task ownership: automatic repair and manager escalation preserve the source `assigneeAgentId`; reassignment requires an explicit decision or a policy-defined serious failure
- source-scoped recovery routing is cause-keyed: lost processes, missing successful-run dispositions, and output-inactivity terminations retry the original agent when invokable; provider-quota failures create/reuse a scheduled wait-recovery monitor without a takeover wake; workspace validation and unknown causes route to the manager ladder
- recovery-scoped wakes replace the normal deliverable execution contract with a cause-specific recovery contract, and successful repair returns the issue to the recorded original owner by default while recording `handed_back` versus `owner_completed`

16
doc/execution-semantics.md
View File

@ -1,7 +1,7 @@
# Execution Semantics
Status: Current implementation guide
Date: 2026-07-23
Date: 2026-08-18
Audience: Product and engineering
This document explains how Paperclip interprets issue assignment, issue status, execution runs, wakeups, parent/sub-issue structure, and blocker relationships.
@ -389,6 +389,8 @@ A valid recovery action must name:
A source-scoped recovery action is the default form. Use it when the next safe move is to repair the source issue's liveness directly: move the source issue back to `todo` so it can be retried, clarify disposition, re-establish a monitor, record a false positive, or delegate real follow-up work from the source issue.
Recovery-action ownership and source-task ownership are separate contracts. Assigning a manager or board owner to a recovery action authorizes that owner to repair or route the recovery action; it does not write that owner into the source issue's `assigneeAgentId`. Automatic retry and escalation preserve the source assignee. Reassignment requires an explicit source-task decision or a policy-defined serious-failure path, with the normal company, authorization, budget, checkout, active-run-lock, governed-action, and activity-log checks.
Use an issue-backed recovery action only when the recovery is genuinely independent work or when source-scoped handling would be unsafe or unclear. Examples include:
- long or cross-agent repair work with its own assignee, subtasks, or blockers
@ -549,7 +551,13 @@ A continuation that the staleness gate cancelled with `issue_continuation_waitin
Recovery rule for a parked-for-review continuation:
- if the issue has a real waiting target — open (non-terminal) sub-tasks or existing unresolved blockers — Paperclip converts the deliberate wait into a first-class dependency wait: it sets the issue `blocked` by those issues, keeps the original assignee, and posts a plain-language comment explaining that the task will resume automatically when its dependencies finish. The issue then self-resumes through the normal `issue_blockers_resolved` path; no recovery action or escalation owner is involved
- if the issue has no waiting target, the park is indistinguishable from a genuine strand and falls through to the standard §9.2 escalation, preserving stranded detection
- if the issue has no current typed waiting target and the original owner is invokable, Paperclip classifies it as `deliberate_wait_without_target` and gives that owner five normal-model disposition-repair attempts: immediate, then after 60, 120, 240, and 480 seconds, with up to 10 percent jitter on delayed attempts
- before every attempt, Paperclip revalidates unresolved blockers and children, interactions, linked approvals, monitors, execution stages, queued wakes, active runs, work products, owner invokability, and budget or governance gates. Any real live or waiting path suppresses the retry
- the retry bound is keyed by an idempotent durable source-state fingerprint. Comments, repeated parked summaries, and equivalent prose do not reset it. Durable changes such as source status or assignee changes, dependency or interaction changes, approval or execution-policy changes, monitor changes, or work-product changes may create a new fingerprint
- on upgrade, consecutive historical `issue_continuation_waiting_on_review` cancellations for the same accepted interaction and still-unchanged durable source state seed this same counter. Five applicable pre-upgrade parks therefore exhaust the ceiling immediately; the absence of a historical `deliberate_wait_without_target` recovery-action row does not grant five new attempts
- the action persists the unchanged fingerprint, source-attempt count, due time, source owner, and return owner. Startup and periodic reconciliation reuse that state, fold the action when a current typed wait appears, and reschedule or escalate an expired attempt that has no live scheduled run. Idempotency keys prevent a restart from creating duplicate wakes or scheduled runs
- after five attempts with the same fingerprint, Paperclip opens one separate source-scoped manager recovery action. The manager owns only path repair; the source assignee remains unchanged. The manager gets five separate attempts on the same immediate, 60, 120, 240, and 480 second bounded-delay schedule before exhaustion escalates visibly to the board
- a recovery action is a healthy wait only while its owner has a live run, queued wake, scheduled retry, typed wait, or explicit board escalation. Source liveness and every blocker-chain projection use that same nested result
An accepted interaction supersedes a continuation park recorded before that acceptance. A queued continuation carrying a parseable `interactionResolvedAt` must not be cancelled solely because an older continuation summary says to wait for review or approval. Interaction-continuation recovery is bounded: after three consecutive continuation wakes are cancelled without a run starting, recovery converts a real dependency wait when one exists or escalates the missing execution path visibly instead of requeueing forever.
@ -780,6 +788,8 @@ Examples:
The recovery action stays source-scoped by default. The source issue should show the recovery owner, cause, evidence, next action, and wake or monitor policy in its own thread/detail surface.
The recovery owner owns the repair action, not the source deliverable. Manager escalation must preserve the source issue assignee unless an operator makes an explicit reassignment decision or an applicable serious-failure policy authorizes transfer.
Create an issue-backed recovery action only when a separate issue is the right execution object. In that fallback form, the source issue remains visible and is blocked on the recovery issue when blocking is necessary for correctness. The recovery owner must restore a live path, resolve the source issue manually, delegate real follow-up work, or record the reason the signal is a false positive.
Instance-level issue-graph liveness auto-recovery is disabled by default. When enabled, its lookback window means "dependency paths updated within the last N hours"; older findings remain advisory and are counted as outside the configured lookback instead of creating recovery actions automatically. This is an operator noise control, not the older staleness delay for determining whether a chain is old enough to surface.
@ -809,7 +819,7 @@ Paperclip still does not:
The recovery model is intentionally conservative:
- preserve ownership
- retry once when the control plane lost execution continuity
- use the cause-specific bound when the control plane lost execution continuity; deliberate waits without a target use five fingerprinted original-owner disposition repairs
- open an explicit recovery action when the system can identify a bounded recovery owner/action
- escalate visibly when the system cannot safely keep going

13
packages/db/src/agent-wakeup-requests-schema.test.ts
View File

@ -15,4 +15,17 @@ describe("agent wakeup request schema", () => {
]);
expect(index?.config.where).toBeDefined();
});
it("atomically deduplicates disposition-repair attempts per company", () => {
const index = getTableConfig(agentWakeupRequests).indexes.find(
(candidate) => candidate.config.name === "agent_wakeup_requests_disposition_repair_idempotency_uq",
);
expect(index?.config.unique).toBe(true);
expect(index?.config.columns.map((column) => (column as { name: string }).name)).toEqual([
"company_id",
"idempotency_key",
]);
expect(index?.config.where).toBeDefined();
});
});

2
packages/db/src/migrations/0226_tan_colossus.sql Normal file
View File

@ -0,0 +1,2 @@
-- paperclip:migration-safety-ignore large-create-index-not-concurrently: Drizzle migrations run transactionally, so CONCURRENTLY is unavailable. This release introduces the disposition-repair key namespace, so deployed databases have no matching rows before this required atomic-deduplication index is added.
CREATE UNIQUE INDEX IF NOT EXISTS "agent_wakeup_requests_disposition_repair_idempotency_uq" ON "agent_wakeup_requests" USING btree ("company_id","idempotency_key") WHERE "agent_wakeup_requests"."idempotency_key" LIKE 'issue_disposition_repair:%' AND "agent_wakeup_requests"."status" <> 'skipped';

39622
packages/db/src/migrations/meta/0226_snapshot.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

7
packages/db/src/migrations/meta/_journal.json generated
View File

@ -1569,6 +1569,13 @@
"when": 1787101144413,
"tag": "0225_drop_claude_setup_token_sessions",
"breakpoints": true
},
{
"idx": 226,
"version": "7",
"when": 1787261199301,
"tag": "0226_tan_colossus",
"breakpoints": true
}
]
}

3
packages/db/src/schema/agent_wakeup_requests.ts
View File

@ -40,6 +40,9 @@ export const agentWakeupRequests = pgTable(
reviewPathRecoveryIdempotencyUq: uniqueIndex("agent_wakeup_requests_review_path_recovery_idempotency_uq")
.on(table.companyId, table.idempotencyKey)
.where(sql`${table.idempotencyKey} LIKE 'issue_review_path_lost:%' AND ${table.status} <> 'skipped'`),
dispositionRepairIdempotencyUq: uniqueIndex("agent_wakeup_requests_disposition_repair_idempotency_uq")
.on(table.companyId, table.idempotencyKey)
.where(sql`${table.idempotencyKey} LIKE 'issue_disposition_repair:%' AND ${table.status} <> 'skipped'`),
companyPayloadIssueIdx: index("agent_wakeup_requests_company_payload_issue_idx").on(
table.companyId,
sql`(${table.payload} ->> 'issueId')`,

3
packages/shared/src/constants.ts
View File

@ -378,6 +378,7 @@ export type IssueSurfaceVisibility = (typeof ISSUE_SURFACE_VISIBILITIES)[number]
export const ISSUE_RECOVERY_ACTION_KINDS = [
"missing_disposition",
"deliberate_wait_without_target",
"stranded_assigned_issue",
"workspace_validation",
"configuration_validation",
@ -386,6 +387,8 @@ export const ISSUE_RECOVERY_ACTION_KINDS = [
] as const;
export type IssueRecoveryActionKind = (typeof ISSUE_RECOVERY_ACTION_KINDS)[number];
export const ISSUE_DISPOSITION_REPAIR_RETRY_REASON = "issue_disposition_repair";
export const ISSUE_RECOVERY_ACTION_STATUSES = [
"active",
"escalated",

1
packages/shared/src/index.ts
View File

@ -301,6 +301,7 @@ export {
ISSUE_WATCHDOG_DISCOVERY_KINDS,
ISSUE_SURFACE_VISIBILITIES,
ISSUE_RECOVERY_ACTION_KINDS,
ISSUE_DISPOSITION_REPAIR_RETRY_REASON,
ISSUE_RECOVERY_ACTION_STATUSES,
ISSUE_RECOVERY_ACTION_OWNER_TYPES,
ISSUE_RECOVERY_ACTION_OUTCOMES,

1
packages/shared/src/types/issue.ts
View File

@ -220,6 +220,7 @@ export interface IssueRelationIssueSummary {
assigneeUserId: string | null;
terminalBlockers?: IssueRelationIssueSummary[];
activeRecoveryAction?: IssueRecoveryAction | null;
scheduledRetry?: IssueScheduledRetry | null;
}
export type IssueBlockerDiagnosticFlag =

74
server/src/__tests__/disposition-repair.test.ts Normal file
View File

@ -0,0 +1,74 @@
import { describe, expect, it } from "vitest";
import { ISSUE_RECOVERY_ACTION_KINDS } from "@paperclipai/shared";
import { classifyContinuationFailure } from "../services/recovery/service.ts";
import {
DISPOSITION_REPAIR_BASE_DELAYS_MS,
DISPOSITION_REPAIR_MAX_ATTEMPTS,
RECOVERY_OWNER_BASE_DELAYS_MS,
RECOVERY_OWNER_MAX_ATTEMPTS,
dispositionRepairDelayMs,
recoveryOwnerDelayMs,
} from "../services/recovery/disposition-repair.ts";
const deliberateWaitRun = {
id: "run-1",
agentId: "agent-1",
status: "cancelled" as const,
error: "Continuation parked",
errorCode: "issue_continuation_waiting_on_review",
contextSnapshot: {},
livenessState: null,
startedAt: new Date("2026-08-11T00:00:00.000Z"),
createdAt: new Date("2026-08-11T00:00:00.000Z"),
};
describe("owner-sticky disposition repair", () => {
it("classifies a deliberate wait without a target into its dedicated bounded lane", () => {
expect(ISSUE_RECOVERY_ACTION_KINDS).toContain("deliberate_wait_without_target");
expect(classifyContinuationFailure(deliberateWaitRun)).toEqual({
kind: "deliberate_wait_without_target",
maxAttempts: 5,
baseBackoffMs: 60_000,
errorCode: "issue_continuation_waiting_on_review",
});
});
it("uses the persisted five-attempt owner-sticky schedule with bounded deterministic jitter", () => {
const fingerprint = "disposition_repair:v1:example";
const timings = [1, 2, 3, 4, 5].map((attempt) => dispositionRepairDelayMs(attempt, fingerprint));
expect(DISPOSITION_REPAIR_MAX_ATTEMPTS).toBe(5);
expect(DISPOSITION_REPAIR_BASE_DELAYS_MS).toEqual([0, 60_000, 120_000, 240_000, 480_000]);
expect(timings[0]).toEqual({ baseDelayMs: 0, jitterMs: 0, delayMs: 0 });
expect(timings[1]?.baseDelayMs).toBe(60_000);
expect(timings[1]?.jitterMs).toBeGreaterThanOrEqual(0);
expect(timings[1]?.jitterMs).toBeLessThanOrEqual(6_000);
expect(timings[2]?.baseDelayMs).toBe(120_000);
expect(timings[2]?.jitterMs).toBeGreaterThanOrEqual(0);
expect(timings[2]?.jitterMs).toBeLessThanOrEqual(12_000);
expect(timings[3]?.baseDelayMs).toBe(240_000);
expect(timings[3]?.jitterMs).toBeLessThanOrEqual(24_000);
expect(timings[4]?.baseDelayMs).toBe(480_000);
expect(timings[4]?.jitterMs).toBeLessThanOrEqual(48_000);
expect(dispositionRepairDelayMs(2, fingerprint)).toEqual(timings[1]);
expect(() => dispositionRepairDelayMs(6, fingerprint)).toThrow(/Invalid disposition repair attempt/);
});
it("keeps recovery-owner retries separate and bounded", () => {
const fingerprint = "disposition_repair:v1:manager";
const timings = [1, 2, 3, 4, 5].map((attempt) => recoveryOwnerDelayMs(attempt, fingerprint));
expect(RECOVERY_OWNER_MAX_ATTEMPTS).toBe(5);
expect(RECOVERY_OWNER_BASE_DELAYS_MS).toEqual([0, 60_000, 120_000, 240_000, 480_000]);
expect(timings[0]).toEqual({ baseDelayMs: 0, jitterMs: 0, delayMs: 0 });
expect(timings[1]?.baseDelayMs).toBe(60_000);
expect(timings[1]?.jitterMs).toBeLessThanOrEqual(6_000);
expect(timings[2]?.baseDelayMs).toBe(120_000);
expect(timings[2]?.jitterMs).toBeLessThanOrEqual(12_000);
expect(timings[3]?.baseDelayMs).toBe(240_000);
expect(timings[3]?.jitterMs).toBeLessThanOrEqual(24_000);
expect(timings[4]?.baseDelayMs).toBe(480_000);
expect(timings[4]?.jitterMs).toBeLessThanOrEqual(48_000);
expect(() => recoveryOwnerDelayMs(6, fingerprint)).toThrow(/Invalid recovery owner attempt/);
});
});

667
server/src/__tests__/heartbeat-process-recovery.test.ts
View File

@ -119,6 +119,7 @@ import {
SUCCESSFUL_RUN_MISSING_STATE_REASON,
noticeMetadataReferencesRecoveryAction,
} from "../services/recovery/index.ts";
import { collectDispositionRepairSourceState } from "../services/recovery/disposition-repair.ts";
import {
UNMANAGED_BACKGROUND_TASK_LIVENESS_REASON,
UNMANAGED_BACKGROUND_TASK_STOP_REASON,
@ -3261,7 +3262,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
status: "failed",
error: "Failed to start command",
errorCode: "adapter_failed",
scheduledRetryAttempt: 3,
scheduledRetryAttempt: 5,
scheduledRetryReason: INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
contextSnapshot: {
issueId,
@ -3285,12 +3286,12 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
const result = await heartbeat.scheduleBoundedRetry(runId, {
retryReason: INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
wakeReason: INTERACTION_CONTINUATION_INFRA_WAKE_REASON,
maxAttempts: 3,
maxAttempts: 5,
});
expect(result).toMatchObject({
outcome: "retry_exhausted",
maxAttempts: 3,
maxAttempts: 5,
});
const issue = await db
@ -3329,8 +3330,8 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
resumeFailure: {
status: "needs_attention",
errorCode: "adapter_failed",
attempt: 3,
maxAttempts: 3,
attempt: 5,
maxAttempts: 5,
runId,
recoveryActionId: recoveryAction?.id ?? null,
},
@ -4300,8 +4301,8 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
expect(comments[0]?.body).not.toContain("issue_continuation_waiting_on_review");
});
it("still escalates a continuation parked for review when no open dependency remains", async () => {
const { companyId, issueId } = await seedStrandedIssueFixture({
it("repairs the PAP-16986 deliberate wait through the original owner when no target exists", async () => {
const { companyId, agentId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
@ -4312,10 +4313,627 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
const heartbeat = heartbeatService(db);
const result = await heartbeat.reconcileStrandedAssignedIssues();
// With no real waiting target, the deliberate-wait conversion must not fire;
// genuine-strand detection downstream is preserved.
expect(result.waitingOnReviewResolved).toBe(0);
expect(result.continuationRequeued).toBe(1);
expect(result.dispositionRepairRequeued).toBe(1);
expect(result.escalated).toBe(0);
await expect(sourceBlockerIssueIds(companyId, issueId)).resolves.toEqual([]);
const sourceIssue = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0] ?? null);
expect(sourceIssue).toMatchObject({
status: "in_progress",
assigneeAgentId: agentId,
});
const action = await db
.select()
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
))
.then((rows) => rows[0] ?? null);
expect(action).toMatchObject({
kind: "deliberate_wait_without_target",
status: "active",
ownerAgentId: agentId,
previousOwnerAgentId: agentId,
returnOwnerAgentId: agentId,
attemptCount: 1,
maxAttempts: 5,
});
expect(action?.fingerprint).toMatch(/^disposition_repair:v1:/);
expect(action?.wakePolicy).toMatchObject({
type: "bounded_owner_disposition_repair",
retryAgentId: agentId,
attempt: 1,
maxAttempts: 5,
baseBackoffMs: 0,
jitterMs: 0,
});
const repairRun = await waitForValue(async () => {
const rows = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId));
return rows.find((run) =>
(run.contextSnapshot as { retryReason?: string } | null)?.retryReason ===
"issue_disposition_repair"
) ?? null;
});
expect(repairRun?.contextSnapshot).toMatchObject({
issueId,
retryReason: "issue_disposition_repair",
dispositionRepairFingerprint: action?.fingerprint,
dispositionRepairAttempt: 1,
dispositionRepairMaxAttempts: 5,
});
expect(repairRun?.contextSnapshot).not.toHaveProperty("modelProfile");
expect(repairRun?.contextSnapshot).not.toHaveProperty("allowDeliverableWork");
});
it("folds a persisted disposition-repair action when a current typed wait appears", async () => {
const { companyId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
const sourceIssue = await db.select().from(issues).where(eq(issues.id, issueId)).then((rows) => rows[0]!);
const sourceState = await collectDispositionRepairSourceState(db, { issue: sourceIssue });
const action = await db
.insert(issueRecoveryActions)
.values({
companyId,
sourceIssueId: issueId,
kind: "deliberate_wait_without_target",
status: "active",
ownerType: "agent",
ownerAgentId: sourceIssue.assigneeAgentId,
previousOwnerAgentId: sourceIssue.assigneeAgentId,
returnOwnerAgentId: sourceIssue.assigneeAgentId,
cause: "deliberate_wait_without_target",
fingerprint: sourceState.fingerprint,
evidence: { sourceStateFingerprint: sourceState.fingerprint },
nextAction: "Record a durable disposition.",
wakePolicy: { type: "bounded_owner_disposition_repair", attempt: 1, maxAttempts: 5 },
attemptCount: 1,
maxAttempts: 5,
timeoutAt: new Date(Date.now() - 60_000),
})
.returning()
.then((rows) => rows[0]!);
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, issueId));
await db.insert(issueThreadInteractions).values({
id: randomUUID(),
companyId,
issueId,
kind: "request_confirmation",
status: "pending",
continuationPolicy: "wake_assignee",
payload: { version: 1, prompt: "Confirm the current disposition." },
});
await heartbeatService(db).reconcileStrandedAssignedIssues();
const folded = await db
.select()
.from(issueRecoveryActions)
.where(eq(issueRecoveryActions.id, action.id))
.then((rows) => rows[0] ?? null);
expect(folded).toMatchObject({
status: "resolved",
outcome: "restored",
resolutionNote: "durable_path_restored:interaction",
attemptCount: 1,
maxAttempts: 5,
});
});
it("reschedules an expired persisted disposition repair without duplicating the retry", async () => {
const { companyId, agentId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
const sourceIssue = await db.select().from(issues).where(eq(issues.id, issueId)).then((rows) => rows[0]!);
const sourceState = await collectDispositionRepairSourceState(db, { issue: sourceIssue });
const action = await db
.insert(issueRecoveryActions)
.values({
companyId,
sourceIssueId: issueId,
kind: "deliberate_wait_without_target",
status: "active",
ownerType: "agent",
ownerAgentId: agentId,
previousOwnerAgentId: agentId,
returnOwnerAgentId: agentId,
cause: "deliberate_wait_without_target",
fingerprint: sourceState.fingerprint,
evidence: { sourceStateFingerprint: sourceState.fingerprint },
nextAction: "Record a durable disposition.",
wakePolicy: { type: "bounded_owner_disposition_repair", attempt: 1, maxAttempts: 5 },
attemptCount: 1,
maxAttempts: 5,
timeoutAt: new Date(Date.now() - 60_000),
})
.returning()
.then((rows) => rows[0]!);
const restartedHeartbeat = heartbeatService(db);
await restartedHeartbeat.reconcileStrandedAssignedIssues();
await restartedHeartbeat.reconcileStrandedAssignedIssues();
const [rescheduledAction, retries] = await Promise.all([
db
.select()
.from(issueRecoveryActions)
.where(eq(issueRecoveryActions.id, action.id))
.then((rows) => rows[0] ?? null),
db
.select()
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, agentId),
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId' = ${action.id}`,
sql`${heartbeatRuns.contextSnapshot} ->> 'dispositionRepairAttempt' = '2'`,
)),
]);
expect(rescheduledAction).toMatchObject({
status: "active",
attemptCount: 2,
maxAttempts: 5,
});
expect(rescheduledAction?.wakePolicy).toMatchObject({
type: "bounded_owner_disposition_repair",
attempt: 2,
maxAttempts: 5,
});
expect(retries).toHaveLength(1);
expect(retries[0]).toMatchObject({
status: "scheduled_retry",
scheduledRetryAttempt: 2,
scheduledRetryReason: "issue_disposition_repair",
});
});
it("atomically deduplicates concurrent disposition-repair reconciliation", async () => {
const { companyId, agentId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
const sourceIssue = await db.select().from(issues).where(eq(issues.id, issueId)).then((rows) => rows[0]!);
const sourceState = await collectDispositionRepairSourceState(db, { issue: sourceIssue });
const action = await db
.insert(issueRecoveryActions)
.values({
companyId,
sourceIssueId: issueId,
kind: "deliberate_wait_without_target",
status: "active",
ownerType: "agent",
ownerAgentId: agentId,
previousOwnerAgentId: agentId,
returnOwnerAgentId: agentId,
cause: "deliberate_wait_without_target",
fingerprint: sourceState.fingerprint,
evidence: { sourceStateFingerprint: sourceState.fingerprint },
nextAction: "Record a durable disposition.",
wakePolicy: { type: "bounded_owner_disposition_repair", attempt: 1, maxAttempts: 5 },
attemptCount: 1,
maxAttempts: 5,
timeoutAt: new Date(Date.now() - 60_000),
})
.returning()
.then((rows) => rows[0]!);
await Promise.all([
heartbeatService(db).reconcileStrandedAssignedIssues(),
heartbeatService(db).reconcileStrandedAssignedIssues(),
]);
const [requests, retries, scheduledActivities] = await Promise.all([
db
.select()
.from(agentWakeupRequests)
.where(and(
eq(agentWakeupRequests.companyId, companyId),
sql`${agentWakeupRequests.idempotencyKey} LIKE 'issue_disposition_repair:%'`,
sql`${agentWakeupRequests.status} <> 'skipped'`,
)),
db
.select()
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, agentId),
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId' = ${action.id}`,
sql`${heartbeatRuns.contextSnapshot} ->> 'dispositionRepairAttempt' = '2'`,
)),
db
.select()
.from(activityLog)
.where(and(
eq(activityLog.companyId, companyId),
eq(activityLog.action, "issue.disposition_repair_scheduled"),
eq(activityLog.entityId, action.id),
)),
]);
expect(requests).toHaveLength(1);
expect(retries).toHaveLength(1);
expect(scheduledActivities).toHaveLength(1);
});
it("does not reset disposition repair for prose but does reset for durable source state", async () => {
const { companyId, agentId, runId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
const sourceIssue = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0]!);
const initial = await collectDispositionRepairSourceState(db, { issue: sourceIssue });
await db.insert(issueComments).values({
companyId,
issueId,
authorAgentId: agentId,
body: "Parked summary: waiting for review, with no typed target.",
});
const afterProse = await collectDispositionRepairSourceState(db, { issue: sourceIssue });
expect(afterProse.fingerprint).toBe(initial.fingerprint);
await db
.update(issues)
.set({ executionPolicy: { mode: "auto" } })
.where(eq(issues.id, issueId));
const changedIssue = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0]!);
const afterDurableChange = await collectDispositionRepairSourceState(db, { issue: changedIssue });
expect(afterDurableChange.fingerprint).not.toBe(initial.fingerprint);
await db
.update(heartbeatRuns)
.set({
contextSnapshot: {
issueId,
taskId: issueId,
wakeReason: "issue_disposition_repair",
retryReason: "issue_disposition_repair",
dispositionRepairFingerprint: initial.fingerprint,
dispositionRepairAttempt: 5,
dispositionRepairMaxAttempts: 5,
},
})
.where(eq(heartbeatRuns.id, runId));
const result = await heartbeatService(db).reconcileStrandedAssignedIssues();
expect(result.dispositionRepairRequeued).toBe(1);
expect(result.escalated).toBe(0);
const action = await db
.select()
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
))
.then((rows) => rows[0] ?? null);
expect(action).toMatchObject({
fingerprint: afterDurableChange.fingerprint,
attemptCount: 1,
maxAttempts: 5,
status: "active",
});
});
it("refuses source and manager attempt six without transferring the source", async () => {
const { companyId, agentId, runId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
const managerId = randomUUID();
await db.insert(agents).values({
id: managerId,
companyId,
name: "Recovery CTO",
role: "cto",
status: "idle",
adapterType: "codex_local",
adapterConfig: {},
runtimeConfig: {},
permissions: {},
});
await db.update(agents).set({ reportsTo: managerId }).where(eq(agents.id, agentId));
const sourceIssue = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0]!);
const state = await collectDispositionRepairSourceState(db, { issue: sourceIssue });
await db
.update(heartbeatRuns)
.set({
contextSnapshot: {
issueId,
taskId: issueId,
wakeReason: "issue_disposition_repair",
retryReason: "issue_disposition_repair",
dispositionRepairFingerprint: state.fingerprint,
dispositionRepairAttempt: 5,
dispositionRepairMaxAttempts: 5,
},
})
.where(eq(heartbeatRuns.id, runId));
const result = await heartbeatService(db).reconcileStrandedAssignedIssues();
expect(result.dispositionRepairRequeued).toBe(0);
expect(result.escalated).toBe(1);
const sourceAfter = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0] ?? null);
expect(sourceAfter).toMatchObject({
status: "blocked",
assigneeAgentId: agentId,
});
const action = await db
.select()
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
))
.then((rows) => rows[0] ?? null);
expect(action).toMatchObject({
kind: "deliberate_wait_without_target",
status: "active",
ownerAgentId: managerId,
previousOwnerAgentId: agentId,
returnOwnerAgentId: agentId,
attemptCount: 1,
maxAttempts: 5,
resolutionNote: "unchanged_source_state_exhausted",
});
expect(action?.evidence).toMatchObject({
terminalReason: "unchanged_source_state_exhausted",
sourceAttemptCount: 5,
sourceMaxAttempts: 5,
});
const sourceAttemptSix = await db
.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, agentId),
sql`${heartbeatRuns.contextSnapshot} ->> 'dispositionRepairAttempt' = '6'`,
));
expect(sourceAttemptSix).toHaveLength(0);
expect(action?.wakePolicy).toMatchObject({
type: "bounded_recovery_owner",
attempt: 1,
maxAttempts: 5,
preservesSourceAssignee: true,
});
const firstManagerRun = await waitForValue(async () => db
.select()
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, managerId),
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId' = ${action!.id}`,
))
.then((rows) => rows[0] ?? null));
await db
.update(heartbeatRuns)
.set({ status: "failed", errorCode: "process_lost", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, firstManagerRun.id));
await db
.update(agentWakeupRequests)
.set({ status: "completed" })
.where(eq(agentWakeupRequests.runId, firstManagerRun.id));
const restartedHeartbeat = heartbeatService(db);
for (let attempt = 2; attempt <= 5; attempt += 1) {
await restartedHeartbeat.reconcileStrandedAssignedIssues();
const managerRun = await db
.select()
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, managerId),
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId' = ${action!.id}`,
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryOwnerAttempt' = ${String(attempt)}`,
))
.then((rows) => rows[0] ?? null);
expect(managerRun).toMatchObject({
status: "scheduled_retry",
scheduledRetryAttempt: attempt,
scheduledRetryReason: "recovery_owner_retry",
});
await restartedHeartbeat.reconcileStrandedAssignedIssues();
const duplicateRuns = await db
.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, managerId),
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId' = ${action!.id}`,
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryOwnerAttempt' = ${String(attempt)}`,
));
expect(duplicateRuns).toHaveLength(1);
await db
.update(heartbeatRuns)
.set({ status: "failed", errorCode: "process_lost", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, managerRun!.id));
await db
.update(agentWakeupRequests)
.set({ status: "completed" })
.where(eq(agentWakeupRequests.runId, managerRun!.id));
}
await restartedHeartbeat.reconcileStrandedAssignedIssues();
const exhaustedAction = await db
.select()
.from(issueRecoveryActions)
.where(eq(issueRecoveryActions.id, action!.id))
.then((rows) => rows[0] ?? null);
expect(exhaustedAction).toMatchObject({
status: "escalated",
ownerType: "board",
ownerAgentId: null,
attemptCount: 5,
maxAttempts: 5,
resolutionNote: "recovery_owner_retry_exhausted",
});
expect(exhaustedAction?.wakePolicy).toMatchObject({
type: "board_escalation",
reason: "recovery_owner_retry_exhausted",
attempt: 5,
maxAttempts: 5,
preservesSourceAssignee: true,
});
const managerAttemptSix = await db
.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.agentId, managerId),
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId' = ${action!.id}`,
sql`${heartbeatRuns.contextSnapshot} ->> 'recoveryOwnerAttempt' = '6'`,
));
expect(managerAttemptSix).toHaveLength(0);
const sourceAfterManagerExhaustion = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0] ?? null);
expect(sourceAfterManagerExhaustion).toMatchObject({
status: "blocked",
assigneeAgentId: agentId,
});
});
it("routes a non-invokable source owner to recovery without reassigning the source", async () => {
const { companyId, agentId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId));
const result = await heartbeatService(db).reconcileStrandedAssignedIssues();
expect(result.escalated).toBe(1);
const sourceIssue = await db
.select()
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0] ?? null);
expect(sourceIssue).toMatchObject({
status: "blocked",
assigneeAgentId: agentId,
});
const action = await db
.select()
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
))
.then((rows) => rows[0] ?? null);
expect(action).toMatchObject({
kind: "deliberate_wait_without_target",
status: "escalated",
ownerType: "board",
ownerAgentId: null,
previousOwnerAgentId: agentId,
returnOwnerAgentId: agentId,
attemptCount: 0,
maxAttempts: 5,
resolutionNote: "owner_not_invokable",
});
});
it("does not consume a disposition-repair attempt when on-demand wakes are disabled", async () => {
const { companyId, agentId, issueId } = await seedStrandedIssueFixture({
status: "in_progress",
runStatus: "cancelled",
retryReason: "issue_continuation_needed",
runErrorCode: "issue_continuation_waiting_on_review",
});
await db
.update(agents)
.set({ runtimeConfig: { heartbeat: { wakeOnDemand: false } } })
.where(eq(agents.id, agentId));
const result = await heartbeatService(db).reconcileStrandedAssignedIssues();
expect(result.dispositionRepairRequeued).toBe(0);
expect(result.escalated).toBe(1);
const [action, repairWakeups] = await Promise.all([
db
.select()
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
))
.then((rows) => rows[0] ?? null),
db
.select()
.from(agentWakeupRequests)
.where(and(
eq(agentWakeupRequests.companyId, companyId),
eq(agentWakeupRequests.agentId, agentId),
sql`${agentWakeupRequests.idempotencyKey} LIKE 'issue_disposition_repair:%'`,
)),
]);
expect(action).toMatchObject({
kind: "deliberate_wait_without_target",
status: "escalated",
ownerType: "board",
ownerAgentId: null,
attemptCount: 0,
resolutionNote: "owner_not_invokable",
});
expect(repairWakeups).toHaveLength(0);
});
it("clears the detached warning when the run reports activity again", async () => {
@ -5156,7 +5774,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
});
expect(sourceIssue).toMatchObject({
status: "blocked",
assigneeAgentId: agentId,
assigneeAgentId: sourceAssigneeAgentId,
});
const recoveryAction = await expectSourceScopedStrandedRecoveryAction({
@ -5349,7 +5967,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
});
});
it("escalates accepted interaction continuation recovery after three review-park cancellations", async () => {
it("counts five historical review-park cancellations against the upgraded disposition-repair ceiling", async () => {
const companyId = randomUUID();
const agentId = randomUUID();
const issueId = randomUUID();
@ -5400,7 +6018,7 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
payload: { version: 1, prompt: "Approve the plan?" },
result: { outcome: "accepted" },
});
for (let attempt = 1; attempt <= 3; attempt += 1) {
for (let attempt = 1; attempt <= 5; attempt += 1) {
const finishedAt = new Date(resolvedAt.getTime() + attempt * 60_000);
await db.insert(heartbeatRuns).values({
id: randomUUID(),
@ -5449,8 +6067,29 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
db.select({ body: issueComments.body }).from(issueComments).where(eq(issueComments.issueId, issueId)),
]);
expect(issue?.status).toBe("blocked");
expect(continuationRuns).toHaveLength(3);
expect(comments.some((comment) => comment.body.includes(interactionId))).toBe(true);
expect(continuationRuns).toHaveLength(5);
expect(comments.some((comment) => comment.body.includes("Attempts: 5/5"))).toBe(true);
const action = await db
.select()
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.companyId, companyId),
eq(issueRecoveryActions.sourceIssueId, issueId),
))
.then((rows) => rows[0] ?? null);
expect(action).toMatchObject({
kind: "deliberate_wait_without_target",
status: "escalated",
previousOwnerAgentId: agentId,
returnOwnerAgentId: agentId,
attemptCount: 0,
maxAttempts: 5,
resolutionNote: "unchanged_source_state_exhausted",
});
expect(action?.evidence).toMatchObject({
sourceAttemptCount: 5,
sourceMaxAttempts: 5,
});
});
it("skips accepted interaction recovery after its continuation succeeds", async () => {

259
server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts
View File

@ -51,6 +51,14 @@ const mockCompanyService = vi.hoisted(() => ({
getById: vi.fn(),
}));
const mockBudgetService = vi.hoisted(() => ({
getInvocationBlock: vi.fn(async () => null),
}));
const mockProjectService = vi.hoisted(() => ({
getById: vi.fn(async () => null),
}));
const mockDocumentService = vi.hoisted(() => ({
upsertIssueDocument: vi.fn(),
}));
@ -193,6 +201,7 @@ function registerRouteMocks() {
ISSUE_LIST_MAX_LIMIT: 500,
accessService: () => mockAccessService,
agentService: () => mockAgentService,
budgetService: () => mockBudgetService,
clampIssueListLimit: (value: number) => Math.min(Math.max(value, 1), 500),
companySkillService: () => ({
completeTestRunForIssue: vi.fn(async () => null),
@ -236,7 +245,7 @@ function registerRouteMocks() {
issueThreadInteractionService: () => mockIssueThreadInteractionService,
taskWatchdogService: () => mockTaskWatchdogService,
logActivity: mockLogActivity,
projectService: () => ({}),
projectService: () => mockProjectService,
routineService: () => ({
syncRunStatusForIssue: vi.fn(async () => undefined),
}),
@ -260,6 +269,8 @@ function makeIssue(overrides: Record<string, unknown> = {}) {
title: "Owned active issue",
executionPolicy: null,
executionState: null,
checkoutRunId: null,
executionRunId: null,
hiddenAt: null,
...overrides,
};
@ -293,21 +304,27 @@ function createRunContextDb(
const firstRun = runRows[0] ?? {};
const runAgentId = typeof firstRun.agentId === "string" ? firstRun.agentId : ownerAgentId;
const runAgentCompanyId = typeof firstRun.agentCompanyId === "string" ? firstRun.agentCompanyId : companyId;
const rowsForSelection = (selection: Record<string, unknown>) => {
const rowsForSelection = async (selection: Record<string, unknown>) => {
const keys = Object.keys(selection);
if (keys.includes("entityId")) return [];
if (keys.includes("contextSnapshot")) return runRows;
if (keys.includes("agentCompanyId")) return runRows;
if (keys.length === 0) {
const issue = await mockIssueService.getById(issueId);
return issue ? [issue] : [];
}
return [{ id: runAgentId, companyId: runAgentCompanyId, permissions: {}, role: "engineer", reportsTo: null }];
};
const buildQuery = (selection: Record<string, unknown>) => {
const rows = rowsForSelection(selection);
const whereResult = {
orderBy: vi.fn(async () => []),
limit: vi.fn(() => ({
then: async (resolve: (limitedRows: unknown[]) => unknown) => resolve(rows),
then: async (resolve: (limitedRows: unknown[]) => unknown) => resolve(await rowsForSelection(selection)),
})),
then: async (resolve: (selectedRows: unknown[]) => unknown) => resolve(rows),
for: vi.fn(() => ({
then: async (resolve: (selectedRows: unknown[]) => unknown) => resolve(await rowsForSelection(selection)),
})),
then: async (resolve: (selectedRows: unknown[]) => unknown) => resolve(await rowsForSelection(selection)),
};
const query = {
innerJoin: vi.fn(() => query),
@ -315,12 +332,14 @@ function createRunContextDb(
};
return query;
};
return {
transaction: async (callback: (tx: Record<string, never>) => Promise<unknown>) => callback({}),
const dbStub = {
transaction: async (callback: (tx: typeof dbStub) => Promise<unknown>) => callback(dbStub),
select: vi.fn((selection: Record<string, unknown> = {}) => ({
from: vi.fn(() => buildQuery(selection)),
})),
insert: vi.fn(() => ({ values: vi.fn(async () => undefined) })),
};
return dbStub;
}
async function createApp(actor: Record<string, unknown>, db?: unknown) {
@ -426,6 +445,10 @@ describe("agent issue mutation checkout ownership", () => {
mockAgentService.list.mockReset();
mockAgentService.resolveByReference.mockReset();
mockCompanyService.getById.mockReset();
mockBudgetService.getInvocationBlock.mockReset();
mockBudgetService.getInvocationBlock.mockResolvedValue(null);
mockProjectService.getById.mockReset();
mockProjectService.getById.mockResolvedValue(null);
mockIssueService.addComment.mockReset();
mockIssueService.assertCheckoutOwner.mockReset();
mockIssueService.create.mockReset();
@ -1705,14 +1728,10 @@ describe("agent issue mutation checkout ownership", () => {
expect(mockIssueRecoveryActionService.resolveActiveForIssue).not.toHaveBeenCalled();
});
it("allows the named recovery owner to resolve a board-owned source issue", async () => {
it("rejects the named recovery owner completing a board-owned source issue", async () => {
mockIssueService.getById.mockResolvedValue(
makeIssue({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" }),
);
mockIssueService.update.mockImplementation(async (_id: string, patch: Record<string, unknown>) => ({
...makeIssue({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" }),
...patch,
}));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId,
@ -1726,8 +1745,144 @@ describe("agent issue mutation checkout ownership", () => {
sourceIssueStatus: "done",
});
expect(res.status, JSON.stringify(res.body)).toBe(403);
expect(res.body.details?.code).toBe("recovery_source_authority_required");
expect(mockIssueService.update).not.toHaveBeenCalled();
expect(mockIssueRecoveryActionService.resolveActiveForIssue).not.toHaveBeenCalled();
});
it("rejects a recovery owner completing an independently agent-owned source issue", async () => {
mockIssueService.getById.mockResolvedValue(
makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }),
);
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
});
const res = await request(await createApp(peerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({
actionId: recoveryActionId,
outcome: "restored",
sourceIssueStatus: "done",
});
expect(res.status, JSON.stringify(res.body)).toBe(403);
expect(res.body.details?.code).toBe("recovery_source_authority_required");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it.each(["done", "cancelled"])(
"rejects recovery-owner PATCH of an agent-owned source to %s",
async (status) => {
mockIssueService.getById.mockResolvedValue(
makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }),
);
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
});
const res = await request(await createApp(peerActor()))
.patch(`/api/issues/${issueId}`)
.send({ status });
expect(res.status, JSON.stringify(res.body)).toBe(403);
expect(res.body.details?.code).toBe("recovery_source_authority_required");
expect(mockIssueService.update).not.toHaveBeenCalled();
},
);
it("rejects recovery-owner reassignment of an independently agent-owned source", async () => {
mockIssueService.getById.mockResolvedValue(
makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }),
);
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
});
mockAgentService.resolveByReference.mockResolvedValue({
ambiguous: false,
agent: makeAgent(peerAgentId),
});
const res = await request(await createApp(peerActor()))
.patch(`/api/issues/${issueId}`)
.send({ assigneeAgentId: peerAgentId });
expect(res.status, JSON.stringify(res.body)).toBe(403);
expect(res.body.details?.code).toBe("recovery_source_authority_required");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it("rejects a recovery owner who is not the current governed review participant", async () => {
mockIssueService.getById.mockResolvedValue(makeIssue({
status: "in_review",
assigneeAgentId: ownerAgentId,
executionState: {
status: "pending",
currentStageId: "88888888-8888-4888-8888-888888888888",
currentStageIndex: 0,
currentStageType: "review",
currentParticipant: { type: "agent", agentId: ownerAgentId },
returnAssignee: { type: "agent", agentId: ownerAgentId },
completedStageIds: [],
lastDecisionId: null,
lastDecisionOutcome: null,
},
}));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
});
const res = await request(await createApp(peerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "done" });
expect(res.status, JSON.stringify(res.body)).toBe(403);
expect(res.body.details?.code).toBe("recovery_source_authority_required");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it("keeps configured review policy authoritative during recovery resolution", async () => {
mockIssueService.getById.mockResolvedValue(makeIssue({
status: "in_review",
assigneeAgentId: ownerAgentId,
reviewPolicy: "human_only",
}));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId,
});
const res = await request(await createApp(ownerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "done" });
expect(res.status, JSON.stringify(res.body)).toBe(403);
expect(res.body.details?.code).toBe("review_policy_denied");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it("allows a recovery owner to record a receipt without mutating a board-owned source", async () => {
mockIssueService.getById.mockResolvedValue(makeIssue({
status: "in_review",
assigneeAgentId: null,
assigneeUserId: "board-user",
}));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId,
});
const res = await request(await createApp(ownerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "in_review" });
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(mockIssueService.update).toHaveBeenCalled();
expect(mockIssueService.update).not.toHaveBeenCalled();
expect(mockIssueRecoveryActionService.resolveActiveForIssue).toHaveBeenCalled();
});
@ -1741,10 +1896,11 @@ describe("agent issue mutation checkout ownership", () => {
}));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId,
ownerAgentId: peerAgentId,
returnOwnerAgentId: ownerAgentId,
});
const res = await request(await createApp(ownerActor()))
const res = await request(await createApp(peerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({
actionId: recoveryActionId,
@ -1753,6 +1909,12 @@ describe("agent issue mutation checkout ownership", () => {
});
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(mockIssueService.update).toHaveBeenCalledWith(
issueId,
expect.not.objectContaining({ assigneeAgentId: expect.anything() }),
expect.anything(),
expect.any(Array),
);
expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith(
ownerAgentId,
expect.objectContaining({
@ -1766,6 +1928,73 @@ describe("agent issue mutation checkout ownership", () => {
);
});
it.each([
["checkoutRunId", ownerRunId],
["executionRunId", ownerRunId],
])("blocks safe hand-back while the source has an active %s", async (lockField, lockRunId) => {
mockIssueService.getById.mockResolvedValue(makeIssue({
status: "blocked",
assigneeAgentId: ownerAgentId,
[lockField]: lockRunId,
}));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
returnOwnerAgentId: ownerAgentId,
});
const res = await request(await createApp(peerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "todo" });
expect(res.status, JSON.stringify(res.body)).toBe(409);
expect(res.body.details?.code).toBe("recovery_source_run_lock");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it("blocks safe hand-back while the original owner's budget is paused", async () => {
mockIssueService.getById.mockResolvedValue(
makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }),
);
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
returnOwnerAgentId: ownerAgentId,
});
mockBudgetService.getInvocationBlock.mockResolvedValue({
scope: "agent",
reason: "hard_limit_reached",
});
const res = await request(await createApp(peerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "todo" });
expect(res.status, JSON.stringify(res.body)).toBe(409);
expect(res.body.details?.code).toBe("recovery_safe_hand_back_budget_blocked");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it("blocks safe hand-back while a governed approval remains pending", async () => {
mockIssueService.getById.mockResolvedValue(
makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }),
);
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId,
ownerAgentId: peerAgentId,
returnOwnerAgentId: ownerAgentId,
});
mockIssueApprovalService.listApprovalsForIssue.mockResolvedValue([{ status: "pending" }]);
const res = await request(await createApp(peerActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "todo" });
expect(res.status, JSON.stringify(res.body)).toBe(409);
expect(res.body.details?.code).toBe("recovery_governed_approval_pending");
expect(mockIssueService.update).not.toHaveBeenCalled();
});
it("uses the authorization decision path for assignment changes", async () => {
const decide = vi.fn(async () => ({
allowed: false,

22
server/src/__tests__/issue-blocker-attention.test.ts
View File

@ -131,7 +131,14 @@ describeEmbeddedPostgres("issue blocker attention", () => {
});
}
async function activeRun(input: { companyId: string; agentId: string; issueId: string; status?: string; current?: boolean }) {
async function activeRun(input: {
companyId: string;
agentId: string;
issueId: string;
status?: string;
current?: boolean;
scheduledRetryAt?: Date;
}) {
const runId = randomUUID();
await db.insert(heartbeatRuns).values({
id: runId,
@ -139,6 +146,7 @@ describeEmbeddedPostgres("issue blocker attention", () => {
agentId: input.agentId,
status: input.status ?? "running",
contextSnapshot: { issueId: input.issueId },
scheduledRetryAt: input.scheduledRetryAt,
});
if (input.current !== false) {
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, input.issueId));
@ -602,7 +610,9 @@ describeEmbeddedPostgres("issue blocker attention", () => {
});
});
it("does not treat a scheduled retry as actively covered work", async () => {
it("does not treat an expired scheduled retry as actively covered work", async () => {
const pinnedNow = new Date("2026-08-20T12:00:00.000Z");
const expiredRetryAt = new Date(pinnedNow.getTime() - 60_000);
const { companyId, agentId } = await createCompany("PBY");
const parentId = await insertIssue({ companyId, identifier: "PBY-1", title: "Parent", status: "blocked" });
const blockerId = await insertIssue({
@ -613,7 +623,13 @@ describeEmbeddedPostgres("issue blocker attention", () => {
assigneeAgentId: agentId,
});
await block({ companyId, blockerIssueId: blockerId, blockedIssueId: parentId });
await activeRun({ companyId, agentId, issueId: blockerId, status: "scheduled_retry" });
await activeRun({
companyId,
agentId,
issueId: blockerId,
status: "scheduled_retry",
scheduledRetryAt: expiredRetryAt,
});
const parent = (await svc.list(companyId, { status: "blocked" })).find((issue) => issue.id === parentId);

25
server/src/__tests__/issue-recovery-actions.test.ts
View File

@ -882,7 +882,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
expect(updatedIssue).toMatchObject({
status: "blocked",
assigneeAgentId: managerId,
assigneeAgentId: coderId,
});
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(updatedRun?.errorCode).toBe("configuration_incomplete");
@ -1412,7 +1412,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
await db
.update(issues)
.set({ status: "blocked", assigneeAgentId: managerId })
.set({ status: "blocked", assigneeAgentId: coderId })
.where(eq(issues.id, sourceIssueId));
const recoveryActionSvc = issueRecoveryActionService(db);
const action = await recoveryActionSvc.upsertSourceScoped({
@ -1832,7 +1832,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
});
});
it("allows the named recovery owner to resolve a board-owned source recovery action", async () => {
it("keeps the named recovery owner from completing a board-owned source issue", async () => {
const { companyId, managerId, sourceIssueId } = await seedCompany();
await db
.update(issues)
@ -1874,18 +1874,15 @@ describeEmbeddedPostgres("issue recovery actions", () => {
sourceIssueStatus: "done",
resolutionNote: "Recovery owner verified the work was intentionally completed.",
})
.expect(200);
.expect(403);
expect(resolved.body.issue).toMatchObject({
id: sourceIssueId,
status: "done",
activeRecoveryAction: null,
});
expect(resolved.body.recoveryAction).toMatchObject({
id: action.id,
status: "resolved",
outcome: "owner_completed",
});
expect(resolved.body.details?.code).toBe("recovery_source_authority_required");
const [sourceAfter, actionAfter] = await Promise.all([
db.select().from(issues).where(eq(issues.id, sourceIssueId)).then((rows) => rows[0]),
db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)).then((rows) => rows[0]),
]);
expect(sourceAfter).toMatchObject({ status: "blocked", assigneeUserId: "board-user" });
expect(actionAfter).toMatchObject({ status: "active", outcome: null });
});
it("rejects blocked recovery resolution when the source issue has no first-class blockers", async () => {

69
server/src/__tests__/issue-scheduled-retry-routes.test.ts
View File

@ -223,6 +223,53 @@ describeEmbeddedPostgres("issue scheduled retry routes", () => {
expect(res.body.scheduledRetry.scheduledRetryAt).toBe(scheduledRetryAt.toISOString());
});
it.each(["queued", "running"] as const)(
"surfaces a %s retry with its real live status",
async (retryStatus) => {
const { companyId, issueId, retryRunId } = await seedIssueWithRetry({ retryStatus });
const res = await request(createApp(boardActor(companyId, "local_implicit"))).get(`/api/issues/${issueId}`);
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(res.body.scheduledRetry).toMatchObject({
runId: retryRunId,
status: retryStatus,
scheduledRetryReason: "transient_failure",
});
},
);
it("includes a blocker's live retry in relation summaries", async () => {
const { companyId, issueId: blockerId, retryRunId } = await seedIssueWithRetry({ retryStatus: "running" });
const blockedId = randomUUID();
await db.insert(issues).values({
id: blockedId,
companyId,
title: "Waiting for recovery",
status: "blocked",
priority: "medium",
});
await db.insert(issueRelations).values({
companyId,
issueId: blockerId,
relatedIssueId: blockedId,
type: "blocks",
});
const res = await request(createApp(boardActor(companyId, "local_implicit"))).get(`/api/issues/${blockedId}`);
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(res.body.blockedBy).toHaveLength(1);
expect(res.body.blockedBy[0]).toMatchObject({
id: blockerId,
scheduledRetry: {
runId: retryRunId,
status: "running",
scheduledRetryReason: "transient_failure",
},
});
});
it("promotes the existing scheduled retry and treats duplicate clicks as idempotent", async () => {
const { companyId, issueId, retryRunId } = await seedIssueWithRetry();
const app = createApp(boardActor(companyId));
@ -338,6 +385,28 @@ describeEmbeddedPostgres("issue scheduled retry routes", () => {
});
});
it("does not promote a scheduled retry after on-demand wakes are disabled", async () => {
const { companyId, agentId, issueId, retryRunId } = await seedIssueWithRetry();
await db
.update(agents)
.set({ runtimeConfig: { heartbeat: { wakeOnDemand: false } } })
.where(eq(agents.id, agentId));
const res = await request(createApp(boardActor(companyId)))
.post(`/api/issues/${issueId}/scheduled-retry/retry-now`)
.send({});
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(res.body).toMatchObject({
outcome: "gate_suppressed",
scheduledRetry: {
runId: retryRunId,
status: "cancelled",
errorCode: "heartbeat_wake_on_demand_disabled",
},
});
});
it("requires board access for retry-now", async () => {
const { companyId, agentId, issueId } = await seedIssueWithRetry();

350
server/src/routes/issues.ts
View File

@ -112,6 +112,7 @@ import * as serviceIndex from "../services/index.js";
import {
accessService,
agentService,
budgetService,
companySkillService,
companyService,
companySearchService,
@ -5123,9 +5124,8 @@ export function issueRoutes(
return false;
}
async function assertRecoveryActionAuthority(
async function requireRecoveryActionAuthority(
req: Request,
res: Response,
issue: { id: string; companyId: string; assigneeAgentId: string | null },
activeRecoveryAction: Awaited<ReturnType<typeof recoveryActionsSvc.getActiveForIssue>>,
input: { source: "issue_update" | "recovery_action_resolution" },
@ -5135,8 +5135,7 @@ export function issueRoutes(
const actorAgentId = req.actor.agentId;
if (!actorAgentId) {
res.status(403).json({ error: "Agent authentication required" });
return false;
throw forbidden("Agent authentication required");
}
if (issue.assigneeAgentId === actorAgentId) return true;
if (
@ -5153,9 +5152,9 @@ export function issueRoutes(
return true;
}
res.status(403).json({
error: "Agent cannot resolve another owner's recovery action",
details: {
throw forbidden(
"Agent cannot resolve another owner's recovery action",
{
issueId: issue.id,
recoveryActionId: activeRecoveryAction.id,
actorAgentId,
@ -5164,8 +5163,157 @@ export function issueRoutes(
source: input.source,
securityPrinciples: ["Least Privilege", "Complete Mediation", "Secure Defaults"],
},
});
return false;
);
}
function activeExecutionParticipantAgentId(issue: { executionState?: unknown }) {
const state = parseIssueExecutionState(issue.executionState);
return state?.status === "pending" && state.currentParticipant?.type === "agent"
? state.currentParticipant.agentId
: null;
}
async function requireRecoverySourceMutationAuthority(
req: Request,
issue: {
id: string;
companyId: string;
status: string;
assigneeAgentId: string | null;
checkoutRunId?: string | null;
executionRunId?: string | null;
executionState?: unknown;
},
) {
if (req.actor.type !== "agent") return;
const actorAgentId = req.actor.agentId;
if (!actorAgentId) throw forbidden("Agent authentication required");
const isSourceOwner = issue.assigneeAgentId === actorAgentId;
const isExecutionParticipant = activeExecutionParticipantAgentId(issue) === actorAgentId;
const hasPolicyGrant = Boolean(
issue.assigneeAgentId &&
await hasActiveCheckoutManagementOverride(actorAgentId, issue.companyId, issue.assigneeAgentId)
);
if (!isSourceOwner && !isExecutionParticipant && !hasPolicyGrant) {
throw forbidden(
"Recovery ownership does not authorize this source issue mutation",
{
code: "recovery_source_authority_required",
issueId: issue.id,
actorAgentId,
assigneeAgentId: issue.assigneeAgentId,
currentExecutionParticipantAgentId: activeExecutionParticipantAgentId(issue),
remediation:
"Have the source owner, current execution participant, board, or a policy-authorized agent perform the source mutation.",
securityPrinciples: ["Least Privilege", "Complete Mediation", "Secure Defaults"],
},
);
}
const actorRunId = req.actor.runId?.trim() || null;
const conflictingRunId = [issue.checkoutRunId, issue.executionRunId]
.find((runId) => runId && runId !== actorRunId);
if (conflictingRunId && !hasPolicyGrant) {
throw conflict("Source issue mutation is locked by another active checkout or run", {
code: "recovery_source_run_lock",
issueId: issue.id,
actorAgentId,
actorRunId,
checkoutRunId: issue.checkoutRunId ?? null,
executionRunId: issue.executionRunId ?? null,
});
}
if (isSourceOwner && issue.status === "in_progress" && !actorRunId && !hasPolicyGrant) {
throw unauthorized("Agent run id required");
}
}
async function assertSafeRecoveryHandBackGates(input: {
req: Request;
issue: {
id: string;
companyId: string;
projectId: string | null;
assigneeAgentId: string | null;
checkoutRunId?: string | null;
executionRunId?: string | null;
executionState?: unknown;
};
recoveryAction: NonNullable<Awaited<ReturnType<typeof recoveryActionsSvc.getActiveForIssue>>>;
}) {
const returnOwnerAgentId = input.recoveryAction.returnOwnerAgentId;
if (!returnOwnerAgentId || input.issue.assigneeAgentId !== returnOwnerAgentId) {
throw forbidden(
"Safe recovery hand-back requires the recorded original owner to remain assigned",
{
code: "recovery_safe_hand_back_owner_mismatch",
issueId: input.issue.id,
assigneeAgentId: input.issue.assigneeAgentId,
returnOwnerAgentId,
},
);
}
const actorRunId = input.req.actor.type === "agent"
? input.req.actor.runId?.trim() || null
: null;
const conflictingRunId = [input.issue.checkoutRunId, input.issue.executionRunId]
.find((runId) => runId && runId !== actorRunId);
if (conflictingRunId) {
throw conflict("Safe recovery hand-back is locked by another active checkout or run", {
code: "recovery_source_run_lock",
issueId: input.issue.id,
actorRunId,
checkoutRunId: input.issue.checkoutRunId ?? null,
executionRunId: input.issue.executionRunId ?? null,
});
}
if (parseIssueExecutionState(input.issue.executionState)?.status === "pending") {
throw conflict("Safe recovery hand-back cannot bypass a pending execution review or approval stage", {
code: "recovery_governed_stage_pending",
issueId: input.issue.id,
});
}
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(input.issue.companyId, input.issue.id);
if (activePauseHold) {
throw conflict("Safe recovery hand-back blocked by active subtree pause hold", {
issueId: input.issue.id,
holdId: activePauseHold.holdId,
rootIssueId: activePauseHold.rootIssueId,
mode: activePauseHold.mode,
});
}
if (input.issue.projectId) {
const project = await projectsSvc.getById(input.issue.projectId);
if (project?.pausedAt) {
throw conflict(
project.pauseReason === "budget"
? "Project is paused because its budget hard-stop was reached"
: "Project is paused",
);
}
}
const approvals = await issueApprovalsSvc.listApprovalsForIssue(input.issue.id);
if (approvals.some((approval) => ACTIVE_REVIEW_APPROVAL_STATUSES.has(String(approval.status)))) {
throw conflict("Safe recovery hand-back cannot bypass a pending governed approval", {
code: "recovery_governed_approval_pending",
issueId: input.issue.id,
});
}
const budgetBlock = await budgetService(db).getInvocationBlock(
input.issue.companyId,
returnOwnerAgentId,
{ issueId: input.issue.id, projectId: input.issue.projectId },
);
if (budgetBlock) {
throw conflict("Safe recovery hand-back is blocked by the source owner's budget or pause gate", {
code: "recovery_safe_hand_back_budget_blocked",
issueId: input.issue.id,
returnOwnerAgentId,
budgetBlock,
});
}
}
async function resolveActiveIssueRun(issue: {
@ -6686,18 +6834,16 @@ export function issueRoutes(
const id = req.params.id as string;
const existing = await getAccessibleResource(req, res, svc.getById(id), "Issue not found");
if (!existing) return;
if (!(await assertAgentIssueMutationAllowed(req, res, existing))) return;
const activeRecoveryAction = await recoveryActionsSvc.getActiveForIssue(existing.companyId, existing.id);
if (
!(await assertRecoveryActionAuthority(
req,
res,
existing,
activeRecoveryAction,
{ source: "recovery_action_resolution" },
))
) {
return;
if (!(await assertIssueReadAllowed(req, res, existing))) return;
if (await assertLowTrustControlPlaneDenied(req, res, existing.companyId, existing)) return;
if (req.actor.type === "agent") {
const boundaryDecision = await decideIssueAccess(req, existing, "issue:mutate");
if (!boundaryDecision.allowed) {
await denyIssueWrite(req, res, existing, issueWriteDenialCodeForDecision(boundaryDecision));
return;
}
if (!requireAgentRunId(req, res)) return;
if (!(await assertCrossIssueInfluenceWithinRunCap(req, res, existing, "update"))) return;
}
const { actionId, outcome, sourceIssueStatus, resolutionNote } = req.body;
@ -6706,29 +6852,35 @@ export function issueRoutes(
}
const actor = getActorInfo(req);
const handBackAgentId = outcome === "restored" && sourceIssueStatus === "todo"
? activeRecoveryAction?.returnOwnerAgentId ?? null
: null;
const recordedOutcome = handBackAgentId
? "handed_back"
: outcome === "restored" && sourceIssueStatus === "done"
? "owner_completed"
: outcome;
const updateFields = sourceIssueStatus ? { status: sourceIssueStatus } : {};
await assertInReviewReviewPath({
existing,
updateFields,
actorType: actor.actorType,
actorId: actor.actorId,
actorAgentId: actor.agentId,
actorRunId: actor.runId,
});
const actionStatus = outcome === "cancelled" ? "cancelled" : "resolved";
const postCommitActivityPublications: ActivityPublication[] = [];
const result = await db.transaction(async (tx) => {
let issue = existing;
if (outcome === "blocked") {
const lockedIssue = await tx
.select()
.from(issueRows)
.where(and(eq(issueRows.companyId, existing.companyId), eq(issueRows.id, existing.id)))
.for("update")
.then((rows) => rows[0] ?? null);
if (!lockedIssue) throw notFound("Issue not found");
const activeRecoveryAction = await recoveryActionsSvc.getActiveForIssue(
lockedIssue.companyId,
lockedIssue.id,
tx,
);
if (!activeRecoveryAction || (actionId && activeRecoveryAction.id !== actionId)) {
throw notFound("Active recovery action not found");
}
await requireRecoveryActionAuthority(
req,
lockedIssue,
activeRecoveryAction,
{ source: "recovery_action_resolution" },
);
let issue = lockedIssue;
const sourceStatusChanged = sourceIssueStatus !== lockedIssue.status;
if (outcome === "blocked" && sourceStatusChanged) {
const unresolvedBlockers = await tx
.select({ id: issueRows.id })
.from(issueRelations)
@ -6747,12 +6899,85 @@ export function issueRoutes(
}
}
if (sourceIssueStatus) {
if (sourceStatusChanged) {
const safeHandBack =
outcome === "restored" &&
sourceIssueStatus === "todo" &&
activeRecoveryAction.returnOwnerAgentId != null &&
lockedIssue.assigneeAgentId === activeRecoveryAction.returnOwnerAgentId;
if (safeHandBack) {
await assertSafeRecoveryHandBackGates({
req,
issue: lockedIssue,
recoveryAction: activeRecoveryAction,
});
} else {
await requireRecoverySourceMutationAuthority(req, lockedIssue);
}
if (
lockedIssue.status === "in_review" &&
(sourceIssueStatus === "done" || sourceIssueStatus === "cancelled") &&
lockedIssue.reviewPolicy != null &&
lockedIssue.reviewPolicy !== "anyone"
) {
await assertIssueReviewVerdictActorAllowed(tx as unknown as Db, {
issue: lockedIssue,
actor: { type: actor.actorType, id: actor.actorId },
});
}
const updateFields: Record<string, unknown> = { status: sourceIssueStatus };
if (!safeHandBack) {
await assertInReviewReviewPath({
existing: lockedIssue,
updateFields,
actorType: actor.actorType,
actorId: actor.actorId,
actorAgentId: actor.agentId,
actorRunId: actor.runId,
});
const executionPolicy = normalizeIssueExecutionPolicy(lockedIssue.executionPolicy ?? null);
const transition = applyIssueExecutionPolicyTransition({
issue: lockedIssue,
policy: executionPolicy,
previousPolicy: executionPolicy,
requestedStatus: sourceIssueStatus,
requestedAssigneePatch: {},
actor: {
agentId: actor.agentId ?? null,
userId: actor.actorType === "user" ? actor.actorId : null,
},
allowBoardOverride: req.actor.type === "board",
commentBody: resolutionNote ?? null,
});
Object.assign(updateFields, transition.patch);
if (transition.decision) {
const decisionId = randomUUID();
const nextExecutionState = updateFields.executionState;
if (!nextExecutionState || typeof nextExecutionState !== "object") {
throw new Error("Execution policy decision patch is missing executionState");
}
updateFields.executionState = { ...nextExecutionState, lastDecisionId: decisionId };
await tx.insert(issueExecutionDecisions).values({
id: decisionId,
companyId: lockedIssue.companyId,
issueId: lockedIssue.id,
stageId: transition.decision.stageId,
stageType: transition.decision.stageType,
actorAgentId: actor.agentId ?? null,
actorUserId: actor.actorType === "user" ? actor.actorId : null,
outcome: transition.decision.outcome,
body: transition.decision.body,
createdByRunId: actor.runId ?? null,
});
}
}
const updatedIssue = await svc.update(
id,
{
status: sourceIssueStatus,
...(handBackAgentId ? { assigneeAgentId: handBackAgentId } : {}),
...updateFields,
actorAgentId: actor.agentId ?? null,
actorUserId: actor.actorType === "user" ? actor.actorId : null,
},
@ -6763,11 +6988,20 @@ export function issueRoutes(
issue = updatedIssue;
}
const recordedOutcome =
outcome === "restored" && issue.status === "todo" &&
activeRecoveryAction.returnOwnerAgentId != null &&
issue.assigneeAgentId === activeRecoveryAction.returnOwnerAgentId
? "handed_back"
: outcome === "restored" && issue.status === "done"
? "owner_completed"
: outcome;
const recoveryAction = await recoveryActionsSvc.resolveActiveForIssue(
{
companyId: existing.companyId,
sourceIssueId: existing.id,
actionId: actionId ?? null,
actionId: activeRecoveryAction.id,
status: actionStatus,
outcome: recordedOutcome,
resolutionNote: resolutionNote ?? null,
@ -9126,17 +9360,29 @@ export function issueRoutes(
const activeRecoveryActionBeforeUpdate = recoveryRelevantSourceMutationRequested
? await recoveryActionsSvc.getActiveForIssue(existing.companyId, existing.id)
: null;
if (
recoveryRelevantSourceMutationRequested &&
!(await assertRecoveryActionAuthority(
if (recoveryRelevantSourceMutationRequested) {
await requireRecoveryActionAuthority(
req,
res,
existing,
activeRecoveryActionBeforeUpdate,
{ source: "issue_update" },
))
) {
return;
);
const recoveryRestrictedSourceMutationRequested =
activeRecoveryActionBeforeUpdate != null &&
(
updateFields.status === "done" ||
updateFields.status === "cancelled" ||
normalizedAssigneeAgentId !== undefined ||
req.body.assigneeUserId !== undefined ||
(
activeExecutionParticipantAgentId(existing) != null &&
typeof updateFields.status === "string" &&
updateFields.status !== existing.status
)
);
if (recoveryRestrictedSourceMutationRequested) {
await requireRecoverySourceMutationAuthority(req, existing);
}
}
if (
resumeRequested !== true &&

16
server/src/services/heartbeat-policy.ts Normal file
View File

@ -0,0 +1,16 @@
import { agents } from "@paperclipai/db";
import { asBoolean, parseObject } from "../adapters/utils.js";
export function isHeartbeatWakeOnDemandEnabled(
agent: Pick<typeof agents.$inferSelect, "runtimeConfig">,
) {
const runtimeConfig = parseObject(agent.runtimeConfig);
const heartbeat = parseObject(runtimeConfig.heartbeat);
return asBoolean(
heartbeat.wakeOnDemand ??
heartbeat.wakeOnAssignment ??
heartbeat.wakeOnOnDemand ??
heartbeat.wakeOnAutomation,
true,
);
}

76
server/src/services/heartbeat.ts
View File

@ -8,6 +8,7 @@ import type { Db } from "@paperclipai/db";
import {
AGENT_DEFAULT_MAX_CONCURRENT_RUNS,
ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY,
ISSUE_DISPOSITION_REPAIR_RETRY_REASON,
MODEL_PROFILE_KEYS,
PROVIDER_QUOTA_MONITOR_SERVICE_NAME,
envBindingSchema,
@ -236,6 +237,7 @@ import {
withRecoveryModelProfileHint,
} from "./recovery/model-profile-hint.js";
import { ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS as RECOVERY_ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, recoveryService } from "./recovery/service.js";
import { collectDispositionRepairSourceState } from "./recovery/disposition-repair.js";
import {
buildIssueReviewPathLostIdempotencyKey,
decideIssueReviewPathRecovery,
@ -255,6 +257,7 @@ import {
DIRECT_NON_INVOKABLE_STATUSES,
type AgentOrgRow,
} from "./agent-invokability.js";
import { isHeartbeatWakeOnDemandEnabled } from "./heartbeat-policy.js";
import {
redactQuarantinedBodyForHigherTrust,
sanitizeQuarantinedCommentForHigherTrust,
@ -10789,6 +10792,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
reason: string;
errorCode:
| "agent_not_invokable"
| "heartbeat_wake_on_demand_disabled"
| "budget_blocked"
| "issue_not_found"
| "issue_reassigned"
@ -10798,7 +10802,8 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
| "issue_execution_lock_changed"
| "issue_review_participant_changed"
| "issue_paused"
| "issue_dependencies_blocked";
| "issue_dependencies_blocked"
| "issue_disposition_repair_superseded";
issueId: string | null;
details: Record<string, unknown>;
};
@ -10848,15 +10853,29 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
};
}
if (!isHeartbeatWakeOnDemandEnabled(agent)) {
return {
allowed: false,
reason: "Scheduled retry suppressed because on-demand agent wakes are disabled",
errorCode: "heartbeat_wake_on_demand_disabled",
issueId,
details: { agentId: agent.id },
};
}
if (!issueId) return { allowed: true };
const issue = await db
.select({
id: issues.id,
companyId: issues.companyId,
status: issues.status,
assigneeAgentId: issues.assigneeAgentId,
assigneeUserId: issues.assigneeUserId,
executionRunId: issues.executionRunId,
executionPolicy: issues.executionPolicy,
executionState: issues.executionState,
monitorNextCheckAt: issues.monitorNextCheckAt,
})
.from(issues)
.where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId)))
@ -10872,6 +10891,35 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
};
}
if (retryReason === ISSUE_DISPOSITION_REPAIR_RETRY_REASON) {
const expectedFingerprint = readNonEmptyString(contextSnapshot.dispositionRepairFingerprint);
const sourceState = await collectDispositionRepairSourceState(db, {
issue,
excludeRunId: run.id,
excludeWakeupRequestId: run.wakeupRequestId,
});
if (
!expectedFingerprint ||
sourceState.fingerprint !== expectedFingerprint ||
sourceState.hasActiveExecutionPath ||
sourceState.hasDurableWaitingPath
) {
return {
allowed: false,
reason: "Scheduled disposition repair suppressed because the source state changed or gained a durable path",
errorCode: "issue_disposition_repair_superseded",
issueId,
details: {
issueId,
expectedFingerprint,
currentFingerprint: sourceState.fingerprint,
hasActiveExecutionPath: sourceState.hasActiveExecutionPath,
durablePathReason: sourceState.durablePathReason,
},
};
}
}
if (issue.assigneeAgentId !== run.agentId) {
if (!isNonAssigneeWorkspaceBusyRetry(retryReason, contextSnapshot)) {
return {
@ -12234,7 +12282,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
return {
enabled: asBoolean(heartbeat.enabled, false),
intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)),
wakeOnDemand: asBoolean(heartbeat.wakeOnDemand ?? heartbeat.wakeOnAssignment ?? heartbeat.wakeOnOnDemand ?? heartbeat.wakeOnAutomation, true),
wakeOnDemand: isHeartbeatWakeOnDemandEnabled(agent),
maxConcurrentRuns: normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns),
skipTimerWhenNoActionableWork: asBoolean(
heartbeat.skipTimerWhenNoActionableWork ??
@ -12788,10 +12836,27 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
const isCurrentReviewParticipant = reviewParticipant?.type === "agent" &&
reviewParticipant.agentId === run.agentId;
const recoveryActionId = readNonEmptyString(context.recoveryActionId);
const authorizedSourceScopedRecovery = wakeReason === "source_scoped_recovery_action" && recoveryActionId
? await db
.select({ id: issueRecoveryActions.id })
.from(issueRecoveryActions)
.where(and(
eq(issueRecoveryActions.id, recoveryActionId),
eq(issueRecoveryActions.companyId, run.companyId),
eq(issueRecoveryActions.sourceIssueId, issue.id),
eq(issueRecoveryActions.ownerAgentId, run.agentId),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
))
.limit(1)
.then((rows) => Boolean(rows[0]))
: false;
if (
issue.assigneeAgentId !== run.agentId &&
!isInteractionWake &&
!isCurrentReviewParticipant &&
!authorizedSourceScopedRecovery &&
!isNonAssigneeWorkspaceBusyRetry(retryReason, context)
) {
return {
@ -17356,6 +17421,13 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
issue.assigneeAgentId === run.agentId &&
(run.status === "failed" || run.status === "timed_out" || run.status === "cancelled");
if (
readNonEmptyString(parseObject(run.contextSnapshot).retryReason) ===
ISSUE_DISPOSITION_REPAIR_RETRY_REASON
) {
return { kind: "released" as const };
}
if (!issueNeedsImmediateRecovery) {
return { kind: "released" as const };
}

13
server/src/services/issue-recovery-actions.ts
View File

@ -35,6 +35,7 @@ export type UpsertIssueRecoveryActionInput = {
maxAttempts?: number | null;
timeoutAt?: Date | null;
lastAttemptAt?: Date | null;
attemptCount?: number;
};
export type ResolveIssueRecoveryActionInput = {
@ -123,8 +124,12 @@ export function issueRecoveryActionService(db: Db) {
}
}
async function getActiveForIssue(companyId: string, sourceIssueId: string): Promise<IssueRecoveryAction | null> {
const row = await db
async function getActiveForIssue(
companyId: string,
sourceIssueId: string,
dbOrTx: DbOrTransaction = db,
): Promise<IssueRecoveryAction | null> {
const row = await dbOrTx
.select()
.from(issueRecoveryActions)
.where(
@ -199,7 +204,7 @@ export function issueRecoveryActionService(db: Db) {
nextAction: input.nextAction,
wakePolicy: input.wakePolicy ?? null,
monitorPolicy: input.monitorPolicy ?? null,
attemptCount: existing.attemptCount + 1,
attemptCount: input.attemptCount ?? existing.attemptCount + 1,
maxAttempts: input.maxAttempts ?? null,
timeoutAt: input.timeoutAt ?? null,
lastAttemptAt: input.lastAttemptAt ?? now,
@ -241,7 +246,7 @@ export function issueRecoveryActionService(db: Db) {
nextAction: input.nextAction,
wakePolicy: input.wakePolicy ?? null,
monitorPolicy: input.monitorPolicy ?? null,
attemptCount: 1,
attemptCount: input.attemptCount ?? 1,
maxAttempts: input.maxAttempts ?? null,
timeoutAt: input.timeoutAt ?? null,
lastAttemptAt: input.lastAttemptAt ?? now,

123
server/src/services/issues.ts
View File

@ -1,6 +1,6 @@
import { Buffer } from "node:buffer";
import { createHash, randomUUID } from "node:crypto";
import { and, asc, desc, eq, gt, gte, inArray, isNull, like, lt, ne, notInArray, or, sql, type SQL } from "drizzle-orm";
import { and, asc, desc, eq, gt, gte, inArray, isNotNull, isNull, like, lt, ne, notInArray, or, sql, type SQL } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
activityLog,
@ -2553,8 +2553,22 @@ async function listIssueBlockerAttentionMap(
explicitWaitingIssueIds.add(parsed.leafIssueId);
}
const recoveryActionRows: Array<{ sourceIssueId: string }> = await dbOrTx
.select({ sourceIssueId: issueRecoveryActions.sourceIssueId })
const recoveryActionRows: Array<{
id: string;
sourceIssueId: string;
status: string;
ownerType: string;
ownerAgentId: string | null;
ownerUserId: string | null;
}> = await dbOrTx
.select({
id: issueRecoveryActions.id,
sourceIssueId: issueRecoveryActions.sourceIssueId,
status: issueRecoveryActions.status,
ownerType: issueRecoveryActions.ownerType,
ownerAgentId: issueRecoveryActions.ownerAgentId,
ownerUserId: issueRecoveryActions.ownerUserId,
})
.from(issueRecoveryActions)
.where(
and(
@ -2563,7 +2577,38 @@ async function listIssueBlockerAttentionMap(
inArray(issueRecoveryActions.sourceIssueId, explicitWaitCandidateIds),
),
);
for (const row of recoveryActionRows) explicitWaitingIssueIds.add(row.sourceIssueId);
const recoveryActionIds = recoveryActionRows.map((row) => row.id);
const liveRecoveryActionIds = new Set<string>();
for (const chunk of chunkList(recoveryActionIds, ISSUE_LIST_RELATED_QUERY_CHUNK_SIZE)) {
const [runRows, wakeRows] = await Promise.all([
dbOrTx
.select({ recoveryActionId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId'` })
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
inArray(heartbeatRuns.status, BLOCKER_ATTENTION_ACTIVE_RUN_STATUSES),
inArray(sql<string>`${heartbeatRuns.contextSnapshot} ->> 'recoveryActionId'`, chunk),
)),
dbOrTx
.select({ recoveryActionId: sql<string | null>`${agentWakeupRequests.payload} ->> 'recoveryActionId'` })
.from(agentWakeupRequests)
.where(and(
eq(agentWakeupRequests.companyId, companyId),
inArray(agentWakeupRequests.status, BLOCKER_ATTENTION_ACTIVE_WAKE_STATUSES),
inArray(sql<string>`${agentWakeupRequests.payload} ->> 'recoveryActionId'`, chunk),
)),
]);
for (const row of [...runRows, ...wakeRows]) {
if (row.recoveryActionId) liveRecoveryActionIds.add(row.recoveryActionId);
}
}
for (const row of recoveryActionRows) {
const healthy =
(row.status === "escalated" && row.ownerType === "board") ||
Boolean(row.ownerUserId) ||
(Boolean(row.ownerAgentId) && liveRecoveryActionIds.has(row.id));
if (healthy) explicitWaitingIssueIds.add(row.sourceIssueId);
}
}
const agentRows: IssueBlockerAttentionAgentRow[] = agentIds.size > 0
@ -2708,7 +2753,12 @@ async function listIssueBlockerAttentionMap(
if (seen.has(nodeId)) return false;
const node = nodesById.get(nodeId);
if (!node || node.companyId !== companyId) return false;
if (node.status === "in_progress" || activeIssueIds.has(node.id)) return true;
if (
node.status === "in_progress" ||
activeIssueIds.has(node.id) ||
explicitWaitingIssueIds.has(node.id) ||
Boolean(node.assigneeUserId)
) return true;
const nextSeen = new Set(seen);
nextSeen.add(nodeId);
@ -4425,9 +4475,18 @@ export function issueService(db: Db) {
return enriched;
}
async function getCurrentScheduledRetryForIssue(issueId: string, companyId: string): Promise<IssueScheduledRetryRow | null> {
const row = await db
async function getCurrentScheduledRetriesForIssues(
issueIds: string[],
companyId: string,
dbOrTx: DbReader = db,
): Promise<Map<string, IssueScheduledRetryRow>> {
const uniqueIssueIds = [...new Set(issueIds)];
if (uniqueIssueIds.length === 0) return new Map();
const contextIssueId = sql<string>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`;
const rows = await dbOrTx
.select({
issueId: contextIssueId,
runId: heartbeatRuns.id,
status: heartbeatRuns.status,
agentId: heartbeatRuns.agentId,
@ -4444,15 +4503,35 @@ export function issueService(db: Db) {
.where(
and(
eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.status, "scheduled_retry"),
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issueId}`,
inArray(heartbeatRuns.status, ["scheduled_retry", "queued", "running"]),
isNotNull(heartbeatRuns.scheduledRetryReason),
inArray(contextIssueId, uniqueIssueIds),
),
)
.orderBy(asc(heartbeatRuns.scheduledRetryAt), asc(heartbeatRuns.createdAt), asc(heartbeatRuns.id))
.limit(1)
.then((rows) => rows[0] ?? null);
.orderBy(
sql`case ${heartbeatRuns.status}
when 'running' then 0
when 'queued' then 1
else 2
end`,
asc(heartbeatRuns.scheduledRetryAt),
asc(heartbeatRuns.createdAt),
asc(heartbeatRuns.id),
);
return row ? { ...row, status: "scheduled_retry" } : null;
const currentByIssueId = new Map<string, IssueScheduledRetryRow>();
for (const row of rows) {
if (currentByIssueId.has(row.issueId)) continue;
const status = row.status;
if (status !== "scheduled_retry" && status !== "queued" && status !== "running") continue;
currentByIssueId.set(row.issueId, { ...row, status });
}
return currentByIssueId;
}
async function getCurrentScheduledRetryForIssue(issueId: string, companyId: string): Promise<IssueScheduledRetryRow | null> {
const currentByIssueId = await getCurrentScheduledRetriesForIssues([issueId], companyId);
return currentByIssueId.get(issueId) ?? null;
}
function deriveIssueCommentAuthorType(comment: {
@ -4984,6 +5063,24 @@ export function issueService(db: Db) {
relations.blocks.sort((a, b) => a.title.localeCompare(b.title));
}
const relationSummaries: IssueRelationIssueSummary[] = [];
const collectRelationSummary = (summary: IssueRelationIssueSummary) => {
relationSummaries.push(summary);
for (const terminal of summary.terminalBlockers ?? []) collectRelationSummary(terminal);
};
for (const relations of empty.values()) {
for (const blocker of relations.blockedBy) collectRelationSummary(blocker);
for (const blocking of relations.blocks) collectRelationSummary(blocking);
}
const scheduledRetryByIssueId = await getCurrentScheduledRetriesForIssues(
relationSummaries.map((summary) => summary.id),
companyId,
dbOrTx,
);
for (const summary of relationSummaries) {
summary.scheduledRetry = scheduledRetryByIssueId.get(summary.id) ?? null;
}
return empty;
}

251
server/src/services/recovery/disposition-repair.ts Normal file
View File

@ -0,0 +1,251 @@
import { createHash } from "node:crypto";
import { and, eq, inArray, ne, notInArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agentWakeupRequests,
approvals,
heartbeatRuns,
issueApprovals,
issueRelations,
issueThreadInteractions,
issueWorkProducts,
issues,
} from "@paperclipai/db";
import { parseIssueExecutionState } from "../issue-execution-policy.js";
const ACTIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"] as const;
export const DISPOSITION_REPAIR_MAX_ATTEMPTS = 5;
export const DISPOSITION_REPAIR_BASE_DELAYS_MS = [0, 60_000, 120_000, 240_000, 480_000] as const;
export const RECOVERY_OWNER_MAX_ATTEMPTS = 5;
export const RECOVERY_OWNER_BASE_DELAYS_MS = [0, 60_000, 120_000, 240_000, 480_000] as const;
type DispositionRepairIssue = Pick<
typeof issues.$inferSelect,
| "id"
| "companyId"
| "status"
| "assigneeAgentId"
| "assigneeUserId"
| "executionPolicy"
| "executionState"
| "monitorNextCheckAt"
>;
export type DispositionRepairSourceState = {
fingerprint: string;
dependencyIssueIds: string[];
hasActiveExecutionPath: boolean;
hasDurableWaitingPath: boolean;
durablePathReason: string | null;
};
function stableJson(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
if (value && typeof value === "object") {
const record = value as Record<string, unknown>;
return `{${Object.keys(record).sort().map((key) => `${JSON.stringify(key)}:${stableJson(record[key])}`).join(",")}}`;
}
return JSON.stringify(value);
}
function boundedRecoveryDelayMs(
attempt: number,
fingerprint: string,
delays: readonly number[],
lane: string,
) {
const baseDelayMs = delays[attempt - 1];
if (baseDelayMs === undefined) throw new Error(`Invalid ${lane} attempt: ${attempt}`);
if (baseDelayMs === 0) return { baseDelayMs, jitterMs: 0, delayMs: 0 };
const jitterBoundMs = Math.floor(baseDelayMs * 0.1);
const sample = Number.parseInt(
createHash("sha256").update(`${fingerprint}:${attempt}`).digest("hex").slice(0, 8),
16,
);
const jitterMs = sample % (jitterBoundMs + 1);
return { baseDelayMs, jitterMs, delayMs: baseDelayMs + jitterMs };
}
export function dispositionRepairDelayMs(attempt: number, fingerprint: string) {
return boundedRecoveryDelayMs(
attempt,
fingerprint,
DISPOSITION_REPAIR_BASE_DELAYS_MS,
"disposition repair",
);
}
export function recoveryOwnerDelayMs(attempt: number, fingerprint: string) {
return boundedRecoveryDelayMs(
attempt,
fingerprint,
RECOVERY_OWNER_BASE_DELAYS_MS,
"recovery owner",
);
}
export async function collectDispositionRepairSourceState(
db: Db,
input: {
issue: DispositionRepairIssue;
excludeRunId?: string | null;
excludeWakeupRequestId?: string | null;
},
): Promise<DispositionRepairSourceState> {
const issue = input.issue;
const [blockers, children, interactions, linkedApprovals, workProducts, activeRuns, queuedWakes] =
await Promise.all([
db
.select({ id: issues.id, status: issues.status, assigneeAgentId: issues.assigneeAgentId })
.from(issueRelations)
.innerJoin(
issues,
and(eq(issues.companyId, issueRelations.companyId), eq(issues.id, issueRelations.issueId)),
)
.where(
and(
eq(issueRelations.companyId, issue.companyId),
eq(issueRelations.relatedIssueId, issue.id),
eq(issueRelations.type, "blocks"),
notInArray(issues.status, ["done", "cancelled"]),
sql`${issues.hiddenAt} is null`,
),
),
db
.select({ id: issues.id, status: issues.status, assigneeAgentId: issues.assigneeAgentId })
.from(issues)
.where(
and(
eq(issues.companyId, issue.companyId),
eq(issues.parentId, issue.id),
notInArray(issues.status, ["done", "cancelled"]),
sql`${issues.hiddenAt} is null`,
),
),
db
.select({
id: issueThreadInteractions.id,
status: issueThreadInteractions.status,
kind: issueThreadInteractions.kind,
continuationPolicy: issueThreadInteractions.continuationPolicy,
updatedAt: issueThreadInteractions.updatedAt,
})
.from(issueThreadInteractions)
.where(
and(
eq(issueThreadInteractions.companyId, issue.companyId),
eq(issueThreadInteractions.issueId, issue.id),
inArray(issueThreadInteractions.status, ["pending", "accepted", "answered"]),
),
),
db
.select({ id: approvals.id, status: approvals.status, decidedAt: approvals.decidedAt })
.from(issueApprovals)
.innerJoin(
approvals,
and(
eq(issueApprovals.approvalId, approvals.id),
eq(issueApprovals.companyId, approvals.companyId),
),
)
.where(
and(
eq(issueApprovals.companyId, issue.companyId),
eq(approvals.companyId, issue.companyId),
eq(issueApprovals.issueId, issue.id),
inArray(approvals.status, ["pending", "revision_requested", "approved"]),
),
),
db
.select({
id: issueWorkProducts.id,
type: issueWorkProducts.type,
status: issueWorkProducts.status,
reviewState: issueWorkProducts.reviewState,
updatedAt: issueWorkProducts.updatedAt,
})
.from(issueWorkProducts)
.where(
and(
eq(issueWorkProducts.companyId, issue.companyId),
eq(issueWorkProducts.issueId, issue.id),
),
),
db
.select({ id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, status: heartbeatRuns.status })
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.companyId, issue.companyId),
inArray(heartbeatRuns.status, [...ACTIVE_RUN_STATUSES]),
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issue.id}`,
input.excludeRunId ? ne(heartbeatRuns.id, input.excludeRunId) : sql`true`,
),
),
db
.select({ id: agentWakeupRequests.id, agentId: agentWakeupRequests.agentId, status: agentWakeupRequests.status })
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, issue.companyId),
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]),
sql`${agentWakeupRequests.payload} ->> 'issueId' = ${issue.id}`,
input.excludeWakeupRequestId
? ne(agentWakeupRequests.id, input.excludeWakeupRequestId)
: sql`true`,
),
),
]);
const pendingExecutionState = parseIssueExecutionState(issue.executionState);
const pendingInteraction = interactions.some((row) => row.status === "pending");
const pendingApproval = linkedApprovals.some((row) =>
row.status === "pending" || row.status === "revision_requested",
);
const durablePathReason = issue.assigneeUserId
? "user_owner"
: blockers.length > 0
? "blocker"
: issue.monitorNextCheckAt && issue.monitorNextCheckAt.getTime() > Date.now()
? "monitor"
: pendingExecutionState?.status === "pending"
? "execution_stage"
: pendingInteraction
? "interaction"
: pendingApproval
? "approval"
: null;
const durableState = {
source: {
status: issue.status,
assigneeAgentId: issue.assigneeAgentId,
assigneeUserId: issue.assigneeUserId,
executionPolicy: issue.executionPolicy,
executionState: issue.executionState,
monitorNextCheckAt: issue.monitorNextCheckAt?.toISOString() ?? null,
},
blockers: blockers.sort((a, b) => a.id.localeCompare(b.id)),
children: children.sort((a, b) => a.id.localeCompare(b.id)),
interactions: interactions
.map((row) => ({ ...row, updatedAt: row.updatedAt.toISOString() }))
.sort((a, b) => a.id.localeCompare(b.id)),
approvals: linkedApprovals
.map((row) => ({ ...row, decidedAt: row.decidedAt?.toISOString() ?? null }))
.sort((a, b) => a.id.localeCompare(b.id)),
workProducts: workProducts
.map((row) => ({ ...row, updatedAt: row.updatedAt.toISOString() }))
.sort((a, b) => a.id.localeCompare(b.id)),
};
const digest = createHash("sha256").update(stableJson(durableState)).digest("hex");
return {
fingerprint: `disposition_repair:v1:${digest}`,
dependencyIssueIds: [...new Set([...blockers.map((row) => row.id), ...children.map((row) => row.id)])],
hasActiveExecutionPath: activeRuns.length > 0 || queuedWakes.length > 0,
hasDurableWaitingPath: durablePathReason !== null,
durablePathReason,
};
}

1166
server/src/services/recovery/service.ts
View File

File diff suppressed because it is too large Load Diff

133
ui/src/components/IssueBlockedNotice.test.tsx
View File

@ -6,8 +6,13 @@ import type { AnchorHTMLAttributes, ReactElement, ReactNode } from "react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { MemoryRouter } from "react-router-dom";
import type { IssueRetryNowOutcome, IssueScheduledRetry } from "@paperclipai/shared";
import type {
IssueRecoveryAction,
IssueRetryNowOutcome,
IssueScheduledRetry,
} from "@paperclipai/shared";
import { IssueBlockedNotice } from "./IssueBlockedNotice";
import { deriveRecoveryCardState } from "./IssueRecoveryActionCard";
import { ToastProvider } from "../context/ToastContext";
const retryNowMock = vi.hoisted(() => vi.fn());
@ -778,4 +783,130 @@ describe("IssueBlockedNotice", () => {
expect(indicator?.getAttribute("data-recovery-kind")).toBe("workspace_validation");
expect(indicator?.textContent).toContain("Workspace recovery needed");
});
describe("owner-sticky retry lineage", () => {
function buildDispositionRepairAction(
wakePolicy: Record<string, unknown>,
overrides: Partial<IssueRecoveryAction> = {},
): IssueRecoveryAction {
return {
id: "rec-3",
companyId: "co-1",
sourceIssueId: "blocker-3",
recoveryIssueId: null,
kind: "deliberate_wait_without_target",
status: "active",
ownerType: "agent",
ownerAgentId: "agent-owner",
ownerUserId: null,
previousOwnerAgentId: "agent-owner",
returnOwnerAgentId: "agent-owner",
cause: "deliberate_wait_without_target",
fingerprint: "fp-3",
evidence: {},
nextAction: "Record a durable disposition.",
wakePolicy,
monitorPolicy: null,
attemptCount: 2,
maxAttempts: 5,
timeoutAt: null,
lastAttemptAt: null,
outcome: null,
resolutionNote: null,
resolvedAt: null,
createdAt: "2026-04-18T19:00:00.000Z",
updatedAt: "2026-04-18T19:00:00.000Z",
...overrides,
};
}
function renderBlockerChip(
action: IssueRecoveryAction,
scheduledRetry?: IssueScheduledRetry | null,
) {
return render(
<IssueBlockedNotice
issueStatus="blocked"
blockers={[
{
id: "blocker-3",
identifier: "PAP-777",
title: "Waiting on nothing",
status: "in_progress",
priority: "medium",
assigneeAgentId: null,
assigneeUserId: null,
activeRecoveryAction: action,
scheduledRetry,
},
]}
/>,
).querySelector('[data-testid="issue-blocked-notice-recovery-indicator"]');
}
it("reports the same liveness state as the source task's recovery card", () => {
const liveAction = buildDispositionRepairAction({
type: "bounded_owner_disposition_repair",
retryAgentId: "agent-owner",
attempt: 2,
maxAttempts: 5,
// SYSTEM_NOW is 2026-04-18T20:00:00Z; three minutes out.
retryAt: "2026-04-18T20:03:00.000Z",
scheduledRunId: "run-b1",
});
const chip = renderBlockerChip(liveAction);
expect(chip?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(chip?.getAttribute("data-recovery-lane")).toBe("source_owner");
expect(chip?.textContent).toContain("Recovery in progress · 2/5");
expect(chip?.getAttribute("title")).toContain("Attempt 2 of 5 · next try in 3m");
// The card the blocker links to must not contradict the chip.
const cardState = deriveRecoveryCardState(liveAction);
expect(cardState).toBe(chip?.getAttribute("data-recovery-state"));
});
it("keeps the blocker chip in progress while the named retry run is live", () => {
const liveAction = buildDispositionRepairAction({
type: "bounded_owner_disposition_repair",
retryAgentId: "agent-owner",
attempt: 3,
maxAttempts: 5,
retryAt: "2026-04-18T19:58:00.000Z",
scheduledRunId: "run-b2",
});
const scheduledRetry: IssueScheduledRetry = {
...baseRetry,
runId: "run-b2",
status: "running",
scheduledRetryAt: "2026-04-18T19:58:00.000Z",
scheduledRetryAttempt: 3,
scheduledRetryReason: "issue_disposition_repair",
};
const chip = renderBlockerChip(liveAction, scheduledRetry);
expect(chip?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(chip?.textContent).toContain("Recovery in progress · 3/5");
expect(chip?.getAttribute("title")).toContain("Attempt 3 of 5 · attempt running now");
expect(deriveRecoveryCardState(liveAction, { scheduledRetry })).toBe("in_progress");
});
it("escalates the blocker chip in step with the card when retries are exhausted", () => {
const exhaustedAction = buildDispositionRepairAction(
{
type: "bounded_owner_disposition_repair",
retryAgentId: "agent-owner",
attempt: 5,
maxAttempts: 5,
retryAt: "2026-04-18T19:59:00.000Z",
},
{ attemptCount: 5 },
);
const chip = renderBlockerChip(exhaustedAction);
expect(chip?.getAttribute("data-recovery-state")).toBe("needed");
expect(chip?.textContent).toContain("Recovery needed");
expect(deriveRecoveryCardState(exhaustedAction)).toBe("needed");
});
});
});

36
ui/src/components/IssueBlockedNotice.tsx
View File

@ -27,22 +27,41 @@ import {
RECOVERY_CHIP_DEFAULT_TONE,
recoveryChipLabel,
} from "../lib/recovery-display";
import {
formatRecoveryLineageSummary,
readRecoveryRetryLineage,
} from "../lib/recovery-lineage";
import { StatusGlyph } from "./StatusGlyph";
function BlockerRecoveryIndicator({ action }: { action: IssueRecoveryAction }) {
const state = deriveActiveRecoveryDisplayState(action);
function BlockerRecoveryIndicator({
action,
scheduledRetry,
}: {
action: IssueRecoveryAction;
/** The blocker's own scheduled retry, used to verify that the stored attempt is in flight. */
scheduledRetry?: IssueScheduledRetry | null;
}) {
const liveness = { scheduledRetry: scheduledRetry ?? null };
const state = deriveActiveRecoveryDisplayState(action, liveness);
if (!state) return null;
const tone = RECOVERY_CHIP_DEFAULT_TONE[state];
const Icon = tone.icon;
const label = recoveryChipLabel(state, action.kind);
// The blocker chip reads the same stored lineage as the source task's recovery card, so
// a parent view never contradicts the task it is waiting on.
const lineage = readRecoveryRetryLineage(action, liveness);
const label = recoveryChipLabel(state, action.kind, lineage);
const detail = lineage ? formatRecoveryLineageSummary(lineage) : null;
return (
<Badge variant="outline"
data-testid="issue-blocked-notice-recovery-indicator"
data-recovery-state={state}
data-recovery-kind={action.kind}
data-recovery-lane={lineage?.lane}
role="status"
aria-label={label}
title={`${label} — open the source task to act.`}
aria-label={detail ? `${label} — ${detail}` : label}
title={detail
? `${label} — ${detail}. Open the source task to act.`
: `${label} — open the source task to act.`}
className={`[&>svg]:size-2.5 gap-0.5 px-1.5 text-(length:--text-nano) ${tone.className}`}
>
<Icon className="h-2.5 w-2.5" aria-hidden />
@ -538,7 +557,12 @@ export function IssueBlockedNotice({
<span className="max-w-(--sz-18rem) truncate font-sans text-(length:--text-micro) text-amber-800 dark:text-amber-200">
{blocker.title}
</span>
{recoveryAction ? <BlockerRecoveryIndicator action={recoveryAction} /> : null}
{recoveryAction ? (
<BlockerRecoveryIndicator
action={recoveryAction}
scheduledRetry={blocker.scheduledRetry}
/>
) : null}
</IssueLinkQuicklook>
);
};

1
ui/src/components/IssueChatThread.tsx
View File

@ -5183,6 +5183,7 @@ export function IssueChatThread({
<IssueRecoveryActionCard
action={recoveryAction}
agentMap={agentMap}
scheduledRetry={scheduledRetry}
onResolve={onResolveRecoveryAction}
onReissueIsolated={onReissueIsolatedRecoveryAction}
reissuePending={reissueIsolatedRecoveryActionPending}

305
ui/src/components/IssueRecoveryActionCard.test.tsx
View File

@ -3,7 +3,7 @@
import { createRoot } from "react-dom/client";
import { flushSync } from "react-dom";
import type { AnchorHTMLAttributes, ReactElement } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { Agent, IssueRecoveryAction } from "@paperclipai/shared";
import { IssueRecoveryActionCard, deriveRecoveryCardState } from "./IssueRecoveryActionCard";
@ -740,3 +740,306 @@ describe("IssueRecoveryActionCard repair workspace (quarantine_restore)", () =>
expect(node.querySelector("[data-testid='recovery-action-repair-trigger']")).not.toBeNull();
});
});
describe("IssueRecoveryActionCard owner-sticky retry lineage", () => {
const NOW = new Date("2026-08-18T12:00:00.000Z");
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(NOW);
});
afterEach(() => {
vi.useRealTimers();
});
function at(offsetMs: number) {
return new Date(NOW.getTime() + offsetMs).toISOString();
}
const bothAgents = new Map([
[ownerAgent.id, ownerAgent],
[returnAgent.id, returnAgent],
]);
/** Phase 1 — the original owner (CodexCoder) is retrying itself. */
function buildSourceLaneAction(overrides: Partial<IssueRecoveryAction> = {}) {
return buildAction({
kind: "deliberate_wait_without_target",
cause: "deliberate_wait_without_target",
ownerAgentId: returnAgent.id,
previousOwnerAgentId: returnAgent.id,
returnOwnerAgentId: returnAgent.id,
nextAction:
"The original owner must replace the parked summary with a terminal, live, blocked, monitored, or typed waiting disposition.",
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: returnAgent.id,
attempt: 2,
maxAttempts: 5,
baseBackoffMs: 60_000,
jitterMs: 3_000,
retryAt: at(3 * 60_000),
scheduledRunId: "00000000-0000-0000-0000-0000000000b1",
},
attemptCount: 2,
maxAttempts: 5,
timeoutAt: at(3 * 60_000),
...overrides,
});
}
/** Phase 2 — a manager (ClaudeCoder) repairs the path; CodexCoder keeps the task. */
function buildRecoveryLaneAction(overrides: Partial<IssueRecoveryAction> = {}) {
return buildSourceLaneAction({
ownerAgentId: ownerAgent.id,
nextAction:
"Repair the source issue disposition or request an explicit reassignment decision without taking source ownership.",
evidence: {
summary: "Run finished but no disposition was chosen.",
sourceAttemptCount: 5,
sourceMaxAttempts: 5,
},
wakePolicy: {
type: "bounded_recovery_owner",
ownerAgentId: ownerAgent.id,
attempt: 1,
maxAttempts: 3,
retryAt: at(60_000),
scheduledRunId: "00000000-0000-0000-0000-0000000000b2",
preservesSourceAssignee: true,
},
attemptCount: 1,
maxAttempts: 3,
timeoutAt: at(60_000),
...overrides,
});
}
it("stays quiet and names the retry lane while the original owner is being retried", () => {
const node = render(
<IssueRecoveryActionCard action={buildSourceLaneAction()} agentMap={bothAgents} />,
);
const section = node.querySelector("section[aria-label]");
expect(section?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(section?.getAttribute("data-recovery-kind")).toBe("deliberate_wait_without_target");
expect(section?.getAttribute("data-recovery-lane")).toBe("source_owner");
expect(node.textContent).toContain("RECOVERY IN PROGRESS");
expect(node.textContent).not.toContain("RECOVERY NEEDED");
expect(node.textContent).toContain("Wait Without A Target");
expect(node.textContent).toContain("The task stays with its owner, and no action is needed yet.");
expect(node.textContent).toContain("Paperclip is retrying the original owner");
});
it("shows the five-attempt budget and the next due time", () => {
const node = render(
<IssueRecoveryActionCard action={buildSourceLaneAction()} agentMap={bothAgents} />,
);
const progress = node.querySelector("[data-testid='recovery-retry-progress']");
expect(progress).not.toBeNull();
expect(progress?.getAttribute("data-recovery-lane")).toBe("source_owner");
expect(progress?.getAttribute("data-recovery-attempt")).toBe("2");
expect(progress?.getAttribute("data-recovery-max-attempts")).toBe("5");
expect(progress?.textContent).toContain("Attempt 2 of 5");
expect(node.querySelector("[data-testid='recovery-next-retry']")?.textContent).toBe(
"Next try in 3m",
);
});
it("keeps the source owner and the recovery owner as separate roles", () => {
const node = render(
<IssueRecoveryActionCard action={buildRecoveryLaneAction()} agentMap={bothAgents} />,
);
const sourceOwner = node.querySelector("[data-testid='recovery-source-owner']");
const recoveryOwner = node.querySelector("[data-testid='recovery-recovery-owner']");
// CodexCoder is the original owner and keeps the deliverable.
expect(sourceOwner?.textContent).toContain("CodexCoder");
expect(sourceOwner?.textContent).toContain("keeps this task");
// ClaudeCoder only repairs the path.
expect(recoveryOwner?.textContent).toContain("ClaudeCoder");
expect(recoveryOwner?.textContent).toContain("repairs the next step only");
expect(recoveryOwner?.textContent).not.toContain("CodexCoder");
expect(node.textContent).toContain(
"the task itself still belongs to its original owner",
);
});
it("labels the source lane as the owner retrying itself", () => {
const node = render(
<IssueRecoveryActionCard action={buildSourceLaneAction()} agentMap={bothAgents} />,
);
expect(
node.querySelector("[data-testid='recovery-recovery-owner']")?.textContent,
).toContain("Original owner — retrying itself");
});
it("reports the spent source attempts once the manager lane opens", () => {
const node = render(
<IssueRecoveryActionCard action={buildRecoveryLaneAction()} agentMap={bothAgents} />,
);
const section = node.querySelector("section[aria-label]");
expect(section?.getAttribute("data-recovery-lane")).toBe("recovery_owner");
expect(section?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(
node.querySelector("[data-testid='recovery-source-attempts']")?.textContent,
).toContain("The original owner used 5 of 5 automatic attempts.");
expect(
node.querySelector("[data-testid='recovery-retry-progress']")?.textContent,
).toContain("Attempt 1 of 3");
});
it("warns strongly only once the automatic path is exhausted", () => {
const node = render(
<IssueRecoveryActionCard
action={buildSourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: returnAgent.id,
attempt: 5,
maxAttempts: 5,
retryAt: at(-60_000),
},
attemptCount: 5,
})}
agentMap={bothAgents}
/>,
);
const section = node.querySelector("section[aria-label]");
expect(section?.getAttribute("data-recovery-state")).toBe("needed");
expect(node.textContent).toContain("RECOVERY NEEDED");
expect(node.textContent).toContain("has used every automatic repair attempt");
expect(node.querySelector("[data-testid='recovery-next-retry']")?.textContent).toBe(
"Automatic retries used up",
);
// The follow-up line must not keep promising a retry that will never run, and the
// generic timeout chip must not reintroduce a stale due time next to it.
expect(node.textContent).toContain("Automatic retries are finished — a decision is needed");
expect(node.textContent).not.toContain("Paperclip is retrying the original owner");
expect(node.textContent).not.toContain("Times out");
});
it("warns strongly when the stored retry came due and never ran", () => {
// PAP-17561: this exact shape rendered "Recovery in progress · Attempt 1 of 5 · Next try
// 5m ago" — a calm card over a lane nothing was working on.
const node = render(
<IssueRecoveryActionCard
action={buildSourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: returnAgent.id,
attempt: 1,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "00000000-0000-0000-0000-0000000000b2",
},
attemptCount: 1,
timeoutAt: at(-5 * 60_000),
})}
agentMap={bothAgents}
/>,
);
const section = node.querySelector("section[aria-label]");
expect(section?.getAttribute("data-recovery-state")).toBe("needed");
expect(node.textContent).toContain("RECOVERY NEEDED");
expect(node.textContent).not.toContain("RECOVERY IN PROGRESS");
const retry = node.querySelector("[data-testid='recovery-next-retry']");
expect(retry?.textContent).toBe("Retry missed 5m ago");
expect(retry?.getAttribute("data-recovery-retry-expired")).toBe("true");
// Nothing may still read as an upcoming attempt or as needing no action.
expect(node.textContent).not.toContain("Next try");
expect(node.textContent).not.toContain("no action is needed yet");
expect(node.textContent).toContain("came due and did not run");
expect(node.textContent).toContain("The scheduled retry did not run");
// The repair lane still must not move the deliverable off its original owner.
expect(node.querySelector("[data-testid='recovery-source-owner']")?.textContent).toContain(
"keeps this task",
);
});
it("stays quiet when the overdue attempt is a verified live run", () => {
const node = render(
<IssueRecoveryActionCard
action={buildSourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: returnAgent.id,
attempt: 2,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "00000000-0000-0000-0000-0000000000b2",
},
timeoutAt: at(-5 * 60_000),
})}
agentMap={bothAgents}
scheduledRetry={{
runId: "00000000-0000-0000-0000-0000000000b2",
status: "running",
agentId: returnAgent.id,
agentName: returnAgent.name,
retryOfRunId: null,
scheduledRetryAt: at(-5 * 60_000),
scheduledRetryAttempt: 2,
scheduledRetryReason: null,
}}
/>,
);
const section = node.querySelector("section[aria-label]");
expect(section?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(node.querySelector("[data-testid='recovery-next-retry']")?.textContent).toBe(
"Attempt running now",
);
expect(node.textContent).not.toContain("Retry missed");
});
it("keeps timing in the retry-progress row only while a lane is live", () => {
const node = render(
<IssueRecoveryActionCard action={buildSourceLaneAction()} agentMap={bothAgents} />,
);
expect(node.textContent).toContain("Paperclip is retrying the original owner");
expect(node.textContent).not.toContain("Times out");
});
it("shows the board escalation without implying the board owns the task", () => {
const node = render(
<IssueRecoveryActionCard
action={buildSourceLaneAction({
status: "escalated",
ownerType: "board",
ownerAgentId: null,
evidence: { sourceAttemptCount: 5, sourceMaxAttempts: 5 },
wakePolicy: {
type: "board_escalation",
reason: "recovery_owner_retry_exhausted",
attempt: 3,
maxAttempts: 3,
preservesSourceAssignee: true,
},
attemptCount: 3,
maxAttempts: 3,
timeoutAt: null,
})}
agentMap={bothAgents}
/>,
);
const section = node.querySelector("section[aria-label]");
expect(section?.getAttribute("data-recovery-state")).toBe("escalated");
expect(section?.getAttribute("data-recovery-lane")).toBe("board");
expect(node.textContent).toContain("Automatic recovery is exhausted");
const recoveryOwner = node.querySelector("[data-testid='recovery-recovery-owner']");
expect(recoveryOwner?.textContent).toContain("Board");
expect(recoveryOwner?.textContent).toContain("decides the next step only");
expect(
node.querySelector("[data-testid='recovery-source-owner']")?.textContent,
).toContain("CodexCoder");
});
it("leaves kinds without a bounded lineage on the original single owner row", () => {
const node = render(
<IssueRecoveryActionCard action={buildAction()} agentMap={bothAgents} />,
);
expect(node.querySelector("section[aria-label]")?.getAttribute("data-recovery-lane")).toBeNull();
expect(node.querySelector("[data-testid='recovery-retry-progress']")).toBeNull();
expect(node.querySelector("[data-testid='recovery-source-owner']")).toBeNull();
expect(node.textContent).toContain("→ Returns to:");
});
});

201
ui/src/components/IssueRecoveryActionCard.tsx
View File

@ -6,6 +6,7 @@ import type {
IssueRecoveryActionKind,
IssueRecoveryActionOutcome,
IssueRecoveryActionStatus,
IssueScheduledRetry,
} from "@paperclipai/shared";
import {
Eye,
@ -35,6 +36,12 @@ import {
deriveRecoveryDisplayState,
type RecoveryDisplayState,
} from "@/lib/recovery-display";
import {
formatRecoveryAttemptLabel,
formatRecoveryRetryOffset,
readRecoveryRetryLineage,
type RecoveryRetryLineage,
} from "@/lib/recovery-lineage";
export type RecoveryCardCardState = RecoveryDisplayState;
export const deriveRecoveryCardState = deriveRecoveryDisplayState;
@ -61,6 +68,12 @@ export interface RecoveryReissueRequest {
export interface IssueRecoveryActionCardProps {
action: IssueRecoveryAction;
agentMap?: ReadonlyMap<string, Agent>;
/**
* The source issue's scheduled retry. It is the only signal that can confirm the run the
* wake policy parked is genuinely in flight, which is what separates a retry the scheduler
* is running from one whose due time quietly passed.
*/
scheduledRetry?: IssueScheduledRetry | null;
/** Preferred state hint (e.g. observe_only when watchdog tone is requested). Falls back to derived state. */
forcedState?: RecoveryCardCardState;
/** Optional click handler for resolve menu actions. If omitted, the buttons are not rendered. */
@ -116,6 +129,7 @@ export interface IssueRecoveryActionCardProps {
const KIND_LABEL: Record<IssueRecoveryActionKind, string> = {
missing_disposition: "Missing Disposition",
deliberate_wait_without_target: "Wait Without A Target",
stranded_assigned_issue: "Stranded Task",
workspace_validation: "Workspace Validation",
configuration_validation: "Configuration Validation",
@ -126,6 +140,8 @@ const KIND_LABEL: Record<IssueRecoveryActionKind, string> = {
const KIND_HEADLINE: Record<IssueRecoveryActionKind, string> = {
missing_disposition:
"This task's run finished, but no next step was chosen. Choose what happens next — try the task again, mark it done, or send it for review.",
deliberate_wait_without_target:
"This task's last run stopped to wait, but there is no reviewer, blocker, monitor, or approval to wait for. Paperclip is repairing the next step; the task stays with its owner.",
stranded_assigned_issue:
"Paperclip retried this task's last run, but there is still no queued run, reviewer, blocker, or other next owner. To get it moving, choose what happens next — try the task again, mark it done, or send it for review.",
workspace_validation:
@ -138,6 +154,10 @@ const KIND_HEADLINE: Record<IssueRecoveryActionKind, string> = {
"Paperclip could not find a clear next step for this open task. Choose whether to continue work, send it for review, mark it done, or record what is blocking it.",
};
/** Shared shell for the retry-timing pill so every timing state reads as the same control. */
const RETRY_PILL_CLASS =
"rounded-md border border-border/50 bg-background/60 px-1.5 py-0.5 text-(length:--text-micro) text-muted-foreground";
const STATE_TONE: Record<RecoveryCardCardState, {
label: string;
containerClass: string;
@ -739,6 +759,10 @@ function readWakePolicySummary(action: IssueRecoveryAction): string | null {
const type = readEvidenceString(policy.type);
if (!type) return null;
if (type === "wake_owner") return "An agent will be asked to choose the next step";
if (type === "bounded_owner_disposition_repair") {
return "Paperclip is retrying the original owner";
}
if (type === "bounded_recovery_owner") return "A recovery owner is repairing the next step";
if (type === "board_escalation") return "Board will decide";
if (type === "manual") return "Manual follow-up needed";
if (type === "manual_repair_required") return "Repair needed before retry";
@ -859,6 +883,59 @@ function RunChip({
return <span className="inline-flex items-center gap-2">{inner}</span>;
}
function formatTimeAbsolute(value: string | Date | null | undefined): string | null {
if (!value) return null;
const date = value instanceof Date ? value : new Date(value);
if (Number.isNaN(date.getTime())) return null;
return date.toLocaleString(undefined, {
month: "short",
day: "numeric",
hour: "numeric",
minute: "2-digit",
});
}
/**
* Headline for an action carrying a bounded retry lineage. It names who keeps the task in
* every phase, because a manager owning the repair must never read as a manager owning
* the deliverable.
*/
function lineageHeadline(lineage: RecoveryRetryLineage): string {
// An attempt that came due and never ran leaves nobody working on this task, even though
// attempts remain on paper. Say so before any lane wording that ends in "no action needed".
if (lineage.retryExpired) {
return "This task's automatic retry came due and did not run, so nothing is moving it forward right now. Someone must retry it or record the next step. The task stays with its original owner.";
}
if (lineage.lane === "source_owner") {
return lineage.exhausted
? "This task's last run stopped to wait, but nothing was waiting for it. The original owner has used every automatic repair attempt, so the next step needs a decision. The task stays with its owner."
: "This task's last run stopped to wait, but nothing was waiting for it. Paperclip is retrying the original owner to record a real next step. The task stays with its owner, and no action is needed yet.";
}
if (lineage.lane === "recovery_owner") {
return "The original owner could not record a next step within its retry budget. A recovery owner is now repairing the path only — the task itself still belongs to its original owner.";
}
return "Automatic recovery is exhausted, so the board must choose the next step. The task itself still belongs to its original owner.";
}
/** Spent/remaining attempts as pips. The readable count lives beside it in text. */
function AttemptMeter({ lineage }: { lineage: RecoveryRetryLineage }) {
if (lineage.maxAttempts === null || lineage.maxAttempts > 12) return null;
const spent = Math.min(lineage.attempt, lineage.maxAttempts);
return (
<span className="inline-flex items-center gap-1" aria-hidden>
{Array.from({ length: lineage.maxAttempts }, (_, index) => (
<span
key={index}
className={cn(
"size-1.5 rounded-full",
index < spent ? "bg-current opacity-80" : "bg-current opacity-25",
)}
/>
))}
</span>
);
}
const RESOLVE_OPTIONS: Array<{
outcome: RecoveryResolveOutcome;
label: string;
@ -900,6 +977,7 @@ const RESOLVE_OPTIONS: Array<{
export function IssueRecoveryActionCard({
action,
agentMap,
scheduledRetry = null,
forcedState,
onResolve,
onReissueIsolated,
@ -914,24 +992,44 @@ export function IssueRecoveryActionCard({
variant = "full",
className,
}: IssueRecoveryActionCardProps) {
const cardState: RecoveryCardCardState = forcedState ?? deriveRecoveryCardState(action);
const liveness = useMemo(() => ({ scheduledRetry }), [scheduledRetry]);
const cardState: RecoveryCardCardState = forcedState ?? deriveRecoveryCardState(action, liveness);
const tone = STATE_TONE[cardState];
const ToneIcon = tone.Icon;
const divergence = useMemo(() => readWorkspaceDivergence(action), [action]);
const lineage = useMemo(() => readRecoveryRetryLineage(action, liveness), [action, liveness]);
const headline = useMemo(() => {
if (cardState === "resolved" && action.outcome) {
return `Recovery resolved as ${OUTCOME_LABEL[action.outcome] ?? action.outcome}.`;
}
if (lineage) return lineageHeadline(lineage);
return KIND_HEADLINE[action.kind] ?? KIND_HEADLINE.missing_disposition;
}, [action.kind, action.outcome, cardState]);
}, [action.kind, action.outcome, cardState, lineage]);
const wakeSummary = readWakePolicySummary(action);
// A lane with no path left must not keep advertising a retry that will never run — whether
// the budget ran out or the scheduled attempt simply never fired.
const wakeSummary = lineage?.retryExpired
? "The scheduled retry did not run — a retry or a decision is needed"
: lineage?.exhausted && lineage.lane !== "board"
? "Automatic retries are finished — a decision is needed"
: readWakePolicySummary(action);
const evidenceSummary = pickEvidenceSummary(action);
const sourceRunId = readEvidenceRunId(action, "sourceRunId") ?? readEvidenceRunId(action, "latestRunId");
const correctiveRunId = readEvidenceRunId(action, "correctiveRunId");
const showAttempt = action.attemptCount > 1 && action.maxAttempts !== null;
// The lineage rows below already carry the attempt budget, so the generic chip only
// covers actions without one.
const showAttempt = !lineage && action.attemptCount > 1 && action.maxAttempts !== null;
const sourceOwnerAgentId = action.returnOwnerAgentId ?? action.previousOwnerAgentId;
const recoveryOwnerIsSourceOwner =
action.ownerType === "agent" &&
action.ownerAgentId !== null &&
action.ownerAgentId === sourceOwnerAgentId;
const retryOffset = lineage ? formatRecoveryRetryOffset(lineage) : null;
const attemptLabel = lineage ? formatRecoveryAttemptLabel(lineage) : null;
const showTimeoutInline = (() => {
// The retry-progress row is the single place a lineage reports its timing.
if (lineage) return false;
if (!action.timeoutAt) return false;
try {
const date = action.timeoutAt instanceof Date ? action.timeoutAt : new Date(action.timeoutAt);
@ -1007,6 +1105,7 @@ export function IssueRecoveryActionCard({
aria-label={`Recovery action: ${ariaState}`}
data-recovery-state={cardState}
data-recovery-kind={action.kind}
data-recovery-lane={lineage?.lane}
className={cn(
"relative w-full overflow-hidden rounded-lg border text-sm shadow-(--shadow-extract-8)",
tone.containerClass,
@ -1044,6 +1143,99 @@ export function IssueRecoveryActionCard({
</header>
{variant === "compact" ? null : (
<dl className={cn("border-t bg-background/40 dark:bg-background/20", tone.divider)}>
{lineage ? (
<>
<MetadataRow label="Task owner">
<span
className="inline-flex flex-wrap items-center gap-1.5"
data-testid="recovery-source-owner"
>
<AgentLink
agentId={sourceOwnerAgentId}
agentMap={agentMap}
fallback="unassigned"
/>
<span className="text-muted-foreground">keeps this task</span>
</span>
</MetadataRow>
<MetadataRow label="Recovery owner">
<span
className="inline-flex flex-wrap items-center gap-1.5"
data-testid="recovery-recovery-owner"
>
{recoveryOwnerIsSourceOwner ? (
<span className="font-medium">Original owner — retrying itself</span>
) : action.ownerType === "agent" && action.ownerAgentId ? (
<>
<AgentLink agentId={action.ownerAgentId} agentMap={agentMap} />
<span className="text-muted-foreground">repairs the next step only</span>
</>
) : action.ownerType === "board" ? (
<>
<span className="font-medium">Board</span>
<span className="text-muted-foreground">decides the next step only</span>
</>
) : action.ownerType === "user" && action.ownerUserId ? (
<span className="font-medium">user {action.ownerUserId.slice(0, 6)}</span>
) : (
<span className="text-muted-foreground">unassigned — pick one to wake them</span>
)}
</span>
</MetadataRow>
<MetadataRow label="Retry progress">
<span
className="inline-flex flex-wrap items-center gap-x-2 gap-y-1"
data-testid="recovery-retry-progress"
data-recovery-lane={lineage.lane}
data-recovery-attempt={lineage.attempt}
data-recovery-max-attempts={lineage.maxAttempts ?? undefined}
>
<AttemptMeter lineage={lineage} />
<span>{attemptLabel ?? "Attempts not bounded"}</span>
{lineage.liveRunId ? (
<span
className={RETRY_PILL_CLASS}
title={formatTimeAbsolute(lineage.nextRetryAt) ?? undefined}
data-testid="recovery-next-retry"
>
Attempt running now
</span>
) : lineage.retryExpired ? (
// The due time is stated plainly as missed. Rendering it as "Next try 5m
// ago" is what made an abandoned lane read as healthy recovery.
<span
className={cn(RETRY_PILL_CLASS, "border-destructive/50 bg-destructive/10 text-destructive")}
title={formatTimeAbsolute(lineage.nextRetryAt) ?? undefined}
data-testid="recovery-next-retry"
data-recovery-retry-expired="true"
>
{retryOffset ? `Retry missed ${retryOffset}` : "Retry missed"}
</span>
) : retryOffset ? (
<span
className={RETRY_PILL_CLASS}
title={formatTimeAbsolute(lineage.nextRetryAt) ?? undefined}
data-testid="recovery-next-retry"
>
{retryOffset === "now" ? "Next try now" : `Next try ${retryOffset}`}
</span>
) : lineage.exhausted ? (
<span className={RETRY_PILL_CLASS} data-testid="recovery-next-retry">
Automatic retries used up
</span>
) : null}
</span>
</MetadataRow>
{lineage.lane !== "source_owner" && lineage.sourceMaxAttempts !== null ? (
<MetadataRow label="Owner retries">
<span data-testid="recovery-source-attempts">
The original owner used {lineage.sourceAttempt ?? lineage.sourceMaxAttempts} of{" "}
{lineage.sourceMaxAttempts} automatic attempts.
</span>
</MetadataRow>
) : null}
</>
) : (
<MetadataRow label="Owner">
<span className="inline-flex flex-wrap items-center gap-1.5">
{action.ownerType === "agent" && action.ownerAgentId ? (
@ -1068,6 +1260,7 @@ export function IssueRecoveryActionCard({
) : null}
</span>
</MetadataRow>
)}
<MetadataRow label="Source run">
<RunChip runId={sourceRunId} agentId={action.previousOwnerAgentId} />
</MetadataRow>

108
ui/src/components/IssueRow.test.tsx
View File

@ -529,4 +529,112 @@ describe("IssueRow", () => {
root.unmount();
});
});
describe("recovery chip liveness", () => {
const NOW = new Date("2026-08-18T12:00:00.000Z");
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(NOW);
});
afterEach(() => {
vi.useRealTimers();
});
function at(offsetMs: number) {
return new Date(NOW.getTime() + offsetMs).toISOString();
}
function recoveryIssue(retryAt: string, scheduledRetry: Issue["scheduledRetry"] = null): Issue {
return createIssue({
status: "in_progress",
scheduledRetry,
activeRecoveryAction: {
id: "action-1",
companyId: "company-1",
sourceIssueId: "issue-1",
recoveryIssueId: null,
kind: "deliberate_wait_without_target",
status: "active",
ownerType: "agent",
ownerAgentId: "agent-owner",
ownerUserId: null,
previousOwnerAgentId: "agent-owner",
returnOwnerAgentId: "agent-owner",
cause: "deliberate_wait_without_target",
fingerprint: "fp",
evidence: {},
nextAction: "Record a real next step.",
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: "agent-owner",
attempt: 1,
maxAttempts: 5,
retryAt,
scheduledRunId: "run-2",
},
monitorPolicy: null,
attemptCount: 1,
maxAttempts: 5,
timeoutAt: retryAt,
lastAttemptAt: retryAt,
outcome: null,
resolutionNote: null,
resolvedAt: null,
createdAt: at(-10 * 60_000),
updatedAt: at(-10 * 60_000),
},
});
}
function renderChip(issue: Issue): HTMLElement | null {
const root = createRoot(container);
act(() => {
root.render(<IssueRow issue={issue} />);
});
const chip = container.querySelector<HTMLElement>(
"[data-testid='issue-row-recovery-indicator']",
);
const snapshot = chip?.cloneNode(true) as HTMLElement | null;
act(() => {
root.unmount();
});
return snapshot;
}
it("stays calm while the next attempt is still ahead", () => {
const chip = renderChip(recoveryIssue(at(3 * 60_000)));
expect(chip?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(chip?.getAttribute("aria-label")).toContain("next try in 3m");
});
it("warns once the stored attempt came due and never ran", () => {
// The inbox chip must reach the same verdict as the source card, so a parent scanning
// the inbox is not told recovery is running when nothing is.
const chip = renderChip(recoveryIssue(at(-5 * 60_000)));
expect(chip?.getAttribute("data-recovery-state")).toBe("needed");
const label = chip?.getAttribute("aria-label") ?? "";
expect(label).toContain("Recovery needed");
expect(label).toContain("retry missed 5m ago");
expect(label).not.toContain("next try");
});
it("stays calm when the overdue attempt is a verified live run", () => {
const chip = renderChip(
recoveryIssue(at(-5 * 60_000), {
runId: "run-2",
status: "running",
agentId: "agent-owner",
agentName: "CodexCoder",
retryOfRunId: null,
scheduledRetryAt: at(-5 * 60_000),
scheduledRetryAttempt: 1,
scheduledRetryReason: null,
}),
);
expect(chip?.getAttribute("data-recovery-state")).toBe("in_progress");
expect(chip?.getAttribute("aria-label")).toContain("attempt running now");
});
});
});

30
ui/src/components/IssueRow.tsx
View File

@ -13,6 +13,11 @@ import {
RECOVERY_CHIP_DEFAULT_TONE,
recoveryChipLabel,
} from "../lib/recovery-display";
import {
formatRecoveryLineageSummary,
readRecoveryRetryLineage,
type RecoveryLivenessContext,
} from "../lib/recovery-lineage";
import { StatusIcon } from "./StatusIcon";
import { productivityReviewTriggerLabel } from "./ProductivityReviewBadge";
import { hasAssignedBacklogBlocker } from "../lib/issue-blockers";
@ -181,7 +186,11 @@ export function IssueRow({
</span>
) : null;
const recoveryAction = issue.activeRecoveryAction ?? null;
const recoveryIndicator = recoveryAction ? renderRecoveryChip(recoveryAction, selected) : null;
// The row already carries the issue's own scheduled retry, so the chip can tell a retry the
// scheduler is actually running from one whose due time simply passed.
const recoveryIndicator = recoveryAction
? renderRecoveryChip(recoveryAction, selected, { scheduledRetry: issue.scheduledRetry ?? null })
: null;
const parkedBlockerIndicator = hasAssignedBacklogBlocker(issue.blockedBy) ? (
<Badge variant="outline"
data-testid="issue-row-parked-blocker"
@ -347,25 +356,34 @@ export function IssueRow({
);
}
function renderRecoveryChip(action: IssueRecoveryAction, selected: boolean): ReactNode {
const state = deriveActiveRecoveryDisplayState(action);
function renderRecoveryChip(
action: IssueRecoveryAction,
selected: boolean,
liveness: RecoveryLivenessContext,
): ReactNode {
const state = deriveActiveRecoveryDisplayState(action, liveness);
if (!state) return null;
const tone = RECOVERY_CHIP_DEFAULT_TONE[state];
const Icon = tone.icon;
const label = recoveryChipLabel(state, action.kind);
const lineage = readRecoveryRetryLineage(action, liveness);
const label = recoveryChipLabel(state, action.kind, lineage);
const detail = lineage ? formatRecoveryLineageSummary(lineage) : null;
return (
<Badge variant="outline"
data-testid="issue-row-recovery-indicator"
data-recovery-state={state}
data-recovery-kind={action.kind}
data-recovery-lane={lineage?.lane}
role="status"
aria-label={label}
aria-label={detail ? `${label} — ${detail}` : label}
className={cn(
"ml-1.5 gap-0.5 text-(length:--text-nano)",
tone.className,
selected ? "!border-muted-foreground !text-muted-foreground" : null,
)}
title={`${label} — open the source task to act.`}
title={detail
? `${label} — ${detail}. Open the source task to act.`
: `${label} — open the source task to act.`}
>
<Icon className="h-2.5 w-2.5" aria-hidden />
{label}

154
ui/src/lib/recovery-display.test.ts
View File

@ -4,6 +4,7 @@ import {
deriveRecoveryDisplayState,
recoveryChipLabel,
} from "./recovery-display";
import { readRecoveryRetryLineage } from "./recovery-lineage";
describe("recoveryChipLabel", () => {
it("returns the workspace-specific label when kind is workspace_validation and state is needed", () => {
@ -18,6 +19,29 @@ describe("recoveryChipLabel", () => {
expect(recoveryChipLabel("needed", "issue_graph_liveness")).toBe("Recovery needed");
});
it("adds the attempt budget to an in-progress chip when a lineage is supplied", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 2,
maxAttempts: 5,
retryAt: "2099-01-01T00:00:00.000Z",
},
});
expect(recoveryChipLabel("in_progress", "deliberate_wait_without_target", lineage)).toBe(
"Recovery in progress · 2/5",
);
});
it("keeps the plain label when no attempt has been spent yet", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: { type: "bounded_owner_disposition_repair", attempt: 0, maxAttempts: 5 },
});
expect(recoveryChipLabel("in_progress", "deliberate_wait_without_target", lineage)).toBe(
"Recovery in progress",
);
});
it("does not override the chip label for non-needed states", () => {
expect(recoveryChipLabel("in_progress", "workspace_validation")).toBe(
"Recovery in progress",
@ -46,4 +70,134 @@ describe("deriveRecoveryDisplayState", () => {
deriveActiveRecoveryDisplayState({ ...base, kind: "workspace_validation" }),
).toBe("needed");
});
const waitBase = { ...base, kind: "deliberate_wait_without_target" as const };
it("stays quiet while a bounded owner retry is stored", () => {
expect(
deriveRecoveryDisplayState({
...waitBase,
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 2,
maxAttempts: 5,
retryAt: "2099-01-01T00:00:00.000Z",
},
}),
).toBe("in_progress");
});
it("stays quiet while a bounded manager retry is stored", () => {
expect(
deriveRecoveryDisplayState({
...waitBase,
wakePolicy: {
type: "bounded_recovery_owner",
attempt: 1,
maxAttempts: 3,
retryAt: "2099-01-01T00:00:00.000Z",
preservesSourceAssignee: true,
},
}),
).toBe("in_progress");
});
it("warns when the lane is exhausted", () => {
expect(
deriveRecoveryDisplayState({
...waitBase,
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 5,
maxAttempts: 5,
retryAt: "2099-01-01T00:00:00.000Z",
},
}),
).toBe("needed");
});
it("warns when the stored retry time has already passed", () => {
// The PAP-17561 false healthy state: attempts remain, so the lane is not exhausted, but
// the attempt it promised came due and never ran. Nothing is moving this task.
expect(
deriveRecoveryDisplayState({
...waitBase,
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: "2020-01-01T00:00:00.000Z",
scheduledRunId: "run-2",
},
}),
).toBe("needed");
});
it("stays quiet when a verified live run is executing the overdue attempt", () => {
expect(
deriveRecoveryDisplayState(
{
...waitBase,
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: "2020-01-01T00:00:00.000Z",
scheduledRunId: "run-2",
},
},
{ scheduledRetry: { runId: "run-2", status: "running" } },
),
).toBe("in_progress");
});
it("still warns when the live run belongs to a different lane", () => {
expect(
deriveRecoveryDisplayState(
{
...waitBase,
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: "2020-01-01T00:00:00.000Z",
scheduledRunId: "run-2",
},
},
{ scheduledRetry: { runId: "run-other", status: "running" } },
),
).toBe("needed");
});
it("warns when no next attempt is stored at all", () => {
expect(
deriveRecoveryDisplayState({
...waitBase,
wakePolicy: { type: "bounded_owner_disposition_repair", attempt: 1, maxAttempts: 5 },
}),
).toBe("needed");
});
it("keeps an escalated board action red even with a lineage", () => {
expect(
deriveRecoveryDisplayState({
...waitBase,
status: "escalated",
wakePolicy: {
type: "board_escalation",
reason: "recovery_owner_retry_exhausted",
preservesSourceAssignee: true,
},
}),
).toBe("escalated");
});
it("does not change kinds that carry no bounded lineage", () => {
expect(
deriveRecoveryDisplayState({ ...base, wakePolicy: { type: "wake_owner" } }),
).toBe("needed");
expect(
deriveRecoveryDisplayState({ ...base, outcome: "delegated", wakePolicy: null }),
).toBe("in_progress");
});
});

45
ui/src/lib/recovery-display.ts
View File

@ -1,5 +1,10 @@
import type { IssueRecoveryAction, IssueRecoveryActionKind } from "@paperclipai/shared";
import { Eye, OctagonAlert, RefreshCw, TriangleAlert } from "lucide-react";
import {
readRecoveryRetryLineage,
type RecoveryLivenessContext,
type RecoveryRetryLineage,
} from "./recovery-lineage";
export type RecoveryDisplayState =
| "needed"
@ -38,30 +43,64 @@ export const RECOVERY_CHIP_DEFAULT_TONE: Record<
},
};
/**
* Every surface derives its recovery tone from this one function, so a source issue and
* the parent views that list it as a blocker never disagree about whether recovery is
* quietly running or actually needs a human.
*/
export type RecoveryDisplayInput = Pick<IssueRecoveryAction, "status" | "kind" | "outcome"> &
Partial<
Pick<IssueRecoveryAction, "wakePolicy" | "evidence" | "attemptCount" | "maxAttempts" | "timeoutAt">
>;
export function deriveRecoveryDisplayState(
action: Pick<IssueRecoveryAction, "status" | "kind" | "outcome">,
action: RecoveryDisplayInput,
context?: RecoveryLivenessContext,
): RecoveryDisplayState {
if (action.status === "resolved") return "resolved";
if (action.status === "escalated") return "escalated";
if (action.status === "cancelled") return "resolved";
if (action.kind === "active_run_watchdog") return "observe_only";
// A bounded retry lineage still holding a durable path is work the server will do on its
// own. Shouting "recovery needed" over it would ask a human to fix something nobody has to
// fix yet, so the calm tone is reserved for a lane with an attempt genuinely still coming.
// Once that attempt comes due unanswered, or the budget runs out, the warning is the honest
// state — nothing is going to move this task without someone stepping in.
const lineage = readRecoveryRetryLineage({
wakePolicy: action.wakePolicy ?? null,
evidence: action.evidence,
attemptCount: action.attemptCount,
maxAttempts: action.maxAttempts,
timeoutAt: action.timeoutAt,
}, context);
if (lineage && lineage.lane !== "board" && lineage.hasDurablePath) return "in_progress";
if (action.outcome === "delegated") return "in_progress";
return "needed";
}
export function deriveActiveRecoveryDisplayState(
action: Pick<IssueRecoveryAction, "status" | "kind" | "outcome">,
action: RecoveryDisplayInput,
context?: RecoveryLivenessContext,
): ActiveRecoveryDisplayState | null {
const state = deriveRecoveryDisplayState(action);
const state = deriveRecoveryDisplayState(action, context);
return state === "resolved" ? null : state;
}
export function recoveryChipLabel(
state: ActiveRecoveryDisplayState,
kind: IssueRecoveryActionKind,
lineage?: RecoveryRetryLineage | null,
): string {
if (kind === "workspace_validation" && state === "needed") {
return "Workspace recovery needed";
}
if (
state === "in_progress" &&
lineage &&
lineage.maxAttempts !== null &&
lineage.attempt > 0
) {
return `Recovery in progress · ${Math.min(lineage.attempt, lineage.maxAttempts)}/${lineage.maxAttempts}`;
}
return RECOVERY_CHIP_DEFAULT_TONE[state].label;
}

415
ui/src/lib/recovery-lineage.test.ts Normal file
View File

@ -0,0 +1,415 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
formatRecoveryAttemptLabel,
formatRecoveryLineageSummary,
formatRecoveryRetryOffset,
readRecoveryRetryLineage,
type RecoveryLineageInput,
} from "./recovery-lineage";
const NOW = new Date("2026-08-18T12:00:00.000Z");
function at(offsetMs: number) {
return new Date(NOW.getTime() + offsetMs).toISOString();
}
function sourceLaneAction(overrides: Partial<RecoveryLineageInput> = {}): RecoveryLineageInput {
return {
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: "agent-owner",
attempt: 2,
maxAttempts: 5,
baseBackoffMs: 60_000,
jitterMs: 3_000,
retryAt: at(3 * 60_000),
scheduledRunId: "run-2",
},
evidence: {},
attemptCount: 2,
maxAttempts: 5,
timeoutAt: at(3 * 60_000),
...overrides,
};
}
function recoveryLaneAction(overrides: Partial<RecoveryLineageInput> = {}): RecoveryLineageInput {
return {
wakePolicy: {
type: "bounded_recovery_owner",
ownerAgentId: "agent-manager",
attempt: 1,
maxAttempts: 3,
retryAt: at(60_000),
scheduledRunId: "run-9",
preservesSourceAssignee: true,
},
evidence: { sourceAttemptCount: 5, sourceMaxAttempts: 5 },
attemptCount: 1,
maxAttempts: 3,
timeoutAt: at(60_000),
...overrides,
};
}
// Liveness is a question about now, so every case pins the clock. Fixtures date their retry
// times from NOW; left on the wall clock, a "future" retry silently becomes a past one and the
// suite would assert the opposite of what it claims.
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(NOW);
});
afterEach(() => {
vi.useRealTimers();
});
describe("readRecoveryRetryLineage", () => {
it("returns null when the action carries no bounded lineage", () => {
expect(readRecoveryRetryLineage({ wakePolicy: null })).toBeNull();
expect(readRecoveryRetryLineage({ wakePolicy: { type: "wake_owner" } })).toBeNull();
expect(readRecoveryRetryLineage({ wakePolicy: { type: "monitor" } })).toBeNull();
});
it("reads the source-owner lane with its stored attempt and due time", () => {
const lineage = readRecoveryRetryLineage(sourceLaneAction());
expect(lineage).not.toBeNull();
expect(lineage!.lane).toBe("source_owner");
expect(lineage!.attempt).toBe(2);
expect(lineage!.maxAttempts).toBe(5);
expect(lineage!.attemptsRemaining).toBe(3);
expect(lineage!.exhausted).toBe(false);
expect(lineage!.nextRetryAt).toBe(at(3 * 60_000));
expect(lineage!.retryAgentId).toBe("agent-owner");
expect(lineage!.hasDurablePath).toBe(true);
// The source lane is the original owner retrying itself, so it reports itself as
// preserving the deliverable even without an explicit policy flag.
expect(lineage!.preservesSourceAssignee).toBe(true);
expect(lineage!.sourceAttempt).toBe(2);
expect(lineage!.sourceMaxAttempts).toBe(5);
});
it("marks a source lane with no attempts left as exhausted and drops its stale due time", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 5,
maxAttempts: 5,
retryAt: at(-60_000),
},
}),
);
expect(lineage!.exhausted).toBe(true);
expect(lineage!.attemptsRemaining).toBe(0);
expect(lineage!.nextRetryAt).toBeNull();
expect(lineage!.hasDurablePath).toBe(false);
});
it("reads the manager lane and keeps the source lane's spent attempts from evidence", () => {
const lineage = readRecoveryRetryLineage(recoveryLaneAction());
expect(lineage!.lane).toBe("recovery_owner");
expect(lineage!.attempt).toBe(1);
expect(lineage!.maxAttempts).toBe(3);
expect(lineage!.retryAgentId).toBe("agent-manager");
expect(lineage!.preservesSourceAssignee).toBe(true);
expect(lineage!.sourceAttempt).toBe(5);
expect(lineage!.sourceMaxAttempts).toBe(5);
expect(lineage!.hasDurablePath).toBe(true);
});
it("treats a board escalation from this contract as an exhausted lane", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: {
type: "board_escalation",
reason: "recovery_owner_retry_exhausted",
attempt: 3,
maxAttempts: 3,
preservesSourceAssignee: true,
},
evidence: { sourceAttemptCount: 5, sourceMaxAttempts: 5 },
});
expect(lineage!.lane).toBe("board");
expect(lineage!.exhausted).toBe(true);
expect(lineage!.nextRetryAt).toBeNull();
expect(lineage!.hasDurablePath).toBe(false);
expect(lineage!.sourceMaxAttempts).toBe(5);
});
it("ignores an unrelated board escalation that is not part of the owner-sticky contract", () => {
expect(
readRecoveryRetryLineage({
wakePolicy: { type: "board_escalation", reason: "provider_quota" },
evidence: {},
}),
).toBeNull();
});
it("falls back to the action's own attempt fields when the policy omits them", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: { type: "bounded_owner_disposition_repair" },
attemptCount: 3,
maxAttempts: 5,
timeoutAt: at(120_000),
});
expect(lineage!.attempt).toBe(3);
expect(lineage!.maxAttempts).toBe(5);
expect(lineage!.nextRetryAt).toBe(at(120_000));
});
it("reports no durable path when nothing is scheduled yet", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: "agent-owner",
attempt: 1,
maxAttempts: 5,
},
});
expect(lineage!.exhausted).toBe(false);
expect(lineage!.hasDurablePath).toBe(false);
});
it("ignores an unparseable stored due time", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: "not-a-date",
},
});
expect(lineage!.nextRetryAt).toBeNull();
});
});
describe("durable-path liveness", () => {
it("keeps a future retry durable and not expired", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(5 * 60_000),
scheduledRunId: "run-2",
},
}),
)!;
expect(lineage.retryExpired).toBe(false);
expect(lineage.hasDurablePath).toBe(true);
expect(lineage.liveRunId).toBeNull();
});
it("treats a retry time in the past as expired with no durable path", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "run-2",
},
timeoutAt: at(-5 * 60_000),
}),
)!;
// Attempts remain on paper, but the one that was promised never ran.
expect(lineage.exhausted).toBe(false);
expect(lineage.attemptsRemaining).toBe(4);
expect(lineage.retryExpired).toBe(true);
expect(lineage.hasDurablePath).toBe(false);
});
it("does not let a bare scheduled run id stand in for liveness", () => {
// The id only records that a run was created once. Without a liveness signal it proves
// nothing, which is why the past-due lane above stays expired despite carrying one.
const lineage = readRecoveryRetryLineage({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
scheduledRunId: "run-2",
},
})!;
expect(lineage.scheduledRunId).toBe("run-2");
expect(lineage.liveRunId).toBeNull();
expect(lineage.hasDurablePath).toBe(false);
});
it("holds a past retry durable while a verified live run is executing it", () => {
const action = sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "run-2",
},
timeoutAt: at(-5 * 60_000),
});
for (const status of ["queued", "running"] as const) {
const lineage = readRecoveryRetryLineage(action, {
scheduledRetry: { runId: "run-2", status },
})!;
expect(lineage.liveRunId).toBe("run-2");
// The due time is behind us precisely because the run picked it up.
expect(lineage.retryExpired).toBe(false);
expect(lineage.hasDurablePath).toBe(true);
}
});
it("rejects a scheduled run that is no longer live", () => {
const action = sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "run-2",
},
timeoutAt: at(-5 * 60_000),
});
for (const status of ["cancelled", "scheduled_retry"] as const) {
const lineage = readRecoveryRetryLineage(action, {
scheduledRetry: { runId: "run-2", status },
})!;
expect(lineage.liveRunId).toBeNull();
expect(lineage.retryExpired).toBe(true);
expect(lineage.hasDurablePath).toBe(false);
}
});
it("ignores a live run that is not the one this lane parked", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "run-2",
},
timeoutAt: at(-5 * 60_000),
}),
{ scheduledRetry: { runId: "run-somebody-else", status: "running" } },
)!;
expect(lineage.liveRunId).toBeNull();
expect(lineage.hasDurablePath).toBe(false);
});
it("keeps an exhausted lane dead even while a live run is reported", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 5,
maxAttempts: 5,
retryAt: at(-60_000),
scheduledRunId: "run-2",
},
}),
{ scheduledRetry: { runId: "run-2", status: "running" } },
)!;
expect(lineage.exhausted).toBe(true);
expect(lineage.hasDurablePath).toBe(false);
// An exhausted lane reports no upcoming attempt, so it is not also "missed".
expect(lineage.retryExpired).toBe(false);
});
it("does not call a barely-due retry expired while surfaces still read it as now", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(-5_000),
},
}),
)!;
expect(formatRecoveryRetryOffset(lineage)).toBe("now");
expect(lineage.retryExpired).toBe(false);
expect(lineage.hasDurablePath).toBe(true);
});
it("honours an injected now", () => {
const action = sourceLaneAction();
expect(readRecoveryRetryLineage(action, { now: NOW.getTime() })!.retryExpired).toBe(false);
expect(
readRecoveryRetryLineage(action, { now: NOW.getTime() + 60 * 60_000 })!.retryExpired,
).toBe(true);
});
});
describe("recovery lineage formatting", () => {
it("formats the attempt budget", () => {
expect(formatRecoveryAttemptLabel(readRecoveryRetryLineage(sourceLaneAction())!)).toBe(
"Attempt 2 of 5",
);
});
it("clamps a spent attempt count to the budget", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: { type: "bounded_owner_disposition_repair", attempt: 6, maxAttempts: 5 },
}),
)!;
expect(formatRecoveryAttemptLabel(lineage)).toBe("Attempt 5 of 5");
});
it("formats the next due time relative to now", () => {
const lineage = readRecoveryRetryLineage(sourceLaneAction())!;
expect(formatRecoveryRetryOffset(lineage)).toBe("in 3m");
expect(formatRecoveryLineageSummary(lineage)).toBe("Attempt 2 of 5 · next try in 3m");
});
it("reports a missed retry instead of an upcoming one (PAP-17561 regression)", () => {
// The reported false healthy state was exactly "Attempt 1 of 5 · next try 5m ago".
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 1,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "run-2",
},
timeoutAt: at(-5 * 60_000),
}),
)!;
const summary = formatRecoveryLineageSummary(lineage);
expect(summary).toBe("Attempt 1 of 5 · retry missed 5m ago");
expect(summary).not.toContain("next try");
});
it("says the attempt is running when a live run is verified", () => {
const lineage = readRecoveryRetryLineage(
sourceLaneAction({
wakePolicy: {
type: "bounded_owner_disposition_repair",
attempt: 2,
maxAttempts: 5,
retryAt: at(-5 * 60_000),
scheduledRunId: "run-2",
},
}),
{ scheduledRetry: { runId: "run-2", status: "running" } },
)!;
expect(formatRecoveryLineageSummary(lineage)).toBe("Attempt 2 of 5 · attempt running now");
});
it("says retries are used up when a lane is exhausted", () => {
const lineage = readRecoveryRetryLineage({
wakePolicy: {
type: "board_escalation",
reason: "recovery_owner_retry_exhausted",
attempt: 3,
maxAttempts: 3,
preservesSourceAssignee: true,
},
})!;
expect(formatRecoveryRetryOffset(lineage)).toBeNull();
expect(formatRecoveryLineageSummary(lineage)).toBe("Attempt 3 of 3 · retries used up");
});
});

232
ui/src/lib/recovery-lineage.ts Normal file
View File

@ -0,0 +1,232 @@
import type { IssueRecoveryAction, IssueScheduledRetry } from "@paperclipai/shared";
import { formatMonitorOffset } from "./issue-monitor";
/**
* Which bounded retry budget the server is currently spending on a recovery action.
*
* The owner-sticky contract keeps one recovery-action row per source issue and rewrites
* its `wakePolicy` as the action moves between budgets: the original owner's repair
* attempts, then a manager's path-repair attempts, then the board. The policy type is
* therefore the canonical lane signal — every surface reads the same stored value rather
* than re-deriving liveness on its own.
*/
export type RecoveryRetryLane = "source_owner" | "recovery_owner" | "board";
export interface RecoveryRetryLineage {
lane: RecoveryRetryLane;
/** Attempts already spent in the current lane. 0 before the first attempt is scheduled. */
attempt: number;
maxAttempts: number | null;
attemptsRemaining: number | null;
/** The lane has no attempts left (always true once the board owns the action). */
exhausted: boolean;
/** When the stored next attempt is due, as an ISO string. Null once a lane is exhausted. */
nextRetryAt: string | null;
/**
* The stored attempt came due and nothing picked it up. The row still reports the time it
* was due — an expired attempt is a missed promise worth showing, not a fact to hide — but
* it no longer counts as a path anyone is waiting on.
*/
retryExpired: boolean;
/** The run the server parked for the next attempt, when it recorded one. */
scheduledRunId: string | null;
/**
* The parked run, once a caller has confirmed it is genuinely in flight. A bare
* `scheduledRunId` is only a record that a run was once created, so it is never enough on
* its own; this field is set only from a liveness signal the caller passed in.
*/
liveRunId: string | null;
/** The agent the stored attempt will wake. */
retryAgentId: string | null;
/** The server recorded that this lane does not move the source deliverable. */
preservesSourceAssignee: boolean;
/** Attempts the original owner spent before the manager lane opened. */
sourceAttempt: number | null;
sourceMaxAttempts: number | null;
/**
* Something will still move this lane forward without anyone intervening: either a stored
* attempt that is still ahead of us, or a parked run confirmed to be in flight. This — not
* the mere existence of an open recovery action, and not a due time that already passed —
* is what keeps a surface quiet.
*/
hasDurablePath: boolean;
}
const SOURCE_LANE_POLICY = "bounded_owner_disposition_repair";
const RECOVERY_LANE_POLICY = "bounded_recovery_owner";
const BOARD_LANE_POLICY = "board_escalation";
/** Scheduled-run states that mean the parked attempt is actually in flight right now. */
const LIVE_SCHEDULED_RUN_STATUSES = new Set(["queued", "running"]);
/**
* How far past its due time a stored attempt may sit before it counts as missed.
*
* `formatMonitorOffset` collapses anything inside a minute-rounded zero to "now", so reusing
* that same band keeps the label and the verdict from ever contradicting each other: while a
* surface still reads "next try now", the lane is still treated as durable. Past that, the
* scheduler had its chance and did not take it.
*/
const RETRY_EXPIRY_GRACE_MS = 30_000;
export type RecoveryLineageInput = Pick<IssueRecoveryAction, "wakePolicy"> &
Partial<Pick<IssueRecoveryAction, "evidence" | "attemptCount" | "maxAttempts" | "timeoutAt">>;
/**
* The source issue's scheduled-retry record, which is what lets a surface tell a run that is
* really executing from a run id the server wrote down once and never started.
*/
export type RecoveryScheduledRetryInput = Partial<Pick<IssueScheduledRetry, "runId" | "status">>;
/**
* Everything outside the stored action that decides whether a lane is still live. Callers
* that cannot observe the scheduled run leave `scheduledRetry` unset and get the conservative
* answer, so a surface can under-promise but never over-promise.
*/
export interface RecoveryLivenessContext {
/** Epoch ms to treat as now. Defaults to the wall clock. */
now?: number;
scheduledRetry?: RecoveryScheduledRetryInput | null;
}
function asRecord(value: unknown): Record<string, unknown> | null {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: null;
}
function asNonEmptyString(value: unknown): string | null {
if (typeof value !== "string") return null;
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : null;
}
function asCount(value: unknown): number | null {
if (typeof value !== "number" || !Number.isFinite(value)) return null;
const floored = Math.floor(value);
return floored >= 0 ? floored : null;
}
function asIsoDate(value: unknown): string | null {
if (!value) return null;
if (!(value instanceof Date) && typeof value !== "string") return null;
const date = value instanceof Date ? value : new Date(value);
return Number.isNaN(date.getTime()) ? null : date.toISOString();
}
/**
* Confirm the parked run is in flight. When the policy named a specific run, only that run
* counts — a different live run on the issue is someone else's work and says nothing about
* whether this retry will happen.
*/
function resolveLiveRunId(
scheduledRunId: string | null,
scheduledRetry: RecoveryScheduledRetryInput | null | undefined,
): string | null {
if (!scheduledRetry) return null;
const runId = asNonEmptyString(scheduledRetry.runId);
const status = asNonEmptyString(scheduledRetry.status);
if (!runId || !status || !LIVE_SCHEDULED_RUN_STATUSES.has(status)) return null;
if (scheduledRunId !== null && scheduledRunId !== runId) return null;
return runId;
}
/**
* Read the bounded retry lineage the server stored on a recovery action, or null when the
* action does not carry one (older kinds, or a board escalation that is not part of the
* owner-sticky contract).
*/
export function readRecoveryRetryLineage(
action: RecoveryLineageInput,
context?: RecoveryLivenessContext,
): RecoveryRetryLineage | null {
const policy = asRecord(action.wakePolicy);
if (!policy) return null;
const type = asNonEmptyString(policy.type);
const preservesSourceAssignee = policy.preservesSourceAssignee === true;
const evidence = asRecord(action.evidence) ?? {};
const evidenceSourceAttempt = asCount(evidence.sourceAttemptCount);
const evidenceSourceMaxAttempts = asCount(evidence.sourceMaxAttempts);
let lane: RecoveryRetryLane;
if (type === SOURCE_LANE_POLICY) lane = "source_owner";
else if (type === RECOVERY_LANE_POLICY) lane = "recovery_owner";
else if (
type === BOARD_LANE_POLICY &&
(preservesSourceAssignee || evidenceSourceMaxAttempts !== null)
) {
lane = "board";
} else return null;
const attempt = asCount(policy.attempt) ?? asCount(action.attemptCount) ?? 0;
const maxAttempts = asCount(policy.maxAttempts) ?? asCount(action.maxAttempts);
const scheduledRunId = asNonEmptyString(policy.scheduledRunId);
const storedRetryAt = asIsoDate(policy.retryAt) ?? asIsoDate(action.timeoutAt);
const exhausted = lane === "board" || (maxAttempts !== null && attempt >= maxAttempts);
const nextRetryAt = exhausted ? null : storedRetryAt;
const liveRunId = resolveLiveRunId(scheduledRunId, context?.scheduledRetry);
// A run that is already executing is the reason its due time is behind us, so a confirmed
// live run is never "missed" — it is the attempt happening.
const retryExpired =
!exhausted &&
liveRunId === null &&
nextRetryAt !== null &&
Date.parse(nextRetryAt) + RETRY_EXPIRY_GRACE_MS < (context?.now ?? Date.now());
return {
lane,
attempt,
maxAttempts,
attemptsRemaining: maxAttempts === null ? null : Math.max(0, maxAttempts - attempt),
exhausted,
nextRetryAt,
retryExpired,
scheduledRunId,
liveRunId,
retryAgentId: asNonEmptyString(policy.retryAgentId) ?? asNonEmptyString(policy.ownerAgentId),
preservesSourceAssignee: preservesSourceAssignee || lane === "source_owner",
sourceAttempt: lane === "source_owner" ? attempt : evidenceSourceAttempt,
sourceMaxAttempts: lane === "source_owner" ? maxAttempts : evidenceSourceMaxAttempts,
hasDurablePath:
!exhausted && (liveRunId !== null || (nextRetryAt !== null && !retryExpired)),
};
}
/** "Attempt 2 of 5", or null when the server did not record a bounded budget. */
export function formatRecoveryAttemptLabel(lineage: RecoveryRetryLineage): string | null {
if (lineage.maxAttempts === null) return null;
return `Attempt ${Math.min(lineage.attempt, lineage.maxAttempts)} of ${lineage.maxAttempts}`;
}
/** "in 3m" / "now" / "3m ago" for the stored next attempt, or null when none is stored. */
export function formatRecoveryRetryOffset(lineage: RecoveryRetryLineage): string | null {
if (!lineage.nextRetryAt) return null;
try {
return formatMonitorOffset(lineage.nextRetryAt);
} catch {
return null;
}
}
/**
* One compact sentence fragment shared by the card and the blocker chips so every surface
* reports the same attempt count and due time.
*/
export function formatRecoveryLineageSummary(lineage: RecoveryRetryLineage): string | null {
const parts: string[] = [];
const attempt = formatRecoveryAttemptLabel(lineage);
if (attempt) parts.push(attempt);
const offset = formatRecoveryRetryOffset(lineage);
if (lineage.liveRunId) {
parts.push("attempt running now");
} else if (lineage.retryExpired) {
// Never "next try 5m ago": a due time in the past is a missed attempt, and phrasing it as
// an upcoming one is exactly the false healthy state this helper exists to prevent.
parts.push(offset ? `retry missed ${offset}` : "retry missed");
} else if (offset) {
parts.push(offset === "now" ? "next try now" : `next try ${offset}`);
} else if (lineage.exhausted) {
parts.push("retries used up");
}
return parts.length > 0 ? parts.join(" · ") : null;
}

170
ui/storybook/stories/source-issue-recovery.stories.tsx
View File

@ -9,6 +9,12 @@ import { storybookAgentMap, storybookAgents, createIssue } from "../fixtures/pap
const claudeAgent = storybookAgents.find((agent) => agent.name.toLowerCase().startsWith("claude")) ?? storybookAgents[0]!;
const codexAgent = storybookAgents.find((agent) => agent.name.toLowerCase().startsWith("codex")) ?? storybookAgents[0]!;
// The recovery lane must be shown by an agent that is genuinely not the source owner,
// otherwise the "separate roles" story cannot demonstrate anything.
const managerAgent =
storybookAgents.find((agent) => agent.name.toLowerCase() === "cto") ??
storybookAgents.find((agent) => agent.id !== codexAgent.id) ??
codexAgent;
function StoryFrame({ title, description, children }: { title: string; description?: string; children: ReactNode }) {
return (
@ -190,6 +196,159 @@ function AllStatesPanel() {
);
}
/**
* Owner-sticky repair lineage (PAP-17484). One recovery-action row moves through three
* budgets — the original owner retrying itself, a manager repairing the path, then the
* board — and the source owner never changes.
*/
function inMinutes(minutes: number) {
return new Date(Date.now() + minutes * 60_000).toISOString();
}
function buildSourceLaneAction(overrides: Partial<IssueRecoveryAction> = {}) {
return buildAction({
kind: "deliberate_wait_without_target",
cause: "deliberate_wait_without_target",
ownerAgentId: codexAgent.id,
previousOwnerAgentId: codexAgent.id,
returnOwnerAgentId: codexAgent.id,
fingerprint: "disposition_repair:v1:9f2c",
evidence: {
summary: "The run parked on a review wait, but no reviewer, approval, or monitor exists.",
sourceRunId: "7accd7a4-c9ca-4db2-9233-3228a037cc09",
},
nextAction:
"The original owner must replace the parked summary with a terminal, live, blocked, monitored, or typed waiting disposition.",
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: codexAgent.id,
attempt: 2,
maxAttempts: 5,
baseBackoffMs: 60_000,
jitterMs: 4_200,
retryAt: inMinutes(3),
scheduledRunId: "2606404d-3859-4142-ba37-3228a037cc09",
},
attemptCount: 2,
maxAttempts: 5,
timeoutAt: inMinutes(3),
...overrides,
});
}
function buildRecoveryLaneAction(overrides: Partial<IssueRecoveryAction> = {}) {
return buildSourceLaneAction({
ownerAgentId: managerAgent.id,
nextAction:
"Repair the source issue disposition or request an explicit reassignment decision without taking source ownership.",
evidence: {
summary: "Five original-owner repair attempts finished without a durable source change.",
sourceAttemptCount: 5,
sourceMaxAttempts: 5,
terminalReason: "disposition_repair_attempts_exhausted",
},
wakePolicy: {
type: "bounded_recovery_owner",
ownerAgentId: managerAgent.id,
attempt: 1,
maxAttempts: 3,
retryAt: inMinutes(1),
scheduledRunId: "9c1c5f5a-2e0b-4d1a-9d6f-3228a037cc09",
preservesSourceAssignee: true,
},
attemptCount: 1,
maxAttempts: 3,
timeoutAt: inMinutes(1),
...overrides,
});
}
function RetryLineagePanel() {
return (
<div className="grid gap-5 lg:grid-cols-1">
<CardPanel
caption="Lane 1 · Original owner retrying itself — quiet, no action needed"
action={buildSourceLaneAction()}
/>
<CardPanel
caption="Lane 1 (exhausted) · Every automatic attempt is spent — strong warning"
action={buildSourceLaneAction({
attemptCount: 5,
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: codexAgent.id,
attempt: 5,
maxAttempts: 5,
retryAt: inMinutes(-2),
},
})}
canFalsePositive
/>
<CardPanel
caption="Lane 2 · Manager repairs the path only — the original owner keeps the task"
action={buildRecoveryLaneAction()}
/>
<CardPanel
caption="Lane 3 · Board escalation — still not a change of task ownership"
action={buildRecoveryLaneAction({
status: "escalated",
ownerType: "board",
ownerAgentId: null,
attemptCount: 3,
maxAttempts: 3,
timeoutAt: null,
wakePolicy: {
type: "board_escalation",
reason: "recovery_owner_retry_exhausted",
attempt: 3,
maxAttempts: 3,
preservesSourceAssignee: true,
},
})}
canFalsePositive
/>
<section className="space-y-2">
<div className="text-[11px] font-semibold uppercase tracking-[0.16em] text-muted-foreground">
Parent blocker chips · same actions, same liveness state
</div>
<IssueBlockedNotice
issueStatus="blocked"
blockers={[
buildBlocker({
id: "lineage-1",
identifier: "PAP-17417",
title: "Original owner is being retried",
activeRecoveryAction: buildSourceLaneAction(),
}),
buildBlocker({
id: "lineage-2",
identifier: "PAP-17418",
title: "Automatic repair exhausted",
status: "blocked",
activeRecoveryAction: buildSourceLaneAction({
attemptCount: 5,
wakePolicy: {
type: "bounded_owner_disposition_repair",
retryAgentId: codexAgent.id,
attempt: 5,
maxAttempts: 5,
retryAt: inMinutes(-2),
},
}),
}),
buildBlocker({
id: "lineage-3",
identifier: "PAP-17399",
title: "Manager is repairing the path",
activeRecoveryAction: buildRecoveryLaneAction(),
}),
]}
/>
</section>
</div>
);
}
function buildBlocker(
overrides: Partial<IssueRelationIssueSummary> = {},
): IssueRelationIssueSummary {
@ -435,6 +594,17 @@ export const RecoveryActionCardStates: Story = {
),
};
export const OwnerStickyRetryLineage: Story = {
render: () => (
<StoryFrame
title="Owner-sticky repair lineage"
description="A stored retry keeps the card quiet; the strong warning is reserved for an exhausted or missing path. The task owner and the recovery owner are always shown as separate roles, and parent blocker chips report the same liveness state as the source card."
>
<RetryLineagePanel />
</StoryFrame>
),
};
export const InboxRowChips: Story = {
render: () => (
<StoryFrame