diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index a60bb4bec7..805743e0c1 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -521,6 +521,8 @@ V1 non-terminal liveness rule: - recovery-action ownership is separate from source-task ownership: automatic repair and board escalation preserve both source assignee fields; reassignment requires an explicit board decision or a policy-defined serious failure - source-scoped recovery routing is cause-keyed: bounded continuity and disposition repair may retry only the original agent; provider-quota failures create/reuse a scheduled wait-recovery monitor; every other exhausted or unsafe path creates/reuses a board-owned recovery action with `routingPolicy: board_escalation_no_takeover_v1` and no substitute-agent wake - legacy active agent-owned recovery actions remain readable, resolvable, and API-compatible after upgrade, but reconciliation does not enqueue another takeover wake for them +- active-run output silence is an informational board UI signal at one hour (`suspicious`) and four hours (`critical`); it does not create or update issues or recovery actions, comment on or block source work, change assignments, or wake an agent +- board snooze and continue decisions suppress the run signal until their stored re-arm time; a false-positive decision suppresses it permanently for that run; open legacy evaluation issues remain readable and manually resolvable without automatic refresh Detailed ownership, execution, blocker, active-run watchdog, crash-recovery, and non-terminal liveness semantics are documented in `doc/execution-semantics.md`. diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 9964cdea4c..9f707d6280 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -580,7 +580,7 @@ On startup and on the periodic recovery loop, Paperclip now does five things in 1. reap orphaned `running` runs 2. resume persisted `queued` runs 3. reconcile stranded assigned work -4. scan silent active runs, revalidate their source issues, and either fold source-resolved watchdogs or create/update explicit watchdog recovery actions +4. scan silent active runs only for source-aware terminal folding and legacy cleanup; API reads classify ordinary output silence for the board UI 5. reconcile productivity reviews The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition. @@ -717,27 +717,26 @@ An active run can still be unhealthy even when its process is `running`. Papercl The recovery service owns this contract: - classify active-run output silence as `ok`, `suspicious`, `critical`, `snoozed`, or `not_applicable` -- collect bounded evidence from run logs, recent run events, child issues, and blockers -- preserve redaction and truncation before evidence is written to issue descriptions -- create at most one open watchdog recovery action per run; issue-backed implementations use `stale_active_run_evaluation` issues -- honor active snooze decisions before creating more review work +- honor active snooze and continue decisions on the run +- permanently suppress the signal for a run after a `dismissed_false_positive` decision - build the `outputSilence` summary shown by live-run and active-run API responses +- retain links to open legacy `stale_active_run_evaluation` issues without refreshing or changing them -Suspicious silence creates a medium-priority watchdog recovery action for the selected recovery owner. Critical silence raises that recovery action to high priority and, when issue-backed evaluation is needed for correctness, blocks the source issue on the explicit evaluation task without cancelling the active process. +Suspicious and critical silence are informational board UI signals. They do not create an issue or recovery action. They do not comment on or block the source issue. They do not change an assignment, wake an agent, cancel the active process, or change the run. The board uses the existing run controls when it decides that intervention is necessary. -Watchdog decisions are explicit operator/recovery-owner decisions: +Watchdog decisions are explicit board decisions stored against the run: -- `snooze` records an operator-chosen future quiet-until time and suppresses scan-created review work during that window +- `snooze` records an operator-chosen future quiet-until time and hides the signal during that window - `continue` records that the current evidence is acceptable, does not cancel or mutate the active run, and sets a 30-minute default re-arm window before the watchdog evaluates the still-silent run again -- `dismissed_false_positive` records why the review was not actionable +- `dismissed_false_positive` records why the signal was not actionable and suppresses it permanently for that run -Operators should prefer `snooze` for known time-bounded quiet periods. `continue` is only a short acknowledgement of the current evidence; if the run remains silent after the re-arm window, the periodic watchdog scan can create or update review work again. +Operators should prefer `snooze` for known time-bounded quiet periods. `continue` is only a short acknowledgement of the current evidence. If the run remains silent after the re-arm window, the UI signal appears again. -The board can record watchdog decisions. The assigned owner of an issue-backed watchdog evaluation can also record them. Other agents cannot. +The signal reappears in the UI after a snooze or continue window expires. No review work is created when it reappears. The board can record decisions without an evaluation issue. For compatibility, the assigned owner of an open legacy evaluation issue can also record a decision that is bound to that issue and run. Other agents cannot. ### Source-aware watchdog folding -Active-run watchdog work is source-aware. Before the watchdog creates, refreshes, escalates, or blocks on reviewer work, it must re-read the linked source issue and decide whether the watchdog signal is still about productive source work or only about stale run/process bookkeeping. +The active-run cleanup scan is source-aware. It re-reads the linked source issue and decides whether a still-running handle represents productive source work or stale run/process bookkeeping. It does not create reviewer work. Fold watchdog work when all of these are true: @@ -746,17 +745,17 @@ Fold watchdog work when all of these are true: - durable source activity from the same run proves the source issue reached that terminal disposition after the stale-run or output-silence evidence point - there is no independent evidence that the still-running or detached process is doing harmful work, still owns external cleanup that needs an operator decision, or needs a separate security/ownership review -Folding means resolving or cancelling the watchdog recovery action or issue-backed evaluation through the explicit recovery lifecycle. It must preserve the run id, source issue, detected silence or detached-process evidence, terminal source activity, decision reason, and best-effort process cleanup result. It must be idempotent for the `(companyId, runId, sourceIssueId)` signal and must not recursively recover the watchdog evaluation issue itself. +Folding means finalizing the stale run and resolving any legacy watchdog recovery action or issue-backed evaluation through the explicit recovery lifecycle. It must preserve the run id, source issue, detected silence or detached-process evidence, terminal source activity, decision reason, and best-effort process cleanup result. It must be idempotent for the `(companyId, runId, sourceIssueId)` signal and must not recursively recover a watchdog evaluation issue itself. -Do not fold watchdog work only because the run is quiet. The watchdog must still create or continue reviewer work when: +Do not fold a run only because it is quiet. Keep the informational signal visible when: - the source issue is still `todo` or `in_progress`, because productive work may still be happening or stuck - the source issue remains `in_progress` after a successful run with no valid disposition, because the successful-run handoff path owns that bounded correction - the run terminated or disappeared while the source issue remains `in_progress` without a live path, because stranded assigned recovery owns that continuity repair - the source issue is terminal but there is no durable same-run terminal activity after the stale evidence point -- there is independent evidence that the process may still be mutating external state, leaking resources, crossing company or ownership boundaries, or otherwise needs operator review +- there is independent evidence that the process may still be mutating external state, leaking resources, crossing company or ownership boundaries, or otherwise needs an operator decision -In the normal non-terminal case, critical silence can still create issue-backed evaluation work and block the source issue when blocking is necessary for correctness. In the source-resolved case, a completed source issue should not acquire a new manager review or blocker merely because an old run handle stayed active; only real unresolved work should block work. +In the normal non-terminal case, critical silence remains a UI signal and does not block the source issue. In the source-resolved case, a completed source issue does not acquire a new review or blocker merely because an old run handle stayed active. Only real unresolved work should block work. This is distinct from productivity review. Productivity review asks whether an assigned source issue has unusual progression patterns, such as no-comment terminal-run streaks, long active duration, or high churn. Source-resolved watchdog folding asks whether a stale active-run signal outlived a source issue that already reached a valid terminal disposition. One does not substitute for the other. @@ -786,7 +785,6 @@ Examples: - automatic stranded-work retry was already exhausted - a dependency graph has an invalid/uninvokable owner, unassigned blocker, or invalid review participant -- an active run is silent past the watchdog threshold The recovery action stays source-scoped by default. Stranded-task escalation is board-owned and records the cause, evidence, next action, source and return owner, `routingPolicy: board_escalation_no_takeover_v1`, and wake or monitor policy in the source thread/detail surface. diff --git a/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts b/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts index de08684702..1c886ee632 100644 --- a/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts +++ b/server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts @@ -4,6 +4,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest import { activityLog, agents, + agentWakeupRequests, companies, createDb, heartbeatRunEvents, @@ -22,47 +23,8 @@ import { ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS, ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, - heartbeatService, -} from "../services/heartbeat.ts"; -import { recoveryService } from "../services/recovery/service.ts"; -import { getRunLogStore } from "../services/run-log-store.ts"; - -const mockAdapterExecute = vi.hoisted(() => - vi.fn(async () => ({ - exitCode: 0, - signal: null, - timedOut: false, - errorMessage: null, - summary: "Acknowledged stale-run evaluation.", - provider: "test", - model: "test-model", - })), -); - -vi.mock("../telemetry.ts", () => ({ - getTelemetryClient: () => ({ track: vi.fn() }), -})); - -vi.mock("@paperclipai/shared/telemetry", async () => { - const actual = await vi.importActual( - "@paperclipai/shared/telemetry", - ); - return { - ...actual, - trackAgentFirstHeartbeat: vi.fn(), - }; -}); - -vi.mock("../adapters/index.ts", async () => { - const actual = await vi.importActual("../adapters/index.ts"); - return { - ...actual, - getServerAdapter: vi.fn(() => ({ - supportsLocalAgentJwt: false, - execute: mockAdapterExecute, - })), - }; -}); + recoveryService, +} from "../services/recovery/service.js"; const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -90,9 +52,7 @@ async function truncateCompaniesWithDeadlockRetry(db: ReturnType setTimeout(resolve, 50 * (attempt + 1))); } } @@ -108,14 +68,6 @@ describeEmbeddedPostgres("active-run output watchdog", () => { }, 30_000); afterEach(async () => { - for (let attempt = 0; attempt < 100; attempt += 1) { - const activeRuns = await db - .select({ id: heartbeatRuns.id }) - .from(heartbeatRuns) - .where(sql`${heartbeatRuns.status} in ('queued', 'running')`); - if (activeRuns.length === 0) break; - await new Promise((resolve) => setTimeout(resolve, 25)); - } await truncateCompaniesWithDeadlockRetry(db); }); @@ -127,10 +79,9 @@ describeEmbeddedPostgres("active-run output watchdog", () => { now: Date; ageMs: number; withOutput?: boolean; - logChunk?: string; - sourceStatus?: "in_progress" | "done" | "cancelled"; + sourceStatus?: "in_progress" | "blocked" | "done" | "cancelled"; sourceOriginKind?: string; - sameRunTerminalEvidence?: "activity" | "comment"; + sameRunTerminalEvidence?: boolean; }) { const companyId = randomUUID(); const managerId = randomUUID(); @@ -203,28 +154,11 @@ describeEmbeddedPostgres("active-run output watchdog", () => { lastOutputSeq: opts.withOutput ? 3 : 0, lastOutputStream: opts.withOutput ? "stdout" : null, contextSnapshot: { issueId }, - stdoutExcerpt: "OPENAI_API_KEY=sk-test-secret-value should not leak", logBytes: 0, }); - if (opts.logChunk) { - const store = getRunLogStore(); - const handle = await store.begin({ companyId, agentId: coderId, runId }); - const logBytes = await store.append(handle, { - stream: "stdout", - chunk: opts.logChunk, - ts: startedAt.toISOString(), - }); - await db - .update(heartbeatRuns) - .set({ - logStore: handle.store, - logRef: handle.logRef, - logBytes, - }) - .where(eq(heartbeatRuns.id, runId)); - } await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); - if (opts.sameRunTerminalEvidence === "activity") { + + if (opts.sameRunTerminalEvidence) { await db.insert(activityLog).values({ companyId, actorType: "agent", @@ -241,774 +175,440 @@ describeEmbeddedPostgres("active-run output watchdog", () => { }, createdAt: terminalEvidenceAt, }); - } else if (opts.sameRunTerminalEvidence === "comment") { - await db.insert(issueComments).values({ - companyId, - issueId, - authorAgentId: coderId, - authorType: "agent", - createdByRunId: runId, - body: "Completed and verified.", - createdAt: terminalEvidenceAt, - updatedAt: terminalEvidenceAt, - }); } - return { companyId, managerId, coderId, issueId, runId, issuePrefix }; + + return { companyId, managerId, coderId, issueId, runId, issuePrefix, startedAt }; } - it("creates one medium-priority evaluation issue for a suspicious silent run", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, runId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); + function createRecovery() { + const enqueueWakeup = vi.fn(); + return { enqueueWakeup, recovery: recoveryService(db, { enqueueWakeup }) }; + } - const first = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const second = await heartbeat.scanSilentActiveRuns({ now, companyId }); + async function buildSummary(runId: string, now: Date) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + if (!run) throw new Error(`Missing test run ${runId}`); + return recoveryService(db, { enqueueWakeup: vi.fn() }).buildRunOutputSilence(run, now); + } - expect(first.created).toBe(1); - expect(second.created).toBe(0); - expect(second.existing).toBe(1); + async function expectNoReviewArtifacts(input: { + companyId: string; + issueId: string; + coderId: string; + managerId: string; + }) { + const [evaluations, comments, relations, actions, wakes, source, coder, manager] = await Promise.all([ + db.select().from(issues).where(and( + eq(issues.companyId, input.companyId), + eq(issues.originKind, "stale_active_run_evaluation"), + )), + db.select().from(issueComments).where(eq(issueComments.issueId, input.issueId)), + db.select().from(issueRelations).where(eq(issueRelations.companyId, input.companyId)), + db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, input.companyId)), + db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, input.companyId)), + db.select().from(issues).where(eq(issues.id, input.issueId)).then((rows) => rows[0]), + db.select().from(agents).where(eq(agents.id, input.coderId)).then((rows) => rows[0]), + db.select().from(agents).where(eq(agents.id, input.managerId)).then((rows) => rows[0]), + ]); - const evaluations = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluations).toHaveLength(1); - expect(["todo", "in_progress"]).toContain(evaluations[0]?.status); - expect(evaluations[0]).toMatchObject({ - priority: "medium", - assigneeAgentId: managerId, - assigneeAdapterOverrides: { modelProfile: "cheap" }, - originId: runId, - originFingerprint: `stale_active_run:${companyId}:${runId}`, - }); - expect(evaluations[0]?.description).toContain("Decision Checklist"); - expect(evaluations[0]?.description).not.toContain("sk-test-secret-value"); - }); - - it("redacts sensitive values from actual run-log evidence", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const leakedJwt = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"; - const leakedGithubToken = "ghp_1234567890abcdefghijklmnopqrstuvwxyz"; - const { companyId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - logChunk: [ - "Authorization: Bearer live-bearer-token-value", - `POST payload {"apiKey":"json-secret-value","token":"${leakedJwt}"}`, - `GITHUB_TOKEN=${leakedGithubToken}`, - ].join("\n"), - }); - const heartbeat = heartbeatService(db); - - await heartbeat.scanSilentActiveRuns({ now, companyId }); - - const [evaluation] = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluation?.description).toContain("***REDACTED***"); - expect(evaluation?.description).not.toContain("live-bearer-token-value"); - expect(evaluation?.description).not.toContain("json-secret-value"); - expect(evaluation?.description).not.toContain(leakedJwt); - expect(evaluation?.description).not.toContain(leakedGithubToken); - }); - - it("raises critical stale-run evaluations with high priority but does not hard-block the source issue", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, issueId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - - const result = await heartbeat.scanSilentActiveRuns({ now, companyId }); - - expect(result.created).toBe(1); - const [evaluation] = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluation?.priority).toBe("high"); - - // Evaluation issues are observability-only — they must NOT be added as hard blockers - // on the source issue. That caused the self-amplifying loop (KIV-1590). - const blockerRelations = await db - .select() - .from(issueRelations) - .where(and(eq(issueRelations.companyId, companyId), eq(issueRelations.relatedIssueId, issueId))); - expect(blockerRelations).toHaveLength(0); - - const [source] = await db.select().from(issues).where(eq(issues.id, issueId)); - expect(source?.status).not.toBe("blocked"); - }); - - it("emits the source-issue escalation comment only once across repeated critical scans", async () => { - // Regression: when the same evaluation issue stays open and the watchdog re-evaluates the - // run as critical on every scan cycle, ensureSourceIssueCommentedForStaleEvaluation must NOT - // re-add the escalation comment to the source issue. Without the idempotency guard the - // source-issue thread is spammed once per scan cycle. - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, issueId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - - const first = await heartbeat.scanSilentActiveRuns({ now, companyId }); - expect(first.created).toBe(1); - const evaluationIssueId = first.evaluationIssueIds[0]!; - - const commentsAfterFirst = await db - .select({ id: issueComments.id }) - .from(issueComments) - .where(eq(issueComments.issueId, issueId)); - expect(commentsAfterFirst).toHaveLength(1); - - // Run the scan again at a slightly later time — the evaluation issue is still open - // and the run is still critical, so the existing-branch path re-invokes - // ensureSourceIssueCommentedForStaleEvaluation. The guard must suppress the second comment. - const later = new Date(now.getTime() + 60_000); - const second = await heartbeat.scanSilentActiveRuns({ now: later, companyId }); - expect(second.created).toBe(0); - - const commentsAfterSecond = await db - .select({ id: issueComments.id }) - .from(issueComments) - .where(eq(issueComments.issueId, issueId)); - expect(commentsAfterSecond).toHaveLength(1); - - // Activity-log escalation entries are the persistence record — also exactly one. - const escalations = await db - .select({ id: activityLog.id }) - .from(activityLog) - .where( - and( - eq(activityLog.companyId, companyId), - eq(activityLog.action, "heartbeat.output_stale_escalated"), - eq(activityLog.entityType, "issue"), - eq(activityLog.entityId, issueId), - sql`${activityLog.details} ->> 'evaluationIssueId' = ${evaluationIssueId}`, - ), - ); - expect(escalations).toHaveLength(1); - }); - - it("skips ticket creation when the source issue is blocked (idle is expected)", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, issueId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - - // Mark the source issue as blocked before scanning - await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issueId)); - - const result = await heartbeat.scanSilentActiveRuns({ now, companyId }); - - expect(result.created).toBe(0); - expect(result.skipped).toBe(1); - - const evaluations = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); expect(evaluations).toHaveLength(0); - }); + expect(comments).toHaveLength(0); + expect(relations).toHaveLength(0); + expect(actions).toHaveLength(0); + expect(wakes).toHaveLength(0); + expect(source?.assigneeAgentId).toBe(input.coderId); + expect(coder?.status).toBe("running"); + expect(manager?.status).toBe("idle"); + } - it("does not re-file after a stale-run evaluation is dismissed as false positive", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, runId, managerId } = await seedRunningRun({ - now, + it.each([ + { + level: "suspicious" as const, ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - const recovery = recoveryService(db, { enqueueWakeup: vi.fn() }); - - const first = await heartbeat.scanSilentActiveRuns({ now, companyId }); - expect(first.created).toBe(1); - const evaluationIssueId = first.evaluationIssueIds[0]!; - - // Reviewer records a dismissed_false_positive decision and closes the ticket - await recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - decision: "dismissed_false_positive", - evaluationIssueId, - reason: "SADE is blocked on KIV-1066 — silence is expected", - now, - }); - await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId)); - - // Scan again — should not create a new ticket because of the dismissed_false_positive decision - const second = await heartbeat.scanSilentActiveRuns({ now, companyId }); - expect(second.created).toBe(0); - expect(second.skipped).toBe(1); - - const allEvaluations = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(allEvaluations).toHaveLength(1); - }); - - it("folds terminal source issues with same-run durable evidence instead of creating watchdog work", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, coderId, issueId, runId } = await seedRunningRun({ - now, + }, + { + level: "critical" as const, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, - sourceStatus: "done", - sameRunTerminalEvidence: "activity", - }); - const heartbeat = heartbeatService(db); - - const result = await heartbeat.scanSilentActiveRuns({ now, companyId }); - - expect(result).toMatchObject({ created: 0, folded: 1, skipped: 0 }); - const evaluations = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluations).toHaveLength(0); - - const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); - expect(run?.status).toBe("succeeded"); - expect(run?.errorCode).toBeNull(); - expect(run?.finishedAt?.toISOString()).toBe(now.toISOString()); - expect(run?.resultJson).toMatchObject({ - sourceResolvedWatchdogFold: { - sourceIssueId: issueId, - sourceIssueStatus: "done", - sameRunEvidenceKind: "activity", - evaluationIssueId: null, - evaluationIssueIdentifier: null, - cleanup: { outcome: "no_process_metadata" }, - }, - }); - - const [source] = await db.select().from(issues).where(eq(issues.id, issueId)); - expect(source?.executionRunId).toBeNull(); - const [agent] = await db.select().from(agents).where(eq(agents.id, coderId)); - expect(agent?.status).toBe("idle"); - const [decision] = await db - .select() - .from(heartbeatRunWatchdogDecisions) - .where(eq(heartbeatRunWatchdogDecisions.runId, runId)); - expect(decision?.decision).toBe("dismissed_false_positive"); - const [event] = await db - .select() - .from(heartbeatRunEvents) - .where(eq(heartbeatRunEvents.runId, runId)); - expect(event?.message).toContain("Source-resolved watchdog fold"); - }); - - it("still escalates terminal source issues without same-run terminal evidence", async () => { + }, + ])("surfaces $level silence without creating recovery work", async ({ level, ageMs }) => { const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, runId } = await seedRunningRun({ + const seeded = await seedRunningRun({ now, ageMs }); + const { enqueueWakeup, recovery } = createRecovery(); + + await expect(recovery.buildRunOutputSilence( + (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0]!, now, - ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, - sourceStatus: "done", + )).resolves.toMatchObject({ + level, + silenceAgeMs: ageMs, + suspicionThresholdMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, + criticalThresholdMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, + evaluationIssueId: null, + evaluationIssueIdentifier: null, + evaluationIssueAssigneeAgentId: null, }); - const heartbeat = heartbeatService(db); - const result = await heartbeat.scanSilentActiveRuns({ now, companyId }); + const first = await recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId }); + const second = await recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId }); + expect(first).toMatchObject({ scanned: 1, created: 0, existing: 0, escalated: 0, skipped: 1 }); + expect(second).toMatchObject({ scanned: 1, created: 0, existing: 0, escalated: 0, skipped: 1 }); + expect(first.evaluationIssueIds).toEqual([]); + expect(second.evaluationIssueIds).toEqual([]); + expect(enqueueWakeup).not.toHaveBeenCalled(); + await expectNoReviewArtifacts(seeded); - expect(result).toMatchObject({ created: 1, folded: 0 }); - const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); - expect(run?.status).toBe("running"); - const [evaluation] = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluation?.originId).toBe(runId); - expect(evaluation?.parentId).toBeNull(); - }); - - it("still escalates when a same-run comment is followed by another actor marking the source done", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, issueId, runId, issuePrefix } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, - sourceStatus: "in_progress", - sameRunTerminalEvidence: "comment", - }); - const completedAt = new Date(now.getTime() - 5 * 60_000); - await db - .update(issues) - .set({ status: "done", completedAt, updatedAt: completedAt }) - .where(eq(issues.id, issueId)); - await db.insert(activityLog).values({ - companyId, - actorType: "user", - actorId: "board-user", - agentId: null, - runId: null, - action: "issue.updated", - entityType: "issue", - entityId: issueId, - details: { - identifier: `${issuePrefix}-1`, - status: "done", - _previous: { status: "in_progress" }, - }, - createdAt: completedAt, - }); - const heartbeat = heartbeatService(db); - - const result = await heartbeat.scanSilentActiveRuns({ now, companyId }); - - expect(result).toMatchObject({ created: 1, folded: 0 }); - const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); - expect(run?.status).toBe("running"); - const [evaluation] = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluation?.originId).toBe(runId); - expect(evaluation?.parentId).toBeNull(); - }); - - it("folds existing evaluation and active watchdog recovery action idempotently", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, issueId, runId, issuePrefix } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, - sourceStatus: "done", - sameRunTerminalEvidence: "activity", - }); - const evaluationIssueId = randomUUID(); - await db.insert(issues).values({ - id: evaluationIssueId, - companyId, - title: "Existing stale evaluation", - status: "todo", - priority: "high", - assigneeAgentId: managerId, - issueNumber: 2, - identifier: `${issuePrefix}-2`, - originKind: "stale_active_run_evaluation", - originId: runId, - originRunId: runId, - originFingerprint: `stale_active_run:${companyId}:${runId}`, - }); - await db.insert(issueRelations).values({ - companyId, - issueId: evaluationIssueId, - relatedIssueId: issueId, - type: "blocks", - }); - await db.insert(issueRecoveryActions).values({ - companyId, - sourceIssueId: issueId, - recoveryIssueId: evaluationIssueId, - kind: "active_run_watchdog", - status: "active", - ownerType: "agent", - ownerAgentId: managerId, - cause: "active_run_watchdog", - fingerprint: `active-run-watchdog:${companyId}:${runId}:${issueId}`, - evidence: { runId }, - nextAction: "Review stale active run", - }); - const heartbeat = heartbeatService(db); - - const first = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const second = await heartbeat.scanSilentActiveRuns({ now, companyId }); - - expect(first).toMatchObject({ created: 0, folded: 1 }); - expect(second).toMatchObject({ scanned: 0, created: 0, folded: 0 }); - const [evaluation] = await db.select().from(issues).where(eq(issues.id, evaluationIssueId)); - expect(evaluation?.status).toBe("done"); - const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); - expect(run?.resultJson).toMatchObject({ - sourceResolvedWatchdogFold: { - sourceIssueId: issueId, - sourceIssueStatus: "done", - evaluationIssueId, - evaluationIssueIdentifier: `${issuePrefix}-2`, - }, - }); - const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); - expect(action?.status).toBe("resolved"); - expect(action?.outcome).toBe("false_positive"); const decisions = await db .select() .from(heartbeatRunWatchdogDecisions) - .where(eq(heartbeatRunWatchdogDecisions.runId, runId)); - expect(decisions).toHaveLength(1); + .where(eq(heartbeatRunWatchdogDecisions.runId, seeded.runId)); + expect(decisions).toHaveLength(0); }); - it("refuses recovery-on-recovery stale-run recursion", async () => { + it("keeps blocked and recovery-origin sources artifact-free", async () => { const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId } = await seedRunningRun({ + const blocked = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "blocked", + }); + const recursive = await seedRunningRun({ now, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, sourceOriginKind: "stale_active_run_evaluation", }); - const heartbeat = heartbeatService(db); + const { enqueueWakeup, recovery } = createRecovery(); - const result = await heartbeat.scanSilentActiveRuns({ now, companyId }); + await expect(recovery.scanSilentActiveRuns({ now, companyId: blocked.companyId })) + .resolves.toMatchObject({ created: 0, skipped: 1 }); + await expect(recovery.scanSilentActiveRuns({ now, companyId: recursive.companyId })) + .resolves.toMatchObject({ created: 0, skipped: 1 }); + await expectNoReviewArtifacts(blocked); - expect(result).toMatchObject({ created: 0, skipped: 1 }); - const evaluations = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluations).toHaveLength(1); + const recursiveIssues = await db.select().from(issues).where(eq(issues.companyId, recursive.companyId)); + expect(recursiveIssues).toHaveLength(1); + expect(recursiveIssues[0]?.id).toBe(recursive.issueId); + expect(await db.select().from(issueComments).where(eq(issueComments.issueId, recursive.issueId))).toHaveLength(0); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, recursive.companyId))).toHaveLength(0); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, recursive.companyId))).toHaveLength(0); + expect(enqueueWakeup).not.toHaveBeenCalled(); }); - it("skips snoozed runs and healthy noisy runs", async () => { + it("stores board snooze decisions directly on the run", async () => { const now = new Date("2026-04-22T20:00:00.000Z"); - const stale = await seedRunningRun({ + const seeded = await seedRunningRun({ now, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, }); - const noisy = await seedRunningRun({ + const { recovery } = createRecovery(); + const snoozedUntil = new Date(now.getTime() + 60 * 60 * 1000); + + const decision = await recovery.recordWatchdogDecision({ + runId: seeded.runId, + actor: { type: "board" }, + decision: "snooze", + snoozedUntil, + reason: "Known quiet compile", + now, + }); + expect(decision).toMatchObject({ + runId: seeded.runId, + evaluationIssueId: null, + decision: "snooze", + snoozedUntil, + }); + await expect(buildSummary(seeded.runId, now)).resolves.toMatchObject({ + level: "snoozed", + snoozedUntil, + evaluationIssueId: null, + }); + await expect(buildSummary(seeded.runId, new Date(snoozedUntil.getTime() + 1))).resolves.toMatchObject({ + level: "critical", + snoozedUntil: null, + }); + await expectNoReviewArtifacts(seeded); + }); + + it("re-arms board continue decisions after 30 minutes without creating artifacts", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, + }); + const { enqueueWakeup, recovery } = createRecovery(); + const decision = await recovery.recordWatchdogDecision({ + runId: seeded.runId, + actor: { type: "board" }, + decision: "continue", + reason: "Keep watching this run", + now, + }); + const rearmAt = new Date(now.getTime() + ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS); + + expect(decision.evaluationIssueId).toBeNull(); + expect(decision.snoozedUntil?.toISOString()).toBe(rearmAt.toISOString()); + await expect(buildSummary(seeded.runId, new Date(rearmAt.getTime() - 1))).resolves.toMatchObject({ + level: "snoozed", + evaluationIssueId: null, + }); + await expect(buildSummary(seeded.runId, new Date(rearmAt.getTime() + 1))).resolves.toMatchObject({ + level: "suspicious", + evaluationIssueId: null, + }); + await expect(recovery.scanSilentActiveRuns({ now: new Date(rearmAt.getTime() - 1), companyId: seeded.companyId })) + .resolves.toMatchObject({ snoozed: 1, created: 0 }); + await expect(recovery.scanSilentActiveRuns({ now: new Date(rearmAt.getTime() + 1), companyId: seeded.companyId })) + .resolves.toMatchObject({ skipped: 1, created: 0 }); + expect(enqueueWakeup).not.toHaveBeenCalled(); + await expectNoReviewArtifacts(seeded); + }); + + it("permanently suppresses a run after a board false-positive decision", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, + }); + const { enqueueWakeup, recovery } = createRecovery(); + + const decision = await recovery.recordWatchdogDecision({ + runId: seeded.runId, + actor: { type: "board" }, + decision: "dismissed_false_positive", + reason: "This run is expected to remain quiet", + now, + }); + expect(decision.evaluationIssueId).toBeNull(); + await expect(buildSummary(seeded.runId, now)).resolves.toMatchObject({ + level: "not_applicable", + snoozedUntil: null, + evaluationIssueId: null, + }); + const muchLater = new Date(now.getTime() + 24 * 60 * 60 * 1000); + await expect(buildSummary(seeded.runId, muchLater)).resolves.toMatchObject({ + level: "not_applicable", + snoozedUntil: null, + }); + await expect(recovery.scanSilentActiveRuns({ now: muchLater, companyId: seeded.companyId })) + .resolves.toMatchObject({ created: 0, skipped: 1 }); + expect(enqueueWakeup).not.toHaveBeenCalled(); + await expectNoReviewArtifacts(seeded); + }); + + it("folds a terminal source with same-run evidence without creating review work", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "done", + sameRunTerminalEvidence: true, + }); + const { enqueueWakeup, recovery } = createRecovery(); + + const result = await recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId }); + expect(result).toMatchObject({ created: 0, folded: 1, skipped: 0 }); + expect(enqueueWakeup).not.toHaveBeenCalled(); + + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + const [source] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); + const [agent] = await db.select().from(agents).where(eq(agents.id, seeded.coderId)); + expect(run?.status).toBe("succeeded"); + expect(run?.resultJson).toMatchObject({ + sourceResolvedWatchdogFold: { + sourceIssueId: seeded.issueId, + sourceIssueStatus: "done", + evaluationIssueId: null, + cleanup: { outcome: "no_process_metadata" }, + }, + }); + expect(source?.executionRunId).toBeNull(); + expect(agent?.status).toBe("idle"); + expect(await db.select().from(issues).where(and( + eq(issues.companyId, seeded.companyId), + eq(issues.originKind, "stale_active_run_evaluation"), + ))).toHaveLength(0); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, seeded.companyId))).toHaveLength(0); + }); + + it("does not fold or create review work for a terminal source without same-run evidence", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "done", + }); + const { enqueueWakeup, recovery } = createRecovery(); + + await expect(recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .resolves.toMatchObject({ created: 0, folded: 0, skipped: 1 }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + expect(run?.status).toBe("running"); + expect(enqueueWakeup).not.toHaveBeenCalled(); + expect(await db.select().from(issues).where(and( + eq(issues.companyId, seeded.companyId), + eq(issues.originKind, "stale_active_run_evaluation"), + ))).toHaveLength(0); + }); + + it("folds existing legacy evaluation and recovery rows idempotently", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + sourceStatus: "done", + sameRunTerminalEvidence: true, + }); + const evaluationIssueId = randomUUID(); + await db.insert(issues).values({ + id: evaluationIssueId, + companyId: seeded.companyId, + title: "Existing stale evaluation", + status: "todo", + priority: "high", + assigneeAgentId: seeded.managerId, + issueNumber: 2, + identifier: `${seeded.issuePrefix}-2`, + originKind: "stale_active_run_evaluation", + originId: seeded.runId, + originRunId: seeded.runId, + originFingerprint: `stale_active_run:${seeded.companyId}:${seeded.runId}`, + }); + await db.insert(issueRelations).values({ + companyId: seeded.companyId, + issueId: evaluationIssueId, + relatedIssueId: seeded.issueId, + type: "blocks", + }); + await db.insert(issueRecoveryActions).values({ + companyId: seeded.companyId, + sourceIssueId: seeded.issueId, + recoveryIssueId: evaluationIssueId, + kind: "active_run_watchdog", + status: "active", + ownerType: "agent", + ownerAgentId: seeded.managerId, + cause: "active_run_watchdog", + fingerprint: `active-run-watchdog:${seeded.companyId}:${seeded.runId}:${seeded.issueId}`, + evidence: { runId: seeded.runId }, + nextAction: "Review stale active run", + }); + const { recovery } = createRecovery(); + + await expect(recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .resolves.toMatchObject({ created: 0, folded: 1 }); + await expect(recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .resolves.toMatchObject({ scanned: 0, created: 0, folded: 0 }); + const [evaluation] = await db.select().from(issues).where(eq(issues.id, evaluationIssueId)); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, seeded.issueId)); + expect(evaluation?.status).toBe("done"); + expect(action).toMatchObject({ status: "resolved", outcome: "false_positive" }); + expect(await db.select().from(heartbeatRunWatchdogDecisions).where(eq( + heartbeatRunWatchdogDecisions.runId, + seeded.runId, + ))).toHaveLength(1); + expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, seeded.runId))).toHaveLength(1); + }); + + it("keeps open legacy evaluations readable without refreshing or reprioritizing them", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, + }); + const evaluationIssueId = randomUUID(); + const evaluationUpdatedAt = new Date("2026-04-20T12:00:00.000Z"); + await db.insert(issues).values({ + id: evaluationIssueId, + companyId: seeded.companyId, + title: "Legacy silent-run evaluation", + status: "todo", + priority: "medium", + assigneeAgentId: seeded.managerId, + issueNumber: 2, + identifier: `${seeded.issuePrefix}-2`, + originKind: "stale_active_run_evaluation", + originId: seeded.runId, + originRunId: seeded.runId, + originFingerprint: `stale_active_run:${seeded.companyId}:${seeded.runId}`, + updatedAt: evaluationUpdatedAt, + }); + const { enqueueWakeup, recovery } = createRecovery(); + + await expect(buildSummary(seeded.runId, now)).resolves.toMatchObject({ + level: "critical", + evaluationIssueId, + evaluationIssueIdentifier: `${seeded.issuePrefix}-2`, + evaluationIssueAssigneeAgentId: seeded.managerId, + }); + await expect(recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .resolves.toMatchObject({ created: 0, existing: 1, escalated: 0 }); + const [evaluation] = await db.select().from(issues).where(eq(issues.id, evaluationIssueId)); + expect(evaluation).toMatchObject({ status: "todo", priority: "medium", assigneeAgentId: seeded.managerId }); + expect(evaluation?.updatedAt.toISOString()).toBe(evaluationUpdatedAt.toISOString()); + expect(await db.select().from(issueComments).where(eq(issueComments.issueId, evaluationIssueId))).toHaveLength(0); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, seeded.companyId))).toHaveLength(0); + expect(enqueueWakeup).not.toHaveBeenCalled(); + + await expect(recovery.recordWatchdogDecision({ + runId: seeded.runId, + actor: { type: "agent", agentId: seeded.managerId }, + decision: "continue", + evaluationIssueId, + reason: "Resolve through the legacy review", + now, + })).resolves.toMatchObject({ evaluationIssueId, createdByAgentId: seeded.managerId }); + await expect(recovery.recordWatchdogDecision({ + runId: seeded.runId, + actor: { type: "agent", agentId: randomUUID() }, + decision: "continue", + evaluationIssueId, + reason: "Not assigned", + now, + })).rejects.toMatchObject({ status: 403 }); + }); + + it("does not recreate or auto-dismiss a closed legacy evaluation", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ + now, + ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, + }); + const evaluationIssueId = randomUUID(); + await db.insert(issues).values({ + id: evaluationIssueId, + companyId: seeded.companyId, + title: "Closed legacy evaluation", + status: "done", + priority: "medium", + assigneeAgentId: seeded.managerId, + issueNumber: 2, + identifier: `${seeded.issuePrefix}-2`, + originKind: "stale_active_run_evaluation", + originId: seeded.runId, + originRunId: seeded.runId, + originFingerprint: `stale_active_run:${seeded.companyId}:${seeded.runId}`, + }); + const { recovery } = createRecovery(); + + await expect(recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .resolves.toMatchObject({ created: 0, existing: 0, skipped: 1 }); + expect(await db.select().from(issues).where(eq(issues.companyId, seeded.companyId))).toHaveLength(2); + expect(await db.select().from(heartbeatRunWatchdogDecisions).where(eq( + heartbeatRunWatchdogDecisions.runId, + seeded.runId, + ))).toHaveLength(0); + }); + + it("ignores healthy runs that produced recent output", async () => { + const now = new Date("2026-04-22T20:00:00.000Z"); + const seeded = await seedRunningRun({ now, ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000, withOutput: true, }); - await db.insert(heartbeatRunWatchdogDecisions).values({ - companyId: stale.companyId, - runId: stale.runId, - decision: "snooze", - snoozedUntil: new Date(now.getTime() + 60 * 60 * 1000), - reason: "Intentional quiet run", - }); - const heartbeat = heartbeatService(db); + const { recovery } = createRecovery(); - const staleResult = await heartbeat.scanSilentActiveRuns({ now, companyId: stale.companyId }); - const noisyResult = await heartbeat.scanSilentActiveRuns({ now, companyId: noisy.companyId }); - - expect(staleResult).toMatchObject({ created: 0, snoozed: 1 }); - expect(noisyResult).toMatchObject({ scanned: 0, created: 0 }); - }); - - it("records watchdog decisions through recovery owner authorization", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, runId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - const recovery = recoveryService(db, { enqueueWakeup: vi.fn() }); - - const scan = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const evaluationIssueId = scan.evaluationIssueIds[0]; - expect(evaluationIssueId).toBeTruthy(); - - await expect( - recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: randomUUID() }, - decision: "continue", - evaluationIssueId, - reason: "not my recovery issue", - }), - ).rejects.toMatchObject({ status: 403 }); - - const snoozedUntil = new Date(now.getTime() + 60 * 60 * 1000); - const decision = await recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - decision: "snooze", - evaluationIssueId, - reason: "Long compile with no output", - snoozedUntil, - }); - - expect(decision).toMatchObject({ - runId, - evaluationIssueId, - decision: "snooze", - createdByAgentId: managerId, - }); - await expect(recovery.buildRunOutputSilence({ - id: runId, - companyId, - status: "running", - lastOutputAt: null, - lastOutputSeq: 0, - lastOutputStream: null, - processStartedAt: new Date(now.getTime() - ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS - 60_000), - startedAt: new Date(now.getTime() - ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS - 60_000), - createdAt: new Date(now.getTime() - ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS - 60_000), - }, now)).resolves.toMatchObject({ - level: "snoozed", - snoozedUntil, - evaluationIssueId, - }); - }); - - it("re-arms continue decisions after the default quiet window", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, runId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - const recovery = recoveryService(db, { enqueueWakeup: vi.fn() }); - - const scan = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const evaluationIssueId = scan.evaluationIssueIds[0]; - expect(evaluationIssueId).toBeTruthy(); - - const decision = await recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - decision: "continue", - evaluationIssueId, - reason: "Current evidence is acceptable; keep watching.", - now, - }); - const rearmAt = new Date(now.getTime() + ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS); - expect(decision).toMatchObject({ - runId, - evaluationIssueId, - decision: "continue", - createdByAgentId: managerId, - }); - expect(decision.snoozedUntil?.toISOString()).toBe(rearmAt.toISOString()); - - await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId)); - - const beforeRearm = await heartbeat.scanSilentActiveRuns({ - now: new Date(rearmAt.getTime() - 60_000), - companyId, - }); - expect(beforeRearm).toMatchObject({ created: 0, snoozed: 1 }); - - const afterRearm = await heartbeat.scanSilentActiveRuns({ - now: new Date(rearmAt.getTime() + 60_000), - companyId, - }); - expect(afterRearm.created).toBe(1); - expect(afterRearm.evaluationIssueIds[0]).not.toBe(evaluationIssueId); - - const evaluations = await db - .select() - .from(issues) - .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); - expect(evaluations.filter((issue) => !["done", "cancelled"].includes(issue.status))).toHaveLength(1); - }); - - it("rejects agent watchdog decisions using issues not bound to the target run", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, coderId, runId, issuePrefix } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - const recovery = recoveryService(db, { enqueueWakeup: vi.fn() }); - - const scan = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const evaluationIssueId = scan.evaluationIssueIds[0]; - expect(evaluationIssueId).toBeTruthy(); - - const unrelatedIssueId = randomUUID(); - await db.insert(issues).values({ - id: unrelatedIssueId, - companyId, - title: "Assigned but unrelated", - status: "todo", - priority: "medium", - assigneeAgentId: managerId, - issueNumber: 20, - identifier: `${issuePrefix}-20`, - }); - - const otherRunId = randomUUID(); - const otherEvaluationIssueId = randomUUID(); - await db.insert(heartbeatRuns).values({ - id: otherRunId, - companyId, - agentId: coderId, - status: "running", - invocationSource: "assignment", - triggerDetail: "system", - startedAt: new Date(now.getTime() - ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS - 120_000), - processStartedAt: new Date(now.getTime() - ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS - 120_000), - lastOutputAt: null, - lastOutputSeq: 0, - lastOutputStream: null, - contextSnapshot: {}, - logBytes: 0, - }); - await db.insert(issues).values({ - id: otherEvaluationIssueId, - companyId, - title: "Other run evaluation", - status: "todo", - priority: "medium", - assigneeAgentId: managerId, - issueNumber: 21, - identifier: `${issuePrefix}-21`, - originKind: "stale_active_run_evaluation", - originId: otherRunId, - originFingerprint: `stale_active_run:${companyId}:${otherRunId}`, - }); - - const attempts = [ - { decision: "continue" as const, evaluationIssueId: unrelatedIssueId }, - { decision: "dismissed_false_positive" as const, evaluationIssueId: unrelatedIssueId }, - { - decision: "snooze" as const, - evaluationIssueId: unrelatedIssueId, - snoozedUntil: new Date(now.getTime() + 60 * 60 * 1000), - }, - { decision: "continue" as const, evaluationIssueId: otherEvaluationIssueId }, - ]; - - for (const attempt of attempts) { - await expect( - recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - reason: "malicious or stale binding", - ...attempt, - }), - ).rejects.toMatchObject({ status: 403 }); - } - - await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId)); - await expect( - recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - decision: "continue", - evaluationIssueId, - reason: "closed evaluation should not authorize", - }), - ).rejects.toMatchObject({ status: 403 }); - }); - - it("suppresses repeat alerts when evaluation is closed on the board without a watchdog decision", async () => { - // Regression: KIV-1519–KIV-1618 — watchdog re-fired 18+ times for the same run because - // CEO closed each alert as 'done' directly on the board without recording a watchdog decision. - // The unique constraint only prevents duplicates while an issue is open, so once it's closed - // the scanner created a fresh one every cycle. - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, runId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - - // First scan: creates the evaluation issue. - const first = await heartbeat.scanSilentActiveRuns({ now, companyId }); - expect(first.created).toBe(1); - const evaluationIssueId = first.evaluationIssueIds[0]; - expect(evaluationIssueId).toBeTruthy(); - - // Reviewer closes the issue on the board — no watchdog decision recorded. - await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId)); - - // Second scan: should NOT create a new issue. Instead auto-records dismissed_false_positive. - const second = await heartbeat.scanSilentActiveRuns({ now, companyId }); - expect(second.created).toBe(0); - expect(second.skipped).toBe(1); - - // All subsequent scans also skip. - const third = await heartbeat.scanSilentActiveRuns({ now, companyId }); - expect(third.created).toBe(0); - expect(third.skipped).toBe(1); - - // The auto-recorded dismissed_false_positive decision must be persisted. - const decisions = await db - .select() - .from(heartbeatRunWatchdogDecisions) - .where(and(eq(heartbeatRunWatchdogDecisions.runId, runId), eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"))); - expect(decisions).toHaveLength(1); - expect(decisions[0].evaluationIssueId).toBe(evaluationIssueId); - }); - - it("still allows re-arm after continue decision even when issue was closed on board", async () => { - // When a human explicitly recorded a 'continue' decision the watchdog lifecycle is active; - // closing the issue on the board should not permanently suppress future alerts. - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, runId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - const recovery = recoveryService(db, { enqueueWakeup: vi.fn() }); - - const scan = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const evaluationIssueId = scan.evaluationIssueIds[0]; - expect(evaluationIssueId).toBeTruthy(); - - // Human records a 'continue' decision. - await recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - decision: "continue", - evaluationIssueId, - reason: "Expected quiet period.", - now, - }); - - // Board also closes the evaluation issue. - await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId)); - - // After the re-arm window, watchdog should still fire a new alert. - const rearmAt = new Date(now.getTime() + ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS + 60_000); - const rearm = await heartbeat.scanSilentActiveRuns({ now: rearmAt, companyId }); - expect(rearm.created).toBe(1); - expect(rearm.evaluationIssueIds[0]).not.toBe(evaluationIssueId); - }); - - it("validates createdByRunId before storing watchdog decisions", async () => { - const now = new Date("2026-04-22T20:00:00.000Z"); - const { companyId, managerId, runId } = await seedRunningRun({ - now, - ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000, - }); - const heartbeat = heartbeatService(db); - const recovery = recoveryService(db, { enqueueWakeup: vi.fn() }); - - const scan = await heartbeat.scanSilentActiveRuns({ now, companyId }); - const evaluationIssueId = scan.evaluationIssueIds[0]; - expect(evaluationIssueId).toBeTruthy(); - - await expect( - recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId }, - decision: "continue", - evaluationIssueId, - reason: "client supplied another agent run", - createdByRunId: runId, - }), - ).rejects.toMatchObject({ status: 403 }); - - const managerRunId = randomUUID(); - await db.insert(heartbeatRuns).values({ - id: managerRunId, - companyId, - agentId: managerId, - status: "running", - invocationSource: "assignment", - triggerDetail: "system", - startedAt: now, - processStartedAt: now, - lastOutputAt: now, - lastOutputSeq: 1, - lastOutputStream: "stdout", - contextSnapshot: {}, - logBytes: 0, - }); - - const decision = await recovery.recordWatchdogDecision({ - runId, - actor: { type: "agent", agentId: managerId, runId: managerRunId }, - decision: "continue", - evaluationIssueId, - reason: "valid current actor run", - createdByRunId: randomUUID(), - }); - expect(decision.createdByRunId).toBe(managerRunId); + await expect(buildSummary(seeded.runId, now)).resolves.toMatchObject({ level: "ok" }); + await expect(recovery.scanSilentActiveRuns({ now, companyId: seeded.companyId })) + .resolves.toMatchObject({ scanned: 0, created: 0 }); }); }); diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 4d3f08d9ab..9ac53ea64f 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -34,7 +34,6 @@ import { visibleIssueCondition } from "../issue-visibility.js"; import { forbidden, notFound } from "../../errors.js"; import { logger } from "../../middleware/logger.js"; import { isPidAlive, isProcessGroupAlive, terminateLocalService } from "../local-service-supervisor.js"; -import { redactCurrentUserText } from "../../log-redaction.js"; import { redactSensitiveText } from "../../redaction.js"; import { isUniqueViolation } from "../../db-errors.js"; import { logActivity } from "../activity-log.js"; @@ -55,7 +54,6 @@ import { } from "../issue-dependency-wakeups.js"; import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; import { isHeartbeatWakeOnDemandEnabled } from "../heartbeat-policy.js"; -import { getRunLogStore } from "../run-log-store.js"; import { DEFAULT_MAX_SUCCESSFUL_RUN_HANDOFF_ATTEMPTS, FINISH_SUCCESSFUL_RUN_HANDOFF_REASON, @@ -98,7 +96,6 @@ export const ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS = 60 * 60 * 1000; export const ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS = 4 * 60 * 60 * 1000; export const ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS = 30 * 60 * 1000; export const DEFAULT_LIVENESS_REESCALATION_COOLDOWN_MS = 60 * 60 * 1000; -const ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES = 8 * 1024; const STRANDED_ISSUE_RECOVERY_ORIGIN_KIND = RECOVERY_ORIGIN_KINDS.strandedIssueRecovery; const STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND = RECOVERY_ORIGIN_KINDS.staleActiveRunEvaluation; const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext"; @@ -597,15 +594,6 @@ function agentUiLink(agent: { id: string; name: string | null } | null, prefix: return `[${agent.name ?? agent.id}](/${prefix}/agents/${agent.id})`; } -function formatDuration(ms: number | null) { - if (ms === null) return "unknown"; - const minutes = Math.floor(ms / 60_000); - if (minutes < 60) return `${minutes}m`; - const hours = Math.floor(minutes / 60); - const remainingMinutes = minutes % 60; - return remainingMinutes > 0 ? `${hours}h ${remainingMinutes}m` : `${hours}h`; -} - function formatIssueLinksForComment(relations: Array<{ identifier?: string | null }>) { const identifiers = [ ...new Set( @@ -624,36 +612,6 @@ function formatIssueLinksForComment(relations: Array<{ identifier?: string | nul .join(", "); } -function unwrapDatabaseConflictError(error: unknown) { - if (!error || typeof error !== "object") return null; - - const candidate = error as { - code?: string; - constraint?: string; - constraint_name?: string; - message?: string; - cause?: unknown; - }; - - if ( - typeof candidate.code === "string" || - typeof candidate.constraint === "string" || - typeof candidate.constraint_name === "string" - ) { - return candidate; - } - - const cause = candidate.cause; - if (!cause || typeof cause !== "object") return candidate; - - return cause as { - code?: string; - constraint?: string; - constraint_name?: string; - message?: string; - }; -} - function isStrandedIssueRecoveryIssue(issue: Pick) { return isStrandedIssueRecoveryOriginKind(issue.originKind); } @@ -794,13 +752,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) const treeControlSvc = issueTreeControlService(db); const budgets = budgetService(db); const instanceSettings = instanceSettingsService(db); - const runLogStore = getRunLogStore(); let resolvedDependencyWakeBackstopCandidateCursor: string | null = null; - const getCurrentUserRedactionOptions = async () => ({ - enabled: (await instanceSettings.getGeneral()).censorUsernameInLogs, - }); - async function getAgent(agentId: string) { return db.select().from(agents).where(eq(agents.id, agentId)).then((rows) => rows[0] ?? null); } @@ -1346,20 +1299,10 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) .then((rows) => rows[0]?.issuePrefix ?? "PAP"); } - function staleActiveRunOriginFingerprint(companyId: string, runId: string) { - return `stale_active_run:${companyId}:${runId}`; - } - function isTerminalIssueStatus(status: string | null | undefined) { return status === "done" || status === "cancelled"; } - function isRecoveryOriginIssue(issue: typeof issues.$inferSelect) { - return Object.values(RECOVERY_ORIGIN_KINDS).includes( - issue.originKind as typeof RECOVERY_ORIGIN_KINDS[keyof typeof RECOVERY_ORIGIN_KINDS], - ); - } - function silenceStartedAtForRun(run: Pick) { return run.lastOutputAt ?? run.processStartedAt ?? run.startedAt ?? run.createdAt ?? null; } @@ -1369,21 +1312,40 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) return startedAt ? Math.max(0, now.getTime() - startedAt.getTime()) : null; } - async function latestActiveOutputQuietUntilDecision(companyId: string, runId: string, now = new Date()) { - const [row] = await db - .select() - .from(heartbeatRunWatchdogDecisions) - .where( - and( - eq(heartbeatRunWatchdogDecisions.companyId, companyId), - eq(heartbeatRunWatchdogDecisions.runId, runId), - inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]), - gt(heartbeatRunWatchdogDecisions.snoozedUntil, now), - ), - ) - .orderBy(desc(heartbeatRunWatchdogDecisions.createdAt)) - .limit(1); - return row ?? null; + async function activeOutputDecisionState(companyId: string, runId: string, now = new Date()) { + const [quietUntilRows, dismissedRows] = await Promise.all([ + db + .select({ + decision: heartbeatRunWatchdogDecisions.decision, + snoozedUntil: heartbeatRunWatchdogDecisions.snoozedUntil, + }) + .from(heartbeatRunWatchdogDecisions) + .where( + and( + eq(heartbeatRunWatchdogDecisions.companyId, companyId), + eq(heartbeatRunWatchdogDecisions.runId, runId), + inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]), + gt(heartbeatRunWatchdogDecisions.snoozedUntil, now), + ), + ) + .orderBy(desc(heartbeatRunWatchdogDecisions.createdAt)) + .limit(1), + db + .select({ id: heartbeatRunWatchdogDecisions.id }) + .from(heartbeatRunWatchdogDecisions) + .where( + and( + eq(heartbeatRunWatchdogDecisions.companyId, companyId), + eq(heartbeatRunWatchdogDecisions.runId, runId), + eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"), + ), + ) + .limit(1), + ]); + return { + dismissedFalsePositive: dismissedRows.length > 0, + quietUntilDecision: quietUntilRows[0] ?? null, + }; } async function findOpenStaleRunEvaluation(companyId: string, runId: string) { @@ -1392,9 +1354,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) id: issues.id, identifier: issues.identifier, status: issues.status, - priority: issues.priority, assigneeAgentId: issues.assigneeAgentId, - updatedAt: issues.updatedAt, }) .from(issues) .where( @@ -1410,51 +1370,6 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) return row ?? null; } - // Returns a `done` stale-run evaluation issue for this run if one exists. - // Used to detect when a reviewer closed an alert directly on the board without going through - // the watchdog decision API — which would not leave a dismissed_false_positive decision record. - // - // Scoped to `done` only (not `cancelled`): cancellation is used by other system code paths - // and does not imply a reviewer's "false positive" verdict. `done` is the explicit - // board-close path used by reviewers acknowledging the alert. A cancelled evaluation is - // allowed to re-fire on the next scan; if a reviewer wants permanent suppression they - // should mark the alert done or record a watchdog decision. - async function findClosedStaleRunEvaluation(companyId: string, runId: string) { - const [row] = await db - .select({ id: issues.id, identifier: issues.identifier, status: issues.status }) - .from(issues) - .where( - and( - eq(issues.companyId, companyId), - eq(issues.originKind, STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND), - eq(issues.originId, runId), - visibleIssueCondition(), - eq(issues.status, "done"), - ), - ) - .orderBy(desc(issues.updatedAt)) - .limit(1); - return row ?? null; - } - - // Returns true when a reviewer has already dismissed this run's silence as a false positive. - // Used to prevent re-filing after a deliberate close — while still allowing legitimate - // re-arm after a "continue" decision's snooze window expires. - async function hasDismissedFalsePositiveDecision(companyId: string, runId: string) { - const [row] = await db - .select({ id: heartbeatRunWatchdogDecisions.id }) - .from(heartbeatRunWatchdogDecisions) - .where( - and( - eq(heartbeatRunWatchdogDecisions.companyId, companyId), - eq(heartbeatRunWatchdogDecisions.runId, runId), - eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"), - ), - ) - .limit(1); - return row != null; - } - async function buildRunOutputSilence( run: Pick< typeof heartbeatRuns.$inferSelect, @@ -1462,21 +1377,24 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) >, now = new Date(), ): Promise { - const [quietUntilDecision, evaluation] = await Promise.all([ - latestActiveOutputQuietUntilDecision(run.companyId, run.id, now), + const [decisionState, evaluation] = await Promise.all([ + activeOutputDecisionState(run.companyId, run.id, now), findOpenStaleRunEvaluation(run.companyId, run.id), ]); + const { dismissedFalsePositive, quietUntilDecision } = decisionState; const silenceStartedAt = silenceStartedAtForRun(run); const silenceAgeMs = run.status === "running" ? silenceAgeMsForRun(run, now) : null; const level = run.status !== "running" ? "not_applicable" - : quietUntilDecision - ? "snoozed" - : (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS - ? "critical" - : (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS - ? "suspicious" - : "ok"; + : dismissedFalsePositive + ? "not_applicable" + : quietUntilDecision + ? "snoozed" + : (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + ? "critical" + : (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + ? "suspicious" + : "ok"; return { lastOutputAt: run.lastOutputAt ?? null, lastOutputSeq: run.lastOutputSeq ?? 0, @@ -1488,37 +1406,13 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) level, suspicionThresholdMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, criticalThresholdMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, - snoozedUntil: quietUntilDecision?.snoozedUntil ?? null, + snoozedUntil: dismissedFalsePositive ? null : quietUntilDecision?.snoozedUntil ?? null, evaluationIssueId: evaluation?.id ?? null, evaluationIssueIdentifier: evaluation?.identifier ?? null, evaluationIssueAssigneeAgentId: evaluation?.assigneeAgentId ?? null, }; } - function redactWatchdogEvidenceText(value: string, currentUserRedactionOptions: Awaited>) { - return redactSensitiveText(redactCurrentUserText(value, currentUserRedactionOptions)); - } - - function truncateEvidenceText(value: string, maxChars = 4000) { - if (value.length <= maxChars) return value; - return `${value.slice(value.length - maxChars)}\n[truncated earlier evidence]`; - } - - async function readRunLogTailForEvidence(run: typeof heartbeatRuns.$inferSelect) { - if (!run.logStore || !run.logRef || !run.logBytes) return ""; - try { - const offset = Math.max(0, run.logBytes - ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES); - const result = await runLogStore.read( - { store: run.logStore as "local_file", logRef: run.logRef }, - { offset, limitBytes: ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES }, - ); - return result.content; - } catch (err) { - logger.warn({ err, runId: run.id }, "failed to read stale-run watchdog evidence tail"); - return ""; - } - } - async function resolveStaleRunSourceIssue(run: typeof heartbeatRuns.$inferSelect) { const issueId = issueIdFromRunContext(run.contextSnapshot); if (!issueId) return null; @@ -1833,267 +1727,21 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) return { kind: "folded" as const, evaluationIssueId: input.existingEvaluation?.id ?? null }; } - async function resolveStaleRunOwnerAgentId(input: { - run: typeof heartbeatRuns.$inferSelect; - runningAgent: typeof agents.$inferSelect; - sourceIssue: typeof issues.$inferSelect | null; - }) { - const candidateIds: string[] = []; - if (input.sourceIssue?.assigneeAgentId) { - const sourceAssignee = await getAgent(input.sourceIssue.assigneeAgentId); - if (sourceAssignee?.reportsTo) candidateIds.push(sourceAssignee.reportsTo); - } - if (input.runningAgent.reportsTo) candidateIds.push(input.runningAgent.reportsTo); - const roleCandidates = await db - .select() - .from(agents) - .where(and(eq(agents.companyId, input.run.companyId), inArray(agents.role, ["cto", "ceo"]))) - .orderBy(sql`case when ${agents.role} = 'cto' then 0 else 1 end`, asc(agents.createdAt)); - candidateIds.push(...roleCandidates.map((agent) => agent.id)); - - const seen = new Set(); - for (const agentId of candidateIds) { - if (seen.has(agentId)) continue; - seen.add(agentId); - const candidate = await getAgent(agentId); - if (!candidate || candidate.companyId !== input.run.companyId) continue; - const budgetBlock = await budgets.getInvocationBlock(input.run.companyId, candidate.id, { - issueId: input.sourceIssue?.id ?? null, - projectId: input.sourceIssue?.projectId ?? null, - }); - if ( - (await isAgentInvokable(candidate)) && - isHeartbeatWakeOnDemandEnabled(candidate) && - !budgetBlock - ) { - return candidate.id; - } - } - - return null; - } - - async function collectStaleRunEvidence(input: { - run: typeof heartbeatRuns.$inferSelect; - runningAgent: typeof agents.$inferSelect; - sourceIssue: typeof issues.$inferSelect | null; - prefix: string; - now: Date; - }) { - const [tail, recentEvents, childIssues, blockers] = await Promise.all([ - readRunLogTailForEvidence(input.run), - db - .select({ - eventType: heartbeatRunEvents.eventType, - level: heartbeatRunEvents.level, - message: heartbeatRunEvents.message, - createdAt: heartbeatRunEvents.createdAt, - }) - .from(heartbeatRunEvents) - .where(and(eq(heartbeatRunEvents.companyId, input.run.companyId), eq(heartbeatRunEvents.runId, input.run.id))) - .orderBy(desc(heartbeatRunEvents.id)) - .limit(8), - input.sourceIssue - ? db - .select({ id: issues.id, identifier: issues.identifier, title: issues.title, status: issues.status }) - .from(issues) - .where(and(eq(issues.companyId, input.run.companyId), eq(issues.parentId, input.sourceIssue.id), visibleIssueCondition())) - .orderBy(desc(issues.updatedAt)) - .limit(8) - : Promise.resolve([]), - input.sourceIssue - ? db - .select({ id: issues.id, identifier: issues.identifier, title: issues.title, status: issues.status }) - .from(issueRelations) - .innerJoin(issues, eq(issueRelations.issueId, issues.id)) - .where( - and( - eq(issueRelations.companyId, input.run.companyId), - eq(issueRelations.relatedIssueId, input.sourceIssue.id), - eq(issueRelations.type, "blocks"), - ), - ) - .limit(8) - : Promise.resolve([]), - ]); - const currentUserRedactionOptions = await getCurrentUserRedactionOptions(); - const safeTail = truncateEvidenceText(redactWatchdogEvidenceText(tail, currentUserRedactionOptions)); - const silenceAgeMs = silenceAgeMsForRun(input.run, input.now); - return { - safeTail, - silenceAgeMs, - recentEvents: recentEvents.reverse().map((event) => ({ - eventType: event.eventType, - level: event.level, - createdAt: event.createdAt.toISOString(), - message: event.message ? truncateEvidenceText(redactWatchdogEvidenceText(event.message, currentUserRedactionOptions), 300) : null, - })), - childIssues, - blockers, - }; - } - - function buildStaleRunEvaluationDescription(input: { - run: typeof heartbeatRuns.$inferSelect; - runningAgent: typeof agents.$inferSelect; - sourceIssue: typeof issues.$inferSelect | null; - prefix: string; - evidence: Awaited>; - level: "suspicious" | "critical"; - now: Date; - }) { - const sourceIssue = input.sourceIssue - ? issueUiLink({ identifier: input.sourceIssue.identifier, id: input.sourceIssue.id }, input.prefix) - : "none"; - const recentEvents = input.evidence.recentEvents.length > 0 - ? input.evidence.recentEvents.map((event) => - `- ${event.createdAt} \`${event.eventType}\`${event.level ? ` ${event.level}` : ""}: ${event.message ?? "(no message)"}`, - ).join("\n") - : "- none"; - const childIssues = input.evidence.childIssues.length > 0 - ? input.evidence.childIssues.map((issue) => - `- ${issueUiLink({ identifier: issue.identifier, id: issue.id }, input.prefix)} \`${issue.status}\`: ${issue.title}`, - ).join("\n") - : "- none detected"; - const blockers = input.evidence.blockers.length > 0 - ? input.evidence.blockers.map((issue) => - `- ${issueUiLink({ identifier: issue.identifier, id: issue.id }, input.prefix)} \`${issue.status}\`: ${issue.title}`, - ).join("\n") - : "- none detected"; - return [ - `Paperclip detected ${input.level} output silence on an active heartbeat run.`, - "", - "## Run", - "", - `- Run: ${runUiLink(input.run, input.prefix)}`, - `- Agent: ${input.runningAgent.name} (${input.runningAgent.adapterType})`, - `- Invocation: ${input.run.invocationSource}${input.run.triggerDetail ? ` / ${input.run.triggerDetail}` : ""}`, - `- Source issue: ${sourceIssue}`, - `- Started at: ${input.run.startedAt?.toISOString() ?? "unknown"}`, - `- Process started at: ${input.run.processStartedAt?.toISOString() ?? "unknown"}`, - `- Last output at: ${input.run.lastOutputAt?.toISOString() ?? "none recorded"}`, - `- Last output sequence: ${input.run.lastOutputSeq ?? 0}`, - `- Silent for: ${formatDuration(input.evidence.silenceAgeMs)}`, - `- Thresholds: suspicious after ${formatDuration(ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS)}, critical after ${formatDuration(ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS)}`, - `- Process metadata: pid \`${input.run.processPid ?? "unknown"}\`, process group \`${input.run.processGroupId ?? "unknown"}\`, in-memory handle \`${runningProcesses.has(input.run.id) ? "yes" : "no"}\``, - "", - "## Last Output Excerpt", - "", - input.evidence.safeTail ? `\`\`\`text\n${input.evidence.safeTail}\n\`\`\`` : "_No run-log tail was available._", - "", - "## Recent Run Events", - "", - recentEvents, - "", - "## Related Work", - "", - "Active child issues:", - childIssues, - "", - "Current source blockers:", - blockers, - "", - "## Decision Checklist", - "", - "- Continue or snooze if the run is intentionally quiet.", - "- Ask the run owner for context if work may be delegated outside the transcript.", - "- Preserve artifacts, branch state, and useful output before cancellation.", - "- Cancel or recover through the explicit run recovery controls when authorized.", - "- Close this issue as a false positive only after recording the reason.", - ].join("\n"); - } - - function isUniqueStaleRunEvaluationConflict(error: unknown) { - const maybe = unwrapDatabaseConflictError(error); - if (!maybe) return false; - return maybe.code === "23505" && - ( - maybe.constraint === "issues_active_stale_run_evaluation_uq" || - maybe.constraint_name === "issues_active_stale_run_evaluation_uq" || - typeof maybe.message === "string" && maybe.message.includes("issues_active_stale_run_evaluation_uq") - ); - } - - async function ensureSourceIssueCommentedForStaleEvaluation(input: { - sourceIssue: typeof issues.$inferSelect | null; - evaluationIssue: { id: string; identifier: string | null }; - run: typeof heartbeatRuns.$inferSelect; - }) { - if (!input.sourceIssue || ["done", "cancelled"].includes(input.sourceIssue.status)) return false; - // Idempotency guard: if we've already emitted the escalation comment for this - // (sourceIssue, evaluationIssue) pair, skip. Without this, every subsequent scan - // cycle while the evaluation issue is still open re-fires the comment and spams - // the source-issue thread. The activity log row written below is the persistence - // record we check against — a single row per pair is enough to suppress repeats - // even after process restarts. - const [priorEscalation] = await db - .select({ id: activityLog.id }) - .from(activityLog) - .where( - and( - eq(activityLog.companyId, input.sourceIssue.companyId), - eq(activityLog.action, "heartbeat.output_stale_escalated"), - eq(activityLog.entityType, "issue"), - eq(activityLog.entityId, input.sourceIssue.id), - sql`${activityLog.details} ->> 'evaluationIssueId' = ${input.evaluationIssue.id}`, - ), - ) - .limit(1); - if (priorEscalation) return false; - // Evaluation issues are observability-only — do NOT add them to blockedByIssueIds. - // They are already parented under the source issue. Adding them as hard blockers - // creates a self-amplifying loop: block → silence → new alert → block again. - await issuesSvc.addComment(input.sourceIssue.id, [ - "Paperclip detected critical output silence on this issue's active run.", - "", - `- Evaluation issue: ${input.evaluationIssue.identifier ?? input.evaluationIssue.id}`, - `- Run: \`${input.run.id}\``, - "", - "Review the evaluation issue above. The active run has not been cancelled.", - ].join("\n"), { runId: input.run.id }); - await logActivity(db, { - companyId: input.sourceIssue.companyId, - actorType: "system", - actorId: "system", - agentId: null, - runId: input.run.id, - action: "heartbeat.output_stale_escalated", - entityType: "issue", - entityId: input.sourceIssue.id, - details: { - source: "recovery.scan_silent_active_runs", - evaluationIssueId: input.evaluationIssue.id, - }, - }); - return true; - } - - async function createOrUpdateStaleRunEvaluation(input: { + async function inspectSilentActiveRun(input: { run: typeof heartbeatRuns.$inferSelect; now: Date; + dismissedFalsePositive: boolean; }) { const runningAgent = await getAgent(input.run.agentId); if (!runningAgent || runningAgent.companyId !== input.run.companyId) return { kind: "skipped" as const }; const sourceIssue = await resolveStaleRunSourceIssue(input.run); const existing = await findOpenStaleRunEvaluation(input.run.companyId, input.run.id); - if (sourceIssue && isRecoveryOriginIssue(sourceIssue)) { - await logActivity(db, { - companyId: input.run.companyId, - actorType: "system", - actorId: "system", - agentId: input.run.agentId, - runId: input.run.id, - action: "heartbeat.output_stale_recovery_recursion_refused", - entityType: "heartbeat_run", - entityId: input.run.id, - details: { - source: "recovery.scan_silent_active_runs", - sourceIssueId: sourceIssue.id, - sourceIssueIdentifier: sourceIssue.identifier, - sourceIssueOriginKind: sourceIssue.originKind, - existingEvaluationIssueId: existing?.id ?? null, - }, - }); + if ( + sourceIssue && + Object.values(RECOVERY_ORIGIN_KINDS).includes( + sourceIssue.originKind as typeof RECOVERY_ORIGIN_KINDS[keyof typeof RECOVERY_ORIGIN_KINDS], + ) + ) { return { kind: "skipped" as const }; } const silenceStartedAt = silenceStartedAtForRun(input.run); @@ -2117,184 +1765,17 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) } } - // Idle output is expected when the source issue is blocked — skip ticket creation entirely. + // Blocked source work can be intentionally quiet. The issue state already carries + // the durable waiting signal, so the cleanup scan has nothing to do. if (sourceIssue?.status === "blocked") return { kind: "skipped" as const }; - // Dedup: if a reviewer has dismissed this run's silence as a false positive, don't re-file. - // A "continue" decision with a snooze window is allowed to re-arm normally — only an - // explicit dismissed_false_positive blocks all further alerts for this run. - if (await hasDismissedFalsePositiveDecision(input.run.companyId, input.run.id)) { + if (input.dismissedFalsePositive) { return { kind: "skipped" as const }; } - // Dedup: if a prior evaluation issue for this run was closed `done` on the board - // without going through the watchdog decision API, no dismissed_false_positive record exists - // and the watchdog would re-fire every cycle. Auto-record the suppression now so future - // cycles skip immediately via hasDismissedFalsePositiveDecision. - // - // Exception: if any watchdog decision exists (snooze/continue), a human explicitly opted - // in to the watchdog lifecycle — honour that and allow re-arm as designed. - // - // Concurrency: the check-then-insert runs inside a transaction with a per-(company,run) - // advisory lock so two overlapping scans cannot both observe `hasAnyDecision = false` - // and both insert a dismissed_false_positive row. The table has no unique constraint - // on (companyId, runId, decision), so the advisory lock is the serialization point. - const closedEvaluation = await findClosedStaleRunEvaluation(input.run.companyId, input.run.id); - if (closedEvaluation) { - const autoDismissed = await db.transaction(async (tx) => { - await tx.execute( - sql`SELECT pg_advisory_xact_lock(hashtextextended(${`watchdog_dismiss:${input.run.companyId}:${input.run.id}`}, 0))`, - ); - const hasAnyDecision = await tx - .select({ id: heartbeatRunWatchdogDecisions.id }) - .from(heartbeatRunWatchdogDecisions) - .where( - and( - eq(heartbeatRunWatchdogDecisions.companyId, input.run.companyId), - eq(heartbeatRunWatchdogDecisions.runId, input.run.id), - ), - ) - .limit(1) - .then((rows) => rows.length > 0); - if (hasAnyDecision) return false; - await tx.insert(heartbeatRunWatchdogDecisions).values({ - companyId: input.run.companyId, - runId: input.run.id, - evaluationIssueId: closedEvaluation.id, - decision: "dismissed_false_positive", - snoozedUntil: null, - reason: `Auto-recorded: evaluation issue ${closedEvaluation.identifier} was closed as ${closedEvaluation.status} on the board without a watchdog decision.`, - createdByAgentId: null, - createdByUserId: null, - createdByRunId: null, - }); - return true; - }); - if (autoDismissed) { - return { kind: "skipped" as const }; - } - } - - const prefix = await getCompanyIssuePrefix(input.run.companyId); - const evidence = await collectStaleRunEvidence({ - run: input.run, - runningAgent, - sourceIssue, - prefix, - now: input.now, - }); - const level = (evidence.silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS ? "critical" : "suspicious"; - if (existing) { - if (level === "critical" && existing.priority !== "high") { - await issuesSvc.update(existing.id, { - priority: "high", - }); - await issuesSvc.addComment(existing.id, [ - "Critical output silence threshold crossed.", - "", - `- Run: \`${input.run.id}\``, - `- Silent for: ${formatDuration(evidence.silenceAgeMs)}`, - `- Last output at: ${input.run.lastOutputAt?.toISOString() ?? "none recorded"}`, - ].join("\n"), { runId: input.run.id }); - await ensureSourceIssueCommentedForStaleEvaluation({ - sourceIssue, - evaluationIssue: existing, - run: input.run, - }); - return { kind: "escalated" as const, evaluationIssueId: existing.id }; - } - if (level === "critical") { - await ensureSourceIssueCommentedForStaleEvaluation({ - sourceIssue, - evaluationIssue: existing, - run: input.run, - }); - } - return { kind: "existing" as const, evaluationIssueId: existing.id }; - } - - const ownerAgentId = await resolveStaleRunOwnerAgentId({ run: input.run, runningAgent, sourceIssue }); - const description = buildStaleRunEvaluationDescription({ - run: input.run, - runningAgent, - sourceIssue, - prefix, - evidence, - level, - now: input.now, - }); - let evaluation: Awaited>; - try { - evaluation = await issuesSvc.create(input.run.companyId, { - title: `Review silent active run for ${runningAgent.name}`, - description, - status: "todo", - priority: level === "critical" ? "high" : "medium", - parentId: sourceIssue && !["done", "cancelled"].includes(sourceIssue.status) ? sourceIssue.id : null, - projectId: sourceIssue?.projectId ?? null, - goalId: sourceIssue?.goalId ?? null, - billingCode: sourceIssue?.billingCode ?? null, - assigneeAgentId: ownerAgentId, - assigneeAdapterOverrides: recoveryAssigneeAdapterOverrides("status_only"), - originKind: STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND, - originId: input.run.id, - originRunId: input.run.id, - originFingerprint: staleActiveRunOriginFingerprint(input.run.companyId, input.run.id), - }); - } catch (error) { - if (!isUniqueStaleRunEvaluationConflict(error)) throw error; - const raced = await findOpenStaleRunEvaluation(input.run.companyId, input.run.id); - if (!raced) throw error; - return { kind: "existing" as const, evaluationIssueId: raced.id }; - } - - await logActivity(db, { - companyId: input.run.companyId, - actorType: "system", - actorId: "system", - agentId: ownerAgentId, - runId: input.run.id, - action: "heartbeat.output_stale_detected", - entityType: "issue", - entityId: evaluation.id, - details: { - source: "recovery.scan_silent_active_runs", - level, - sourceIssueId: sourceIssue?.id ?? null, - silenceAgeMs: evidence.silenceAgeMs, - lastOutputAt: input.run.lastOutputAt?.toISOString() ?? null, - }, - }); - if (level === "critical") { - await ensureSourceIssueCommentedForStaleEvaluation({ - sourceIssue, - evaluationIssue: evaluation, - run: input.run, - }); - } - if (ownerAgentId) { - await deps.enqueueWakeup(ownerAgentId, { - source: "assignment", - triggerDetail: "system", - reason: "issue_assigned", - payload: withRecoveryModelProfileHint({ - issueId: evaluation.id, - staleRunId: input.run.id, - sourceIssueId: sourceIssue?.id ?? null, - }, "status_only"), - requestedByActorType: "system", - requestedByActorId: null, - contextSnapshot: withRecoveryModelProfileHint({ - issueId: evaluation.id, - taskId: evaluation.id, - wakeReason: "issue_assigned", - source: STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND, - staleRunId: input.run.id, - sourceIssueId: sourceIssue?.id ?? null, - }, "status_only"), - }); - } - return { kind: "created" as const, evaluationIssueId: evaluation.id }; + return existing + ? { kind: "existing" as const, evaluationIssueId: existing.id } + : { kind: "skipped" as const }; } async function scanSilentActiveRuns(opts?: { now?: Date; companyId?: string; issueCreatedAtGte?: Date | null }) { @@ -2346,14 +1827,17 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup }) }; for (const run of candidates) { - if (await latestActiveOutputQuietUntilDecision(run.companyId, run.id, now)) { + const decisionState = await activeOutputDecisionState(run.companyId, run.id, now); + if (decisionState.quietUntilDecision) { result.snoozed += 1; continue; } - const outcome = await createOrUpdateStaleRunEvaluation({ run, now }); - if (outcome.kind === "created") result.created += 1; - else if (outcome.kind === "existing") result.existing += 1; - else if (outcome.kind === "escalated") result.escalated += 1; + const outcome = await inspectSilentActiveRun({ + run, + now, + dismissedFalsePositive: decisionState.dismissedFalsePositive, + }); + if (outcome.kind === "existing") result.existing += 1; else if (outcome.kind === "folded") result.folded += 1; else result.skipped += 1; if ("evaluationIssueId" in outcome && outcome.evaluationIssueId) { diff --git a/ui/src/components/IssueRunLedger.test.tsx b/ui/src/components/IssueRunLedger.test.tsx index 586e462a97..c1c7239a01 100644 --- a/ui/src/components/IssueRunLedger.test.tsx +++ b/ui/src/components/IssueRunLedger.test.tsx @@ -448,7 +448,7 @@ describe("IssueRunLedger", () => { expect(container.textContent).toContain("2 older items not shown"); }); - it("renders stale-run banner, watchdog actions, and silence badge for live runs", () => { + it("renders legacy evaluation context with watchdog actions and a silence badge", () => { const onWatchdogDecision = vi.fn(); renderLedger({ runs: [createRun({ runId: "run-live-1", status: "running", finishedAt: null })], @@ -456,11 +456,12 @@ describe("IssueRunLedger", () => { onWatchdogDecision, }); - expect(container.textContent).toContain("Stale-run watchdog alert"); + expect(container.textContent).toContain("Critical output silence"); expect(container.textContent).toContain("PAP-404"); - expect(container.textContent).toContain("Stale run"); + expect(container.textContent).toContain("Critical silence"); + expect(container.textContent).toContain("Paperclip did not create new delegated recovery work"); const watchdogBanner = Array.from(container.querySelectorAll("p")) - .find((node) => node.textContent?.includes("Stale-run watchdog alert")) + .find((node) => node.textContent?.includes("Critical output silence")) ?.closest("div"); expect(watchdogBanner?.className).toContain("border-red-500/30"); expect(watchdogBanner?.className).toContain("bg-red-500/10"); @@ -479,6 +480,56 @@ describe("IssueRunLedger", () => { }); }); + it.each([ + { + level: "suspicious" as const, + heading: "Output silence watchdog warning", + badge: "Output silence", + }, + { + level: "critical" as const, + heading: "Critical output silence", + badge: "Critical silence", + }, + ])("renders a $level UI-only signal without an evaluation-task link", ({ level, heading, badge }) => { + const onWatchdogDecision = vi.fn(); + const activeRun = createActiveRun(); + renderLedger({ + runs: [createRun({ runId: activeRun.id, status: "running", finishedAt: null })], + activeRun: createActiveRun({ + outputSilence: { + ...activeRun.outputSilence!, + level, + evaluationIssueId: null, + evaluationIssueIdentifier: null, + evaluationIssueAssigneeAgentId: null, + }, + }), + onWatchdogDecision, + }); + + expect(container.textContent).toContain(heading); + expect(container.textContent).toContain(badge); + expect(container.textContent).toContain("Paperclip did not create or assign a recovery task"); + expect(container.textContent).not.toContain("PAP-404"); + expect(container.querySelector('a[href^="/issues/"]')).toBeNull(); + expect(container.textContent).toContain("Continue monitoring"); + expect(container.textContent).toContain("Snooze 1h"); + expect(container.textContent).toContain("Mark false positive"); + + const continueButton = Array.from(container.querySelectorAll("button")).find( + (button) => button.textContent?.includes("Continue monitoring"), + ); + act(() => { + continueButton?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + expect(onWatchdogDecision).toHaveBeenCalledWith({ + runId: "run-live-1", + decision: "continue", + evaluationIssueId: null, + }); + }); + it("renders requested/applied model profile and surfaces fallback reasons", () => { renderLedger({ runs: [ @@ -523,7 +574,7 @@ describe("IssueRunLedger", () => { onWatchdogDecision, }); - expect(container.textContent).toContain("Stale-run watchdog alert"); + expect(container.textContent).toContain("Critical output silence"); expect(container.textContent).toContain("PAP-404"); expect(container.textContent).not.toContain("Continue monitoring"); expect(container.textContent).not.toContain("Snooze 1h"); diff --git a/ui/src/components/IssueRunLedger.tsx b/ui/src/components/IssueRunLedger.tsx index a69693b4b7..c1de0e6a96 100644 --- a/ui/src/components/IssueRunLedger.tsx +++ b/ui/src/components/IssueRunLedger.tsx @@ -143,11 +143,11 @@ type RunOutputSilenceCopy = { const RUN_OUTPUT_SILENCE_COPY: Partial> = { suspicious: { - label: "Silence watch", + label: "Output silence", tone: "border-amber-500/30 bg-amber-500/10 text-amber-700 dark:text-amber-300", }, critical: { - label: "Stale run", + label: "Critical silence", tone: "border-red-500/30 bg-red-500/10 text-red-700 dark:text-red-300", }, snoozed: { @@ -609,7 +609,7 @@ export function IssueRunLedgerContent({ >

{latestSilentRun.outputSilence.level === "critical" - ? "Stale-run watchdog alert" + ? "Critical output silence" : "Output silence watchdog warning"}

@@ -629,6 +629,11 @@ export function IssueRunLedgerContent({ ) : null}

+

+ {latestSilentRun.outputSilence.evaluationIssueIdentifier + ? "This signal is informational. Paperclip did not create new delegated recovery work." + : "This signal is informational. Paperclip did not create or assign a recovery task."} +

{onWatchdogDecision && canRecordWatchdogDecisions ? (