fix(recovery): make silent-run detection UI-only (#12242)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The recovery service detects active runs that stop producing output. > - The dashboard already shows suspicious and critical silence to the board. > - The recovery scan also creates delegated evaluation work for the same signal. > - Output silence alone does not prove that the run or source task needs recovery. > - This pull request keeps the signal and removes automatic recovery artifacts. > - The benefit is a visible watchdog signal without assignment changes, wake requests, or issue noise. ## Linked Issues or Issue Description - Refs #6596 - Refs #7036 - Refs #9475 - Refs #11544 - Refs #11839 - Refs #11961 ## What Changed - Keep the one-hour suspicious level and four-hour critical level in active-run API summaries. - Stop output silence from creating or changing issues, recovery actions, comments, relations, assignments, and wake requests. - Store snooze, continue, and false-positive decisions against the run without an evaluation issue. - Preserve terminal-source folding, orphan cleanup, and open legacy evaluation links. - Show informational watchdog copy and board controls without requiring an evaluation-task link. - Document the UI-only watchdog contract. - Add focused server and UI coverage for artifact-free scans and board decisions. ## Verification - `pnpm -r typecheck` - `pnpm exec vitest run server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts ui/src/components/IssueRunLedger.test.tsx` (32 tests passed) - `pnpm build` - `pnpm check:token-gates` - `git diff --check` - `pnpm test:run` completed locally with 4,772 passing tests. It found 30 unrelated macOS test-harness failures in eight workspace, skill, listener, and runtime exposure files. The failures use `/tmp` and `/private/tmp` as different paths, require Linux `/proc` listener data, or derive invalid HMR ports from the macOS ephemeral range. - The full Linux CI matrix passed on the latest commit. It includes build, typecheck, server tests, worker tests, serialization tests, canary, and e2e tests. - Greptile reviewed the latest commit at 5/5 with no actionable findings. ## Risks - The recovery scan keeps its existing result shape, but its created and escalated counts remain zero for output silence. - A false-positive decision now suppresses the signal for the full life of that run. - Open legacy evaluation issues remain visible and manually resolvable. The scan does not refresh or reprioritize them. - There is no database migration and no API schema change. > I checked `ROADMAP.md`. This change corrects existing watchdog behavior and does not duplicate planned core work. ## Model Used - OpenAI Codex, GPT-5, with extended reasoning, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and the focused tests and non-platform gates pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
GitHub
parent
381b763846
commit
75b6d22aac
|
|
@ -521,6 +521,8 @@ V1 non-terminal liveness rule:
|
|||
- recovery-action ownership is separate from source-task ownership: automatic repair and board escalation preserve both source assignee fields; reassignment requires an explicit board decision or a policy-defined serious failure
|
||||
- source-scoped recovery routing is cause-keyed: bounded continuity and disposition repair may retry only the original agent; provider-quota failures create/reuse a scheduled wait-recovery monitor; every other exhausted or unsafe path creates/reuses a board-owned recovery action with `routingPolicy: board_escalation_no_takeover_v1` and no substitute-agent wake
|
||||
- legacy active agent-owned recovery actions remain readable, resolvable, and API-compatible after upgrade, but reconciliation does not enqueue another takeover wake for them
|
||||
- active-run output silence is an informational board UI signal at one hour (`suspicious`) and four hours (`critical`); it does not create or update issues or recovery actions, comment on or block source work, change assignments, or wake an agent
|
||||
- board snooze and continue decisions suppress the run signal until their stored re-arm time; a false-positive decision suppresses it permanently for that run; open legacy evaluation issues remain readable and manually resolvable without automatic refresh
|
||||
|
||||
Detailed ownership, execution, blocker, active-run watchdog, crash-recovery, and non-terminal liveness semantics are documented in `doc/execution-semantics.md`.
|
||||
|
||||
|
|
|
|||
|
|
@ -580,7 +580,7 @@ On startup and on the periodic recovery loop, Paperclip now does five things in
|
|||
1. reap orphaned `running` runs
|
||||
2. resume persisted `queued` runs
|
||||
3. reconcile stranded assigned work
|
||||
4. scan silent active runs, revalidate their source issues, and either fold source-resolved watchdogs or create/update explicit watchdog recovery actions
|
||||
4. scan silent active runs only for source-aware terminal folding and legacy cleanup; API reads classify ordinary output silence for the board UI
|
||||
5. reconcile productivity reviews
|
||||
|
||||
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
|
||||
|
|
@ -717,27 +717,26 @@ An active run can still be unhealthy even when its process is `running`. Papercl
|
|||
The recovery service owns this contract:
|
||||
|
||||
- classify active-run output silence as `ok`, `suspicious`, `critical`, `snoozed`, or `not_applicable`
|
||||
- collect bounded evidence from run logs, recent run events, child issues, and blockers
|
||||
- preserve redaction and truncation before evidence is written to issue descriptions
|
||||
- create at most one open watchdog recovery action per run; issue-backed implementations use `stale_active_run_evaluation` issues
|
||||
- honor active snooze decisions before creating more review work
|
||||
- honor active snooze and continue decisions on the run
|
||||
- permanently suppress the signal for a run after a `dismissed_false_positive` decision
|
||||
- build the `outputSilence` summary shown by live-run and active-run API responses
|
||||
- retain links to open legacy `stale_active_run_evaluation` issues without refreshing or changing them
|
||||
|
||||
Suspicious silence creates a medium-priority watchdog recovery action for the selected recovery owner. Critical silence raises that recovery action to high priority and, when issue-backed evaluation is needed for correctness, blocks the source issue on the explicit evaluation task without cancelling the active process.
|
||||
Suspicious and critical silence are informational board UI signals. They do not create an issue or recovery action. They do not comment on or block the source issue. They do not change an assignment, wake an agent, cancel the active process, or change the run. The board uses the existing run controls when it decides that intervention is necessary.
|
||||
|
||||
Watchdog decisions are explicit operator/recovery-owner decisions:
|
||||
Watchdog decisions are explicit board decisions stored against the run:
|
||||
|
||||
- `snooze` records an operator-chosen future quiet-until time and suppresses scan-created review work during that window
|
||||
- `snooze` records an operator-chosen future quiet-until time and hides the signal during that window
|
||||
- `continue` records that the current evidence is acceptable, does not cancel or mutate the active run, and sets a 30-minute default re-arm window before the watchdog evaluates the still-silent run again
|
||||
- `dismissed_false_positive` records why the review was not actionable
|
||||
- `dismissed_false_positive` records why the signal was not actionable and suppresses it permanently for that run
|
||||
|
||||
Operators should prefer `snooze` for known time-bounded quiet periods. `continue` is only a short acknowledgement of the current evidence; if the run remains silent after the re-arm window, the periodic watchdog scan can create or update review work again.
|
||||
Operators should prefer `snooze` for known time-bounded quiet periods. `continue` is only a short acknowledgement of the current evidence. If the run remains silent after the re-arm window, the UI signal appears again.
|
||||
|
||||
The board can record watchdog decisions. The assigned owner of an issue-backed watchdog evaluation can also record them. Other agents cannot.
|
||||
The signal reappears in the UI after a snooze or continue window expires. No review work is created when it reappears. The board can record decisions without an evaluation issue. For compatibility, the assigned owner of an open legacy evaluation issue can also record a decision that is bound to that issue and run. Other agents cannot.
|
||||
|
||||
### Source-aware watchdog folding
|
||||
|
||||
Active-run watchdog work is source-aware. Before the watchdog creates, refreshes, escalates, or blocks on reviewer work, it must re-read the linked source issue and decide whether the watchdog signal is still about productive source work or only about stale run/process bookkeeping.
|
||||
The active-run cleanup scan is source-aware. It re-reads the linked source issue and decides whether a still-running handle represents productive source work or stale run/process bookkeeping. It does not create reviewer work.
|
||||
|
||||
Fold watchdog work when all of these are true:
|
||||
|
||||
|
|
@ -746,17 +745,17 @@ Fold watchdog work when all of these are true:
|
|||
- durable source activity from the same run proves the source issue reached that terminal disposition after the stale-run or output-silence evidence point
|
||||
- there is no independent evidence that the still-running or detached process is doing harmful work, still owns external cleanup that needs an operator decision, or needs a separate security/ownership review
|
||||
|
||||
Folding means resolving or cancelling the watchdog recovery action or issue-backed evaluation through the explicit recovery lifecycle. It must preserve the run id, source issue, detected silence or detached-process evidence, terminal source activity, decision reason, and best-effort process cleanup result. It must be idempotent for the `(companyId, runId, sourceIssueId)` signal and must not recursively recover the watchdog evaluation issue itself.
|
||||
Folding means finalizing the stale run and resolving any legacy watchdog recovery action or issue-backed evaluation through the explicit recovery lifecycle. It must preserve the run id, source issue, detected silence or detached-process evidence, terminal source activity, decision reason, and best-effort process cleanup result. It must be idempotent for the `(companyId, runId, sourceIssueId)` signal and must not recursively recover a watchdog evaluation issue itself.
|
||||
|
||||
Do not fold watchdog work only because the run is quiet. The watchdog must still create or continue reviewer work when:
|
||||
Do not fold a run only because it is quiet. Keep the informational signal visible when:
|
||||
|
||||
- the source issue is still `todo` or `in_progress`, because productive work may still be happening or stuck
|
||||
- the source issue remains `in_progress` after a successful run with no valid disposition, because the successful-run handoff path owns that bounded correction
|
||||
- the run terminated or disappeared while the source issue remains `in_progress` without a live path, because stranded assigned recovery owns that continuity repair
|
||||
- the source issue is terminal but there is no durable same-run terminal activity after the stale evidence point
|
||||
- there is independent evidence that the process may still be mutating external state, leaking resources, crossing company or ownership boundaries, or otherwise needs operator review
|
||||
- there is independent evidence that the process may still be mutating external state, leaking resources, crossing company or ownership boundaries, or otherwise needs an operator decision
|
||||
|
||||
In the normal non-terminal case, critical silence can still create issue-backed evaluation work and block the source issue when blocking is necessary for correctness. In the source-resolved case, a completed source issue should not acquire a new manager review or blocker merely because an old run handle stayed active; only real unresolved work should block work.
|
||||
In the normal non-terminal case, critical silence remains a UI signal and does not block the source issue. In the source-resolved case, a completed source issue does not acquire a new review or blocker merely because an old run handle stayed active. Only real unresolved work should block work.
|
||||
|
||||
This is distinct from productivity review. Productivity review asks whether an assigned source issue has unusual progression patterns, such as no-comment terminal-run streaks, long active duration, or high churn. Source-resolved watchdog folding asks whether a stale active-run signal outlived a source issue that already reached a valid terminal disposition. One does not substitute for the other.
|
||||
|
||||
|
|
@ -786,7 +785,6 @@ Examples:
|
|||
|
||||
- automatic stranded-work retry was already exhausted
|
||||
- a dependency graph has an invalid/uninvokable owner, unassigned blocker, or invalid review participant
|
||||
- an active run is silent past the watchdog threshold
|
||||
|
||||
The recovery action stays source-scoped by default. Stranded-task escalation is board-owned and records the cause, evidence, next action, source and return owner, `routingPolicy: board_escalation_no_takeover_v1`, and wake or monitor policy in the source thread/detail surface.
|
||||
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -34,7 +34,6 @@ import { visibleIssueCondition } from "../issue-visibility.js";
|
|||
import { forbidden, notFound } from "../../errors.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import { isPidAlive, isProcessGroupAlive, terminateLocalService } from "../local-service-supervisor.js";
|
||||
import { redactCurrentUserText } from "../../log-redaction.js";
|
||||
import { redactSensitiveText } from "../../redaction.js";
|
||||
import { isUniqueViolation } from "../../db-errors.js";
|
||||
import { logActivity } from "../activity-log.js";
|
||||
|
|
@ -55,7 +54,6 @@ import {
|
|||
} from "../issue-dependency-wakeups.js";
|
||||
import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js";
|
||||
import { isHeartbeatWakeOnDemandEnabled } from "../heartbeat-policy.js";
|
||||
import { getRunLogStore } from "../run-log-store.js";
|
||||
import {
|
||||
DEFAULT_MAX_SUCCESSFUL_RUN_HANDOFF_ATTEMPTS,
|
||||
FINISH_SUCCESSFUL_RUN_HANDOFF_REASON,
|
||||
|
|
@ -98,7 +96,6 @@ export const ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS = 60 * 60 * 1000;
|
|||
export const ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS = 4 * 60 * 60 * 1000;
|
||||
export const ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS = 30 * 60 * 1000;
|
||||
export const DEFAULT_LIVENESS_REESCALATION_COOLDOWN_MS = 60 * 60 * 1000;
|
||||
const ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES = 8 * 1024;
|
||||
const STRANDED_ISSUE_RECOVERY_ORIGIN_KIND = RECOVERY_ORIGIN_KINDS.strandedIssueRecovery;
|
||||
const STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND = RECOVERY_ORIGIN_KINDS.staleActiveRunEvaluation;
|
||||
const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext";
|
||||
|
|
@ -597,15 +594,6 @@ function agentUiLink(agent: { id: string; name: string | null } | null, prefix:
|
|||
return `[${agent.name ?? agent.id}](/${prefix}/agents/${agent.id})`;
|
||||
}
|
||||
|
||||
function formatDuration(ms: number | null) {
|
||||
if (ms === null) return "unknown";
|
||||
const minutes = Math.floor(ms / 60_000);
|
||||
if (minutes < 60) return `${minutes}m`;
|
||||
const hours = Math.floor(minutes / 60);
|
||||
const remainingMinutes = minutes % 60;
|
||||
return remainingMinutes > 0 ? `${hours}h ${remainingMinutes}m` : `${hours}h`;
|
||||
}
|
||||
|
||||
function formatIssueLinksForComment(relations: Array<{ identifier?: string | null }>) {
|
||||
const identifiers = [
|
||||
...new Set(
|
||||
|
|
@ -624,36 +612,6 @@ function formatIssueLinksForComment(relations: Array<{ identifier?: string | nul
|
|||
.join(", ");
|
||||
}
|
||||
|
||||
function unwrapDatabaseConflictError(error: unknown) {
|
||||
if (!error || typeof error !== "object") return null;
|
||||
|
||||
const candidate = error as {
|
||||
code?: string;
|
||||
constraint?: string;
|
||||
constraint_name?: string;
|
||||
message?: string;
|
||||
cause?: unknown;
|
||||
};
|
||||
|
||||
if (
|
||||
typeof candidate.code === "string" ||
|
||||
typeof candidate.constraint === "string" ||
|
||||
typeof candidate.constraint_name === "string"
|
||||
) {
|
||||
return candidate;
|
||||
}
|
||||
|
||||
const cause = candidate.cause;
|
||||
if (!cause || typeof cause !== "object") return candidate;
|
||||
|
||||
return cause as {
|
||||
code?: string;
|
||||
constraint?: string;
|
||||
constraint_name?: string;
|
||||
message?: string;
|
||||
};
|
||||
}
|
||||
|
||||
function isStrandedIssueRecoveryIssue(issue: Pick<typeof issues.$inferSelect, "originKind">) {
|
||||
return isStrandedIssueRecoveryOriginKind(issue.originKind);
|
||||
}
|
||||
|
|
@ -794,13 +752,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
const treeControlSvc = issueTreeControlService(db);
|
||||
const budgets = budgetService(db);
|
||||
const instanceSettings = instanceSettingsService(db);
|
||||
const runLogStore = getRunLogStore();
|
||||
let resolvedDependencyWakeBackstopCandidateCursor: string | null = null;
|
||||
|
||||
const getCurrentUserRedactionOptions = async () => ({
|
||||
enabled: (await instanceSettings.getGeneral()).censorUsernameInLogs,
|
||||
});
|
||||
|
||||
async function getAgent(agentId: string) {
|
||||
return db.select().from(agents).where(eq(agents.id, agentId)).then((rows) => rows[0] ?? null);
|
||||
}
|
||||
|
|
@ -1346,20 +1299,10 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
.then((rows) => rows[0]?.issuePrefix ?? "PAP");
|
||||
}
|
||||
|
||||
function staleActiveRunOriginFingerprint(companyId: string, runId: string) {
|
||||
return `stale_active_run:${companyId}:${runId}`;
|
||||
}
|
||||
|
||||
function isTerminalIssueStatus(status: string | null | undefined) {
|
||||
return status === "done" || status === "cancelled";
|
||||
}
|
||||
|
||||
function isRecoveryOriginIssue(issue: typeof issues.$inferSelect) {
|
||||
return Object.values(RECOVERY_ORIGIN_KINDS).includes(
|
||||
issue.originKind as typeof RECOVERY_ORIGIN_KINDS[keyof typeof RECOVERY_ORIGIN_KINDS],
|
||||
);
|
||||
}
|
||||
|
||||
function silenceStartedAtForRun(run: Pick<typeof heartbeatRuns.$inferSelect, "lastOutputAt" | "processStartedAt" | "startedAt" | "createdAt">) {
|
||||
return run.lastOutputAt ?? run.processStartedAt ?? run.startedAt ?? run.createdAt ?? null;
|
||||
}
|
||||
|
|
@ -1369,21 +1312,40 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
return startedAt ? Math.max(0, now.getTime() - startedAt.getTime()) : null;
|
||||
}
|
||||
|
||||
async function latestActiveOutputQuietUntilDecision(companyId: string, runId: string, now = new Date()) {
|
||||
const [row] = await db
|
||||
.select()
|
||||
.from(heartbeatRunWatchdogDecisions)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
||||
eq(heartbeatRunWatchdogDecisions.runId, runId),
|
||||
inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]),
|
||||
gt(heartbeatRunWatchdogDecisions.snoozedUntil, now),
|
||||
),
|
||||
)
|
||||
.orderBy(desc(heartbeatRunWatchdogDecisions.createdAt))
|
||||
.limit(1);
|
||||
return row ?? null;
|
||||
async function activeOutputDecisionState(companyId: string, runId: string, now = new Date()) {
|
||||
const [quietUntilRows, dismissedRows] = await Promise.all([
|
||||
db
|
||||
.select({
|
||||
decision: heartbeatRunWatchdogDecisions.decision,
|
||||
snoozedUntil: heartbeatRunWatchdogDecisions.snoozedUntil,
|
||||
})
|
||||
.from(heartbeatRunWatchdogDecisions)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
||||
eq(heartbeatRunWatchdogDecisions.runId, runId),
|
||||
inArray(heartbeatRunWatchdogDecisions.decision, ["snooze", "continue"]),
|
||||
gt(heartbeatRunWatchdogDecisions.snoozedUntil, now),
|
||||
),
|
||||
)
|
||||
.orderBy(desc(heartbeatRunWatchdogDecisions.createdAt))
|
||||
.limit(1),
|
||||
db
|
||||
.select({ id: heartbeatRunWatchdogDecisions.id })
|
||||
.from(heartbeatRunWatchdogDecisions)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
||||
eq(heartbeatRunWatchdogDecisions.runId, runId),
|
||||
eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"),
|
||||
),
|
||||
)
|
||||
.limit(1),
|
||||
]);
|
||||
return {
|
||||
dismissedFalsePositive: dismissedRows.length > 0,
|
||||
quietUntilDecision: quietUntilRows[0] ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
async function findOpenStaleRunEvaluation(companyId: string, runId: string) {
|
||||
|
|
@ -1392,9 +1354,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
id: issues.id,
|
||||
identifier: issues.identifier,
|
||||
status: issues.status,
|
||||
priority: issues.priority,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
updatedAt: issues.updatedAt,
|
||||
})
|
||||
.from(issues)
|
||||
.where(
|
||||
|
|
@ -1410,51 +1370,6 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
return row ?? null;
|
||||
}
|
||||
|
||||
// Returns a `done` stale-run evaluation issue for this run if one exists.
|
||||
// Used to detect when a reviewer closed an alert directly on the board without going through
|
||||
// the watchdog decision API — which would not leave a dismissed_false_positive decision record.
|
||||
//
|
||||
// Scoped to `done` only (not `cancelled`): cancellation is used by other system code paths
|
||||
// and does not imply a reviewer's "false positive" verdict. `done` is the explicit
|
||||
// board-close path used by reviewers acknowledging the alert. A cancelled evaluation is
|
||||
// allowed to re-fire on the next scan; if a reviewer wants permanent suppression they
|
||||
// should mark the alert done or record a watchdog decision.
|
||||
async function findClosedStaleRunEvaluation(companyId: string, runId: string) {
|
||||
const [row] = await db
|
||||
.select({ id: issues.id, identifier: issues.identifier, status: issues.status })
|
||||
.from(issues)
|
||||
.where(
|
||||
and(
|
||||
eq(issues.companyId, companyId),
|
||||
eq(issues.originKind, STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND),
|
||||
eq(issues.originId, runId),
|
||||
visibleIssueCondition(),
|
||||
eq(issues.status, "done"),
|
||||
),
|
||||
)
|
||||
.orderBy(desc(issues.updatedAt))
|
||||
.limit(1);
|
||||
return row ?? null;
|
||||
}
|
||||
|
||||
// Returns true when a reviewer has already dismissed this run's silence as a false positive.
|
||||
// Used to prevent re-filing after a deliberate close — while still allowing legitimate
|
||||
// re-arm after a "continue" decision's snooze window expires.
|
||||
async function hasDismissedFalsePositiveDecision(companyId: string, runId: string) {
|
||||
const [row] = await db
|
||||
.select({ id: heartbeatRunWatchdogDecisions.id })
|
||||
.from(heartbeatRunWatchdogDecisions)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
|
||||
eq(heartbeatRunWatchdogDecisions.runId, runId),
|
||||
eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"),
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
return row != null;
|
||||
}
|
||||
|
||||
async function buildRunOutputSilence(
|
||||
run: Pick<
|
||||
typeof heartbeatRuns.$inferSelect,
|
||||
|
|
@ -1462,21 +1377,24 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
>,
|
||||
now = new Date(),
|
||||
): Promise<RunOutputSilenceSummary> {
|
||||
const [quietUntilDecision, evaluation] = await Promise.all([
|
||||
latestActiveOutputQuietUntilDecision(run.companyId, run.id, now),
|
||||
const [decisionState, evaluation] = await Promise.all([
|
||||
activeOutputDecisionState(run.companyId, run.id, now),
|
||||
findOpenStaleRunEvaluation(run.companyId, run.id),
|
||||
]);
|
||||
const { dismissedFalsePositive, quietUntilDecision } = decisionState;
|
||||
const silenceStartedAt = silenceStartedAtForRun(run);
|
||||
const silenceAgeMs = run.status === "running" ? silenceAgeMsForRun(run, now) : null;
|
||||
const level = run.status !== "running"
|
||||
? "not_applicable"
|
||||
: quietUntilDecision
|
||||
? "snoozed"
|
||||
: (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS
|
||||
? "critical"
|
||||
: (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS
|
||||
? "suspicious"
|
||||
: "ok";
|
||||
: dismissedFalsePositive
|
||||
? "not_applicable"
|
||||
: quietUntilDecision
|
||||
? "snoozed"
|
||||
: (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS
|
||||
? "critical"
|
||||
: (silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS
|
||||
? "suspicious"
|
||||
: "ok";
|
||||
return {
|
||||
lastOutputAt: run.lastOutputAt ?? null,
|
||||
lastOutputSeq: run.lastOutputSeq ?? 0,
|
||||
|
|
@ -1488,37 +1406,13 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
level,
|
||||
suspicionThresholdMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS,
|
||||
criticalThresholdMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS,
|
||||
snoozedUntil: quietUntilDecision?.snoozedUntil ?? null,
|
||||
snoozedUntil: dismissedFalsePositive ? null : quietUntilDecision?.snoozedUntil ?? null,
|
||||
evaluationIssueId: evaluation?.id ?? null,
|
||||
evaluationIssueIdentifier: evaluation?.identifier ?? null,
|
||||
evaluationIssueAssigneeAgentId: evaluation?.assigneeAgentId ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function redactWatchdogEvidenceText(value: string, currentUserRedactionOptions: Awaited<ReturnType<typeof getCurrentUserRedactionOptions>>) {
|
||||
return redactSensitiveText(redactCurrentUserText(value, currentUserRedactionOptions));
|
||||
}
|
||||
|
||||
function truncateEvidenceText(value: string, maxChars = 4000) {
|
||||
if (value.length <= maxChars) return value;
|
||||
return `${value.slice(value.length - maxChars)}\n[truncated earlier evidence]`;
|
||||
}
|
||||
|
||||
async function readRunLogTailForEvidence(run: typeof heartbeatRuns.$inferSelect) {
|
||||
if (!run.logStore || !run.logRef || !run.logBytes) return "";
|
||||
try {
|
||||
const offset = Math.max(0, run.logBytes - ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES);
|
||||
const result = await runLogStore.read(
|
||||
{ store: run.logStore as "local_file", logRef: run.logRef },
|
||||
{ offset, limitBytes: ACTIVE_RUN_OUTPUT_EVIDENCE_TAIL_BYTES },
|
||||
);
|
||||
return result.content;
|
||||
} catch (err) {
|
||||
logger.warn({ err, runId: run.id }, "failed to read stale-run watchdog evidence tail");
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
async function resolveStaleRunSourceIssue(run: typeof heartbeatRuns.$inferSelect) {
|
||||
const issueId = issueIdFromRunContext(run.contextSnapshot);
|
||||
if (!issueId) return null;
|
||||
|
|
@ -1833,267 +1727,21 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
return { kind: "folded" as const, evaluationIssueId: input.existingEvaluation?.id ?? null };
|
||||
}
|
||||
|
||||
async function resolveStaleRunOwnerAgentId(input: {
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
runningAgent: typeof agents.$inferSelect;
|
||||
sourceIssue: typeof issues.$inferSelect | null;
|
||||
}) {
|
||||
const candidateIds: string[] = [];
|
||||
if (input.sourceIssue?.assigneeAgentId) {
|
||||
const sourceAssignee = await getAgent(input.sourceIssue.assigneeAgentId);
|
||||
if (sourceAssignee?.reportsTo) candidateIds.push(sourceAssignee.reportsTo);
|
||||
}
|
||||
if (input.runningAgent.reportsTo) candidateIds.push(input.runningAgent.reportsTo);
|
||||
const roleCandidates = await db
|
||||
.select()
|
||||
.from(agents)
|
||||
.where(and(eq(agents.companyId, input.run.companyId), inArray(agents.role, ["cto", "ceo"])))
|
||||
.orderBy(sql`case when ${agents.role} = 'cto' then 0 else 1 end`, asc(agents.createdAt));
|
||||
candidateIds.push(...roleCandidates.map((agent) => agent.id));
|
||||
|
||||
const seen = new Set<string>();
|
||||
for (const agentId of candidateIds) {
|
||||
if (seen.has(agentId)) continue;
|
||||
seen.add(agentId);
|
||||
const candidate = await getAgent(agentId);
|
||||
if (!candidate || candidate.companyId !== input.run.companyId) continue;
|
||||
const budgetBlock = await budgets.getInvocationBlock(input.run.companyId, candidate.id, {
|
||||
issueId: input.sourceIssue?.id ?? null,
|
||||
projectId: input.sourceIssue?.projectId ?? null,
|
||||
});
|
||||
if (
|
||||
(await isAgentInvokable(candidate)) &&
|
||||
isHeartbeatWakeOnDemandEnabled(candidate) &&
|
||||
!budgetBlock
|
||||
) {
|
||||
return candidate.id;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
async function collectStaleRunEvidence(input: {
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
runningAgent: typeof agents.$inferSelect;
|
||||
sourceIssue: typeof issues.$inferSelect | null;
|
||||
prefix: string;
|
||||
now: Date;
|
||||
}) {
|
||||
const [tail, recentEvents, childIssues, blockers] = await Promise.all([
|
||||
readRunLogTailForEvidence(input.run),
|
||||
db
|
||||
.select({
|
||||
eventType: heartbeatRunEvents.eventType,
|
||||
level: heartbeatRunEvents.level,
|
||||
message: heartbeatRunEvents.message,
|
||||
createdAt: heartbeatRunEvents.createdAt,
|
||||
})
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(eq(heartbeatRunEvents.companyId, input.run.companyId), eq(heartbeatRunEvents.runId, input.run.id)))
|
||||
.orderBy(desc(heartbeatRunEvents.id))
|
||||
.limit(8),
|
||||
input.sourceIssue
|
||||
? db
|
||||
.select({ id: issues.id, identifier: issues.identifier, title: issues.title, status: issues.status })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, input.run.companyId), eq(issues.parentId, input.sourceIssue.id), visibleIssueCondition()))
|
||||
.orderBy(desc(issues.updatedAt))
|
||||
.limit(8)
|
||||
: Promise.resolve([]),
|
||||
input.sourceIssue
|
||||
? db
|
||||
.select({ id: issues.id, identifier: issues.identifier, title: issues.title, status: issues.status })
|
||||
.from(issueRelations)
|
||||
.innerJoin(issues, eq(issueRelations.issueId, issues.id))
|
||||
.where(
|
||||
and(
|
||||
eq(issueRelations.companyId, input.run.companyId),
|
||||
eq(issueRelations.relatedIssueId, input.sourceIssue.id),
|
||||
eq(issueRelations.type, "blocks"),
|
||||
),
|
||||
)
|
||||
.limit(8)
|
||||
: Promise.resolve([]),
|
||||
]);
|
||||
const currentUserRedactionOptions = await getCurrentUserRedactionOptions();
|
||||
const safeTail = truncateEvidenceText(redactWatchdogEvidenceText(tail, currentUserRedactionOptions));
|
||||
const silenceAgeMs = silenceAgeMsForRun(input.run, input.now);
|
||||
return {
|
||||
safeTail,
|
||||
silenceAgeMs,
|
||||
recentEvents: recentEvents.reverse().map((event) => ({
|
||||
eventType: event.eventType,
|
||||
level: event.level,
|
||||
createdAt: event.createdAt.toISOString(),
|
||||
message: event.message ? truncateEvidenceText(redactWatchdogEvidenceText(event.message, currentUserRedactionOptions), 300) : null,
|
||||
})),
|
||||
childIssues,
|
||||
blockers,
|
||||
};
|
||||
}
|
||||
|
||||
function buildStaleRunEvaluationDescription(input: {
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
runningAgent: typeof agents.$inferSelect;
|
||||
sourceIssue: typeof issues.$inferSelect | null;
|
||||
prefix: string;
|
||||
evidence: Awaited<ReturnType<typeof collectStaleRunEvidence>>;
|
||||
level: "suspicious" | "critical";
|
||||
now: Date;
|
||||
}) {
|
||||
const sourceIssue = input.sourceIssue
|
||||
? issueUiLink({ identifier: input.sourceIssue.identifier, id: input.sourceIssue.id }, input.prefix)
|
||||
: "none";
|
||||
const recentEvents = input.evidence.recentEvents.length > 0
|
||||
? input.evidence.recentEvents.map((event) =>
|
||||
`- ${event.createdAt} \`${event.eventType}\`${event.level ? ` ${event.level}` : ""}: ${event.message ?? "(no message)"}`,
|
||||
).join("\n")
|
||||
: "- none";
|
||||
const childIssues = input.evidence.childIssues.length > 0
|
||||
? input.evidence.childIssues.map((issue) =>
|
||||
`- ${issueUiLink({ identifier: issue.identifier, id: issue.id }, input.prefix)} \`${issue.status}\`: ${issue.title}`,
|
||||
).join("\n")
|
||||
: "- none detected";
|
||||
const blockers = input.evidence.blockers.length > 0
|
||||
? input.evidence.blockers.map((issue) =>
|
||||
`- ${issueUiLink({ identifier: issue.identifier, id: issue.id }, input.prefix)} \`${issue.status}\`: ${issue.title}`,
|
||||
).join("\n")
|
||||
: "- none detected";
|
||||
return [
|
||||
`Paperclip detected ${input.level} output silence on an active heartbeat run.`,
|
||||
"",
|
||||
"## Run",
|
||||
"",
|
||||
`- Run: ${runUiLink(input.run, input.prefix)}`,
|
||||
`- Agent: ${input.runningAgent.name} (${input.runningAgent.adapterType})`,
|
||||
`- Invocation: ${input.run.invocationSource}${input.run.triggerDetail ? ` / ${input.run.triggerDetail}` : ""}`,
|
||||
`- Source issue: ${sourceIssue}`,
|
||||
`- Started at: ${input.run.startedAt?.toISOString() ?? "unknown"}`,
|
||||
`- Process started at: ${input.run.processStartedAt?.toISOString() ?? "unknown"}`,
|
||||
`- Last output at: ${input.run.lastOutputAt?.toISOString() ?? "none recorded"}`,
|
||||
`- Last output sequence: ${input.run.lastOutputSeq ?? 0}`,
|
||||
`- Silent for: ${formatDuration(input.evidence.silenceAgeMs)}`,
|
||||
`- Thresholds: suspicious after ${formatDuration(ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS)}, critical after ${formatDuration(ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS)}`,
|
||||
`- Process metadata: pid \`${input.run.processPid ?? "unknown"}\`, process group \`${input.run.processGroupId ?? "unknown"}\`, in-memory handle \`${runningProcesses.has(input.run.id) ? "yes" : "no"}\``,
|
||||
"",
|
||||
"## Last Output Excerpt",
|
||||
"",
|
||||
input.evidence.safeTail ? `\`\`\`text\n${input.evidence.safeTail}\n\`\`\`` : "_No run-log tail was available._",
|
||||
"",
|
||||
"## Recent Run Events",
|
||||
"",
|
||||
recentEvents,
|
||||
"",
|
||||
"## Related Work",
|
||||
"",
|
||||
"Active child issues:",
|
||||
childIssues,
|
||||
"",
|
||||
"Current source blockers:",
|
||||
blockers,
|
||||
"",
|
||||
"## Decision Checklist",
|
||||
"",
|
||||
"- Continue or snooze if the run is intentionally quiet.",
|
||||
"- Ask the run owner for context if work may be delegated outside the transcript.",
|
||||
"- Preserve artifacts, branch state, and useful output before cancellation.",
|
||||
"- Cancel or recover through the explicit run recovery controls when authorized.",
|
||||
"- Close this issue as a false positive only after recording the reason.",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function isUniqueStaleRunEvaluationConflict(error: unknown) {
|
||||
const maybe = unwrapDatabaseConflictError(error);
|
||||
if (!maybe) return false;
|
||||
return maybe.code === "23505" &&
|
||||
(
|
||||
maybe.constraint === "issues_active_stale_run_evaluation_uq" ||
|
||||
maybe.constraint_name === "issues_active_stale_run_evaluation_uq" ||
|
||||
typeof maybe.message === "string" && maybe.message.includes("issues_active_stale_run_evaluation_uq")
|
||||
);
|
||||
}
|
||||
|
||||
async function ensureSourceIssueCommentedForStaleEvaluation(input: {
|
||||
sourceIssue: typeof issues.$inferSelect | null;
|
||||
evaluationIssue: { id: string; identifier: string | null };
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
}) {
|
||||
if (!input.sourceIssue || ["done", "cancelled"].includes(input.sourceIssue.status)) return false;
|
||||
// Idempotency guard: if we've already emitted the escalation comment for this
|
||||
// (sourceIssue, evaluationIssue) pair, skip. Without this, every subsequent scan
|
||||
// cycle while the evaluation issue is still open re-fires the comment and spams
|
||||
// the source-issue thread. The activity log row written below is the persistence
|
||||
// record we check against — a single row per pair is enough to suppress repeats
|
||||
// even after process restarts.
|
||||
const [priorEscalation] = await db
|
||||
.select({ id: activityLog.id })
|
||||
.from(activityLog)
|
||||
.where(
|
||||
and(
|
||||
eq(activityLog.companyId, input.sourceIssue.companyId),
|
||||
eq(activityLog.action, "heartbeat.output_stale_escalated"),
|
||||
eq(activityLog.entityType, "issue"),
|
||||
eq(activityLog.entityId, input.sourceIssue.id),
|
||||
sql`${activityLog.details} ->> 'evaluationIssueId' = ${input.evaluationIssue.id}`,
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
if (priorEscalation) return false;
|
||||
// Evaluation issues are observability-only — do NOT add them to blockedByIssueIds.
|
||||
// They are already parented under the source issue. Adding them as hard blockers
|
||||
// creates a self-amplifying loop: block → silence → new alert → block again.
|
||||
await issuesSvc.addComment(input.sourceIssue.id, [
|
||||
"Paperclip detected critical output silence on this issue's active run.",
|
||||
"",
|
||||
`- Evaluation issue: ${input.evaluationIssue.identifier ?? input.evaluationIssue.id}`,
|
||||
`- Run: \`${input.run.id}\``,
|
||||
"",
|
||||
"Review the evaluation issue above. The active run has not been cancelled.",
|
||||
].join("\n"), { runId: input.run.id });
|
||||
await logActivity(db, {
|
||||
companyId: input.sourceIssue.companyId,
|
||||
actorType: "system",
|
||||
actorId: "system",
|
||||
agentId: null,
|
||||
runId: input.run.id,
|
||||
action: "heartbeat.output_stale_escalated",
|
||||
entityType: "issue",
|
||||
entityId: input.sourceIssue.id,
|
||||
details: {
|
||||
source: "recovery.scan_silent_active_runs",
|
||||
evaluationIssueId: input.evaluationIssue.id,
|
||||
},
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
async function createOrUpdateStaleRunEvaluation(input: {
|
||||
async function inspectSilentActiveRun(input: {
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
now: Date;
|
||||
dismissedFalsePositive: boolean;
|
||||
}) {
|
||||
const runningAgent = await getAgent(input.run.agentId);
|
||||
if (!runningAgent || runningAgent.companyId !== input.run.companyId) return { kind: "skipped" as const };
|
||||
const sourceIssue = await resolveStaleRunSourceIssue(input.run);
|
||||
const existing = await findOpenStaleRunEvaluation(input.run.companyId, input.run.id);
|
||||
if (sourceIssue && isRecoveryOriginIssue(sourceIssue)) {
|
||||
await logActivity(db, {
|
||||
companyId: input.run.companyId,
|
||||
actorType: "system",
|
||||
actorId: "system",
|
||||
agentId: input.run.agentId,
|
||||
runId: input.run.id,
|
||||
action: "heartbeat.output_stale_recovery_recursion_refused",
|
||||
entityType: "heartbeat_run",
|
||||
entityId: input.run.id,
|
||||
details: {
|
||||
source: "recovery.scan_silent_active_runs",
|
||||
sourceIssueId: sourceIssue.id,
|
||||
sourceIssueIdentifier: sourceIssue.identifier,
|
||||
sourceIssueOriginKind: sourceIssue.originKind,
|
||||
existingEvaluationIssueId: existing?.id ?? null,
|
||||
},
|
||||
});
|
||||
if (
|
||||
sourceIssue &&
|
||||
Object.values(RECOVERY_ORIGIN_KINDS).includes(
|
||||
sourceIssue.originKind as typeof RECOVERY_ORIGIN_KINDS[keyof typeof RECOVERY_ORIGIN_KINDS],
|
||||
)
|
||||
) {
|
||||
return { kind: "skipped" as const };
|
||||
}
|
||||
const silenceStartedAt = silenceStartedAtForRun(input.run);
|
||||
|
|
@ -2117,184 +1765,17 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
}
|
||||
}
|
||||
|
||||
// Idle output is expected when the source issue is blocked — skip ticket creation entirely.
|
||||
// Blocked source work can be intentionally quiet. The issue state already carries
|
||||
// the durable waiting signal, so the cleanup scan has nothing to do.
|
||||
if (sourceIssue?.status === "blocked") return { kind: "skipped" as const };
|
||||
|
||||
// Dedup: if a reviewer has dismissed this run's silence as a false positive, don't re-file.
|
||||
// A "continue" decision with a snooze window is allowed to re-arm normally — only an
|
||||
// explicit dismissed_false_positive blocks all further alerts for this run.
|
||||
if (await hasDismissedFalsePositiveDecision(input.run.companyId, input.run.id)) {
|
||||
if (input.dismissedFalsePositive) {
|
||||
return { kind: "skipped" as const };
|
||||
}
|
||||
|
||||
// Dedup: if a prior evaluation issue for this run was closed `done` on the board
|
||||
// without going through the watchdog decision API, no dismissed_false_positive record exists
|
||||
// and the watchdog would re-fire every cycle. Auto-record the suppression now so future
|
||||
// cycles skip immediately via hasDismissedFalsePositiveDecision.
|
||||
//
|
||||
// Exception: if any watchdog decision exists (snooze/continue), a human explicitly opted
|
||||
// in to the watchdog lifecycle — honour that and allow re-arm as designed.
|
||||
//
|
||||
// Concurrency: the check-then-insert runs inside a transaction with a per-(company,run)
|
||||
// advisory lock so two overlapping scans cannot both observe `hasAnyDecision = false`
|
||||
// and both insert a dismissed_false_positive row. The table has no unique constraint
|
||||
// on (companyId, runId, decision), so the advisory lock is the serialization point.
|
||||
const closedEvaluation = await findClosedStaleRunEvaluation(input.run.companyId, input.run.id);
|
||||
if (closedEvaluation) {
|
||||
const autoDismissed = await db.transaction(async (tx) => {
|
||||
await tx.execute(
|
||||
sql`SELECT pg_advisory_xact_lock(hashtextextended(${`watchdog_dismiss:${input.run.companyId}:${input.run.id}`}, 0))`,
|
||||
);
|
||||
const hasAnyDecision = await tx
|
||||
.select({ id: heartbeatRunWatchdogDecisions.id })
|
||||
.from(heartbeatRunWatchdogDecisions)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRunWatchdogDecisions.companyId, input.run.companyId),
|
||||
eq(heartbeatRunWatchdogDecisions.runId, input.run.id),
|
||||
),
|
||||
)
|
||||
.limit(1)
|
||||
.then((rows) => rows.length > 0);
|
||||
if (hasAnyDecision) return false;
|
||||
await tx.insert(heartbeatRunWatchdogDecisions).values({
|
||||
companyId: input.run.companyId,
|
||||
runId: input.run.id,
|
||||
evaluationIssueId: closedEvaluation.id,
|
||||
decision: "dismissed_false_positive",
|
||||
snoozedUntil: null,
|
||||
reason: `Auto-recorded: evaluation issue ${closedEvaluation.identifier} was closed as ${closedEvaluation.status} on the board without a watchdog decision.`,
|
||||
createdByAgentId: null,
|
||||
createdByUserId: null,
|
||||
createdByRunId: null,
|
||||
});
|
||||
return true;
|
||||
});
|
||||
if (autoDismissed) {
|
||||
return { kind: "skipped" as const };
|
||||
}
|
||||
}
|
||||
|
||||
const prefix = await getCompanyIssuePrefix(input.run.companyId);
|
||||
const evidence = await collectStaleRunEvidence({
|
||||
run: input.run,
|
||||
runningAgent,
|
||||
sourceIssue,
|
||||
prefix,
|
||||
now: input.now,
|
||||
});
|
||||
const level = (evidence.silenceAgeMs ?? 0) >= ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS ? "critical" : "suspicious";
|
||||
if (existing) {
|
||||
if (level === "critical" && existing.priority !== "high") {
|
||||
await issuesSvc.update(existing.id, {
|
||||
priority: "high",
|
||||
});
|
||||
await issuesSvc.addComment(existing.id, [
|
||||
"Critical output silence threshold crossed.",
|
||||
"",
|
||||
`- Run: \`${input.run.id}\``,
|
||||
`- Silent for: ${formatDuration(evidence.silenceAgeMs)}`,
|
||||
`- Last output at: ${input.run.lastOutputAt?.toISOString() ?? "none recorded"}`,
|
||||
].join("\n"), { runId: input.run.id });
|
||||
await ensureSourceIssueCommentedForStaleEvaluation({
|
||||
sourceIssue,
|
||||
evaluationIssue: existing,
|
||||
run: input.run,
|
||||
});
|
||||
return { kind: "escalated" as const, evaluationIssueId: existing.id };
|
||||
}
|
||||
if (level === "critical") {
|
||||
await ensureSourceIssueCommentedForStaleEvaluation({
|
||||
sourceIssue,
|
||||
evaluationIssue: existing,
|
||||
run: input.run,
|
||||
});
|
||||
}
|
||||
return { kind: "existing" as const, evaluationIssueId: existing.id };
|
||||
}
|
||||
|
||||
const ownerAgentId = await resolveStaleRunOwnerAgentId({ run: input.run, runningAgent, sourceIssue });
|
||||
const description = buildStaleRunEvaluationDescription({
|
||||
run: input.run,
|
||||
runningAgent,
|
||||
sourceIssue,
|
||||
prefix,
|
||||
evidence,
|
||||
level,
|
||||
now: input.now,
|
||||
});
|
||||
let evaluation: Awaited<ReturnType<typeof issuesSvc.create>>;
|
||||
try {
|
||||
evaluation = await issuesSvc.create(input.run.companyId, {
|
||||
title: `Review silent active run for ${runningAgent.name}`,
|
||||
description,
|
||||
status: "todo",
|
||||
priority: level === "critical" ? "high" : "medium",
|
||||
parentId: sourceIssue && !["done", "cancelled"].includes(sourceIssue.status) ? sourceIssue.id : null,
|
||||
projectId: sourceIssue?.projectId ?? null,
|
||||
goalId: sourceIssue?.goalId ?? null,
|
||||
billingCode: sourceIssue?.billingCode ?? null,
|
||||
assigneeAgentId: ownerAgentId,
|
||||
assigneeAdapterOverrides: recoveryAssigneeAdapterOverrides("status_only"),
|
||||
originKind: STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND,
|
||||
originId: input.run.id,
|
||||
originRunId: input.run.id,
|
||||
originFingerprint: staleActiveRunOriginFingerprint(input.run.companyId, input.run.id),
|
||||
});
|
||||
} catch (error) {
|
||||
if (!isUniqueStaleRunEvaluationConflict(error)) throw error;
|
||||
const raced = await findOpenStaleRunEvaluation(input.run.companyId, input.run.id);
|
||||
if (!raced) throw error;
|
||||
return { kind: "existing" as const, evaluationIssueId: raced.id };
|
||||
}
|
||||
|
||||
await logActivity(db, {
|
||||
companyId: input.run.companyId,
|
||||
actorType: "system",
|
||||
actorId: "system",
|
||||
agentId: ownerAgentId,
|
||||
runId: input.run.id,
|
||||
action: "heartbeat.output_stale_detected",
|
||||
entityType: "issue",
|
||||
entityId: evaluation.id,
|
||||
details: {
|
||||
source: "recovery.scan_silent_active_runs",
|
||||
level,
|
||||
sourceIssueId: sourceIssue?.id ?? null,
|
||||
silenceAgeMs: evidence.silenceAgeMs,
|
||||
lastOutputAt: input.run.lastOutputAt?.toISOString() ?? null,
|
||||
},
|
||||
});
|
||||
if (level === "critical") {
|
||||
await ensureSourceIssueCommentedForStaleEvaluation({
|
||||
sourceIssue,
|
||||
evaluationIssue: evaluation,
|
||||
run: input.run,
|
||||
});
|
||||
}
|
||||
if (ownerAgentId) {
|
||||
await deps.enqueueWakeup(ownerAgentId, {
|
||||
source: "assignment",
|
||||
triggerDetail: "system",
|
||||
reason: "issue_assigned",
|
||||
payload: withRecoveryModelProfileHint({
|
||||
issueId: evaluation.id,
|
||||
staleRunId: input.run.id,
|
||||
sourceIssueId: sourceIssue?.id ?? null,
|
||||
}, "status_only"),
|
||||
requestedByActorType: "system",
|
||||
requestedByActorId: null,
|
||||
contextSnapshot: withRecoveryModelProfileHint({
|
||||
issueId: evaluation.id,
|
||||
taskId: evaluation.id,
|
||||
wakeReason: "issue_assigned",
|
||||
source: STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND,
|
||||
staleRunId: input.run.id,
|
||||
sourceIssueId: sourceIssue?.id ?? null,
|
||||
}, "status_only"),
|
||||
});
|
||||
}
|
||||
return { kind: "created" as const, evaluationIssueId: evaluation.id };
|
||||
return existing
|
||||
? { kind: "existing" as const, evaluationIssueId: existing.id }
|
||||
: { kind: "skipped" as const };
|
||||
}
|
||||
|
||||
async function scanSilentActiveRuns(opts?: { now?: Date; companyId?: string; issueCreatedAtGte?: Date | null }) {
|
||||
|
|
@ -2346,14 +1827,17 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
|||
};
|
||||
|
||||
for (const run of candidates) {
|
||||
if (await latestActiveOutputQuietUntilDecision(run.companyId, run.id, now)) {
|
||||
const decisionState = await activeOutputDecisionState(run.companyId, run.id, now);
|
||||
if (decisionState.quietUntilDecision) {
|
||||
result.snoozed += 1;
|
||||
continue;
|
||||
}
|
||||
const outcome = await createOrUpdateStaleRunEvaluation({ run, now });
|
||||
if (outcome.kind === "created") result.created += 1;
|
||||
else if (outcome.kind === "existing") result.existing += 1;
|
||||
else if (outcome.kind === "escalated") result.escalated += 1;
|
||||
const outcome = await inspectSilentActiveRun({
|
||||
run,
|
||||
now,
|
||||
dismissedFalsePositive: decisionState.dismissedFalsePositive,
|
||||
});
|
||||
if (outcome.kind === "existing") result.existing += 1;
|
||||
else if (outcome.kind === "folded") result.folded += 1;
|
||||
else result.skipped += 1;
|
||||
if ("evaluationIssueId" in outcome && outcome.evaluationIssueId) {
|
||||
|
|
|
|||
|
|
@ -448,7 +448,7 @@ describe("IssueRunLedger", () => {
|
|||
expect(container.textContent).toContain("2 older items not shown");
|
||||
});
|
||||
|
||||
it("renders stale-run banner, watchdog actions, and silence badge for live runs", () => {
|
||||
it("renders legacy evaluation context with watchdog actions and a silence badge", () => {
|
||||
const onWatchdogDecision = vi.fn();
|
||||
renderLedger({
|
||||
runs: [createRun({ runId: "run-live-1", status: "running", finishedAt: null })],
|
||||
|
|
@ -456,11 +456,12 @@ describe("IssueRunLedger", () => {
|
|||
onWatchdogDecision,
|
||||
});
|
||||
|
||||
expect(container.textContent).toContain("Stale-run watchdog alert");
|
||||
expect(container.textContent).toContain("Critical output silence");
|
||||
expect(container.textContent).toContain("PAP-404");
|
||||
expect(container.textContent).toContain("Stale run");
|
||||
expect(container.textContent).toContain("Critical silence");
|
||||
expect(container.textContent).toContain("Paperclip did not create new delegated recovery work");
|
||||
const watchdogBanner = Array.from(container.querySelectorAll("p"))
|
||||
.find((node) => node.textContent?.includes("Stale-run watchdog alert"))
|
||||
.find((node) => node.textContent?.includes("Critical output silence"))
|
||||
?.closest("div");
|
||||
expect(watchdogBanner?.className).toContain("border-red-500/30");
|
||||
expect(watchdogBanner?.className).toContain("bg-red-500/10");
|
||||
|
|
@ -479,6 +480,56 @@ describe("IssueRunLedger", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
level: "suspicious" as const,
|
||||
heading: "Output silence watchdog warning",
|
||||
badge: "Output silence",
|
||||
},
|
||||
{
|
||||
level: "critical" as const,
|
||||
heading: "Critical output silence",
|
||||
badge: "Critical silence",
|
||||
},
|
||||
])("renders a $level UI-only signal without an evaluation-task link", ({ level, heading, badge }) => {
|
||||
const onWatchdogDecision = vi.fn();
|
||||
const activeRun = createActiveRun();
|
||||
renderLedger({
|
||||
runs: [createRun({ runId: activeRun.id, status: "running", finishedAt: null })],
|
||||
activeRun: createActiveRun({
|
||||
outputSilence: {
|
||||
...activeRun.outputSilence!,
|
||||
level,
|
||||
evaluationIssueId: null,
|
||||
evaluationIssueIdentifier: null,
|
||||
evaluationIssueAssigneeAgentId: null,
|
||||
},
|
||||
}),
|
||||
onWatchdogDecision,
|
||||
});
|
||||
|
||||
expect(container.textContent).toContain(heading);
|
||||
expect(container.textContent).toContain(badge);
|
||||
expect(container.textContent).toContain("Paperclip did not create or assign a recovery task");
|
||||
expect(container.textContent).not.toContain("PAP-404");
|
||||
expect(container.querySelector('a[href^="/issues/"]')).toBeNull();
|
||||
expect(container.textContent).toContain("Continue monitoring");
|
||||
expect(container.textContent).toContain("Snooze 1h");
|
||||
expect(container.textContent).toContain("Mark false positive");
|
||||
|
||||
const continueButton = Array.from(container.querySelectorAll("button")).find(
|
||||
(button) => button.textContent?.includes("Continue monitoring"),
|
||||
);
|
||||
act(() => {
|
||||
continueButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
});
|
||||
expect(onWatchdogDecision).toHaveBeenCalledWith({
|
||||
runId: "run-live-1",
|
||||
decision: "continue",
|
||||
evaluationIssueId: null,
|
||||
});
|
||||
});
|
||||
|
||||
it("renders requested/applied model profile and surfaces fallback reasons", () => {
|
||||
renderLedger({
|
||||
runs: [
|
||||
|
|
@ -523,7 +574,7 @@ describe("IssueRunLedger", () => {
|
|||
onWatchdogDecision,
|
||||
});
|
||||
|
||||
expect(container.textContent).toContain("Stale-run watchdog alert");
|
||||
expect(container.textContent).toContain("Critical output silence");
|
||||
expect(container.textContent).toContain("PAP-404");
|
||||
expect(container.textContent).not.toContain("Continue monitoring");
|
||||
expect(container.textContent).not.toContain("Snooze 1h");
|
||||
|
|
|
|||
|
|
@ -143,11 +143,11 @@ type RunOutputSilenceCopy = {
|
|||
|
||||
const RUN_OUTPUT_SILENCE_COPY: Partial<Record<RunOutputSilenceLevel, RunOutputSilenceCopy>> = {
|
||||
suspicious: {
|
||||
label: "Silence watch",
|
||||
label: "Output silence",
|
||||
tone: "border-amber-500/30 bg-amber-500/10 text-amber-700 dark:text-amber-300",
|
||||
},
|
||||
critical: {
|
||||
label: "Stale run",
|
||||
label: "Critical silence",
|
||||
tone: "border-red-500/30 bg-red-500/10 text-red-700 dark:text-red-300",
|
||||
},
|
||||
snoozed: {
|
||||
|
|
@ -609,7 +609,7 @@ export function IssueRunLedgerContent({
|
|||
>
|
||||
<p className="font-medium">
|
||||
{latestSilentRun.outputSilence.level === "critical"
|
||||
? "Stale-run watchdog alert"
|
||||
? "Critical output silence"
|
||||
: "Output silence watchdog warning"}
|
||||
</p>
|
||||
<p className="mt-1">
|
||||
|
|
@ -629,6 +629,11 @@ export function IssueRunLedgerContent({
|
|||
</>
|
||||
) : null}
|
||||
</p>
|
||||
<p className="mt-1">
|
||||
{latestSilentRun.outputSilence.evaluationIssueIdentifier
|
||||
? "This signal is informational. Paperclip did not create new delegated recovery work."
|
||||
: "This signal is informational. Paperclip did not create or assign a recovery task."}
|
||||
</p>
|
||||
{onWatchdogDecision && canRecordWatchdogDecisions ? (
|
||||
<div className="mt-2 flex flex-wrap gap-1.5">
|
||||
<button
|
||||
|
|
|
|||
Loading…
Reference in New Issue