History

Dotta 555391fed7 fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in heartbeat runs orchestrated by the server; run lifecycle, retry scheduling, and the dashboard's run-activity metrics are the subsystems involved > - A spike in "failed" tasks traced to three causes: server restarts killing in-flight runs and mislabeling them as failures, deterministic workspace-validation loops when a worktree's branch diverged, and provider quota/usage-limit errors being classified as generic transient failures (putting agents into error state and polluting metrics) > - Killed-then-recovered runs and quota waits are not product failures, so both the runtime behavior and the reporting needed to distinguish them > - This pull request drains runs gracefully on shutdown with idempotent restart retries, self-heals workspace branch mismatches, adds a quota-aware failure class with reset-time retry, separates recovered restart kills from true failures on the dashboard, and documents restart hygiene for operators > - The benefit is fewer spurious failures, automatic recovery instead of manual repair, and dashboard metrics that reflect real failure rates ## Linked Issues or Issue Description No public GitHub issue exists; describing the bug inline per the bug-report template: What happened? In-flight heartbeat runs are marked `failed` when the server restarts, even though a retry later succeeds. Worktrees whose checked-out branch diverges from the issue branch fail workspace validation on every subsequent run with no recovery path. Provider quota/usage-limit responses are treated as generic transient upstream errors, putting agents into an error state and retrying before the quota window resets. The dashboard counts all of these as true failures, inflating failure metrics. Expected behavior Graceful shutdown should interrupt (not fail) running runs and chain exactly one recovery retry. Workspace validation should repair recoverable branch mismatches automatically. Quota errors should get their own error class with the retry scheduled at the provider reset time and the agent left idle. The dashboard should report recovered restart kills separately from true failures. Steps to reproduce 1. Start a heartbeat run, then restart the server (SIGTERM) while it is in flight — the run lands as `failed` with a process-loss error code even when its retry succeeds 2. Check out an issue whose worktree branch has diverged (e.g. after a force-moved branch) — every subsequent run fails `workspace_validation_failed` deterministically 3. Drive an agent into a provider usage-limit window — the run fails as a generic transient upstream error and the agent enters an error state instead of idling until the reset time Paperclip version or commit master (base `c07e650cd`) Deployment mode Self-hosted dev plane (Linux, node server + embedded Postgres) ## What Changed - Graceful shutdown (SIGTERM hook) now marks in-flight runs `interrupted` instead of `failed` and enqueues an idempotent process-loss retry (pre-insert existence check on `retryOfRunId` prevents duplicates; bursts chain exactly one retry per interrupted run) - Run-liveness classification routes `interrupted` to `needs_followup` rather than `failed` - Workspace validation self-heals branch mismatch / missing-branch states instead of failing deterministically on every run - New `provider_quota` error class: session/usage-limit responses schedule the retry at the provider reset time and leave the agent idle (not errored); fixes a case where a quota-terminated run with subtype `success` was misclassified as failed; HTTP 529 remains transient - Dashboard run-activity query separates recovered restart kills from true failures via a recursive CTE over `retry_of_run_id` (ancestors of a succeeded retry count as recovered), adds a per-day failed-by-error-code breakdown, and binds the window start as a timestamptz string - Activity charts UI: amber "Recovered" segment with legend and per-day error-code tooltip; success-rate chart counts recovered runs as successes - New ops runbook: `docs/deploy/dev-plane-restart-hygiene.md` ## Verification - Greptile follow-up fixes on `992705edd`: `pnpm exec vitest run server/src/__tests__/server-startup-feedback-export.test.ts server/src/__tests__/heartbeat-workspace-branch-containment.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` (209 tests); `pnpm --filter @paperclipai/server typecheck`; `git diff --check` - Post-rebase CI fixes: `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts`; `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/workspace-runtime.test.ts server/src/__tests__/heartbeat-workspace-branch-containment.test.ts server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts` (227 tests); `pnpm --filter @paperclipai/server typecheck` - `npm test` server suites covering the changes: `heartbeat-process-recovery`, `heartbeat-stop-metadata`, `heartbeat-retry-scheduling` (95 tests), quota parse + execute/retry-scheduling suites (100 tests), workspace self-heal suites (200 tests), dashboard run-activity tests (3 tests) — all green, typecheck exit 0 - Dashboard CTE cross-checked against a real development database: two restart-burst days moved from 17 to 8 and 19 to 6 true failures once recovered kills were separated, matching manual retry-chain inspection - Screenshot verification of the real ActivityCharts component (recovered segment + tooltip) during QA ## Risks - Behavioral shift: runs killed by a restart no longer surface as `failed`; anyone consuming raw run statuses will see `interrupted` (new status value) — dashboards/queries in this repo were updated accordingly - Retry chaining on repeated restarts is bounded (one chained retry per interruption) but a pathological restart loop still delays work rather than failing it; the runbook covers operator hygiene for that case - Dashboard query adds a recursive CTE; cost is bounded by the day-window row count and was verified against production-sized data - No schema migrations; low migration risk ## Model Used - Claude (Anthropic) — claude-fable-5 via Claude Code / Paperclip agent harness, extended thinking with tool use; implementation commits also produced with Codex CLI (GPT-5 class) agents under the same harness ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>		2026-07-07 20:30:09 -05:00
..
src	fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183 )	2026-07-07 20:30:09 -05:00
CHANGELOG.md	chore: release v0.3.1	2026-03-12 13:09:22 -05:00
README.md	PAPA-430: workspace finalize gates + no-remote-git enforcement (#6969 )	2026-05-29 08:25:29 -07:00
package.json	build(deps-dev): bump @types/node from 22.19.11 to 22.19.21 (#7748 )	2026-06-18 15:46:06 -07:00
tsconfig.json	Fix root TypeScript solution config	2026-03-09 14:09:30 -05:00

README.md

@paperclipai/adapter-utils

Shared utilities for Paperclip adapters: process spawning, environment injection, sandbox/SSH transport, workspace sync, and the round-trip helpers that move code between the local execution-workspace cwd and wherever the agent actually runs.

For the adapter-author guide see docs/adapters/creating-an-adapter.md and the in-repo notes at packages/adapters/AUTHORING.md.

No-remote-git contract

The local execution-workspace cwd is the only persistence boundary across runs. No adapter may depend on a git remote for cross-run state.

Adapters that run the agent on a different host should use the SSH round-trip helpers in src/ssh.ts:

prepareWorkspaceForSshExecution({ spec, localDir, remoteDir }) — bundles the local cwd (tracked files, dirty edits, untracked additions, and the git history needed to reconstruct it) to remoteDir before the run starts. Runs with no git remote configured.
restoreWorkspaceFromSshExecution({ spec, localDir, remoteDir, ... }) — syncs the remote cwd back into localDir after the run, including any new commits the agent created. Also runs with no git remote configured.

prepareRemoteManagedRuntime in src/remote-managed-runtime.ts wraps both calls for adapters that want a per-run remote workspace and an automatic restoreWorkspace() finally hook.

The invariant is pinned by the no-remote-git contract case in src/ssh-fixture.test.ts, which asserts that a remote-only commit propagates to the local worktree through the prepare → restore round-trip with no git remote configured at any point. Do not regress that test.