From 300a89ec136d76f4168e739578ddf144bc91ca2f Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Mon, 31 Aug 2026 13:40:11 -0700 Subject: [PATCH] Detect the qualifier-less Claude usage-limit message in quota classification (#12475) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runtime classifies adapter run failures, and the recovery service uses that classification to decide between automatic retry, a timed provider-quota wait, and a board escalation > - The Claude CLI changed its subscription-limit stop message to "You've hit your limit · resets 2:30am (UTC)", and no quota matcher knows this qualifier-less wording > - A limit-hit run therefore classifies as `adapter_failed` (or `claude_auth_required`), recovery burns its continuation retries against a hard limit, and the issue blocks with the opaque "No live execution path" notice instead of waiting for the reset and retrying automatically > - This pull request teaches the adapter and the recovery service the new wording, and titles stranded-escalation notices from the classified run error code so operators see the cause at a glance > - The benefit is that usage-limit stops self-heal at the provider reset time, and the notices that do post say "Error: usage limit reached" or "Error: not logged in to Claude" instead of a generic title ## Linked Issues or Issue Description No public issue exists; the underlying problem follows the bug template: **What happened?** On a staging deployment, an assigned `in_progress` issue hit the Claude subscription usage limit. The run recorded the error `Claude run failed: subtype=success: You've hit your limit · resets 2:30am (UTC)`. The automatic continuation retry failed the same way in 34 seconds with `errorCode: adapter_failed`. Terminal-run recovery then escalated: the issue moved to `blocked` with the notice "No live execution path" and a board-owned recovery action. The notice gave the operator no indication that the cause was a usage limit with a known reset time. **Expected behavior** A usage-limit stop classifies as `provider_quota` with the reset clock parsed into `retryNotBefore`. The recovery service takes its provider-quota wait path: a system-owned recovery action that waits for the reset time and retries the original assignee automatically. If an escalation notice does post, its title names the classified cause. **Steps to reproduce** 1. Run a `claude_local` agent on an issue until the Claude subscription limit is hit, so the CLI result is "You've hit your limit · resets \ (UTC)". 2. Let terminal-run recovery retry the continuation. 3. Observe the issue block with the "No live execution path" notice instead of a timed quota wait. `classifyAdapterFailureForRecovery` returns `null` for the recorded error text; `CLAUDE_PROVIDER_QUOTA_RE` and `PROVIDER_QUOTA_ERROR_RE` both fail to match it. ## What Changed - `CLAUDE_PROVIDER_QUOTA_RE` and `CLAUDE_EXTRA_USAGE_RESET_RE` (claude-local adapter) accept "you've hit your limit" with no qualifier, alongside the existing "session"/"usage" wordings, so the run classifies as `provider_quota` and the reset clock lands in `retryNotBefore`. - `PROVIDER_QUOTA_ERROR_RE` and `isProviderQuotaRecovery` (recovery service) accept the same wording, so runs recorded before the adapter fix (errorCode `adapter_failed` with the limit text in the error) also route to the quota wait. - `parseProviderQuotaClockReset` parses the "resets 2:30am (UTC)" clock shape alongside the existing "try again at" shape. - `buildStrandedRecoveryEscalationNotice` titles the notice from the source run's classified error code when one is mapped: `provider_quota` → "Error: usage limit reached", `claude_auth_required` → "Error: not logged in to Claude", `acpx_auth_required` → "Error: agent login required". The raw failure text stays withheld from the issue thread; only the server-classified code is surfaced. Unmapped codes keep the existing seed/cause titles. ## Verification - `pnpm vitest run packages/adapters/claude-local/src/server/parse.test.ts server/src/services/recovery/provider-failure-classification.test.ts server/src/services/recovery/stranded-notice.test.ts` — 72 tests pass, including 5 new cases that use the exact new CLI message. - `pnpm vitest run server/src/__tests__/issue-recovery-actions.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts` — 189 tests pass (no reroute regressions from the widened matchers). - `tsc --noEmit` clean for `@paperclipai/adapter-claude-local` and `@paperclipai/server`. ## Risks - Low risk. The regex widenings are additive; every previously matched wording still matches, and the existing negative test ("Workspace storage capacity limit reached." stays unclassified) still passes. - Behavioral shift, intended: an `adapter_failed` run whose error text is the new limit wording now routes to the silent system-owned quota wait instead of a board escalation. This matches how the older limit wordings already behave. - The notice title change only affects escalations whose source run carries one of the three mapped error codes; all other notices render exactly as before. ## Model Used Claude Fable 5 (`claude-fable-5`, Claude Code CLI, extended thinking with tool use). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- .../claude-local/src/server/parse.test.ts | 12 +++++++ .../adapters/claude-local/src/server/parse.ts | 4 +-- .../provider-failure-classification.test.ts | 16 ++++++++++ server/src/services/recovery/service.ts | 6 ++-- .../services/recovery/stranded-notice.test.ts | 31 +++++++++++++++++++ .../src/services/recovery/stranded-notice.ts | 15 ++++++++- 6 files changed, 78 insertions(+), 6 deletions(-) diff --git a/packages/adapters/claude-local/src/server/parse.test.ts b/packages/adapters/claude-local/src/server/parse.test.ts index 13b4964b99..5b291d4056 100644 --- a/packages/adapters/claude-local/src/server/parse.test.ts +++ b/packages/adapters/claude-local/src/server/parse.test.ts @@ -199,6 +199,18 @@ describe("isClaudeTransientUpstreamError", () => { ); }); + it("classifies the qualifier-less limit wording as provider quota and extracts the retry time", () => { + // Current Claude CLI phrasing: no "session"/"usage" qualifier before "limit". + const now = new Date("2026-08-28T22:30:00.000Z"); + const errorMessage = "You've hit your limit · resets 2:30am (UTC)"; + + expect(isClaudeProviderQuotaError({ errorMessage })).toBe(true); + expect(isClaudeTransientUpstreamError({ errorMessage })).toBe(false); + expect(extractClaudeRetryNotBefore({ errorMessage }, now)?.toISOString()).toBe( + "2026-08-29T02:30:00.000Z", + ); + }); + it("classifies Anthropic API rate_limit_error and overloaded_error as transient", () => { expect( isClaudeTransientUpstreamError({ diff --git a/packages/adapters/claude-local/src/server/parse.ts b/packages/adapters/claude-local/src/server/parse.ts index a3f3d95bec..3e636ac2cf 100644 --- a/packages/adapters/claude-local/src/server/parse.ts +++ b/packages/adapters/claude-local/src/server/parse.ts @@ -23,11 +23,11 @@ const URL_RE = /(https?:\/\/[^\s'"`<>()[\]{};,!?]+[^\s'"`<>()[\]{};,!.?:]+)/gi; const CLAUDE_TRANSIENT_UPSTREAM_RE = /(?:rate[-\s]?limit(?:ed)?|rate_limit_error|too\s+many\s+requests|\b429\b|overloaded(?:_error)?|server\s+overloaded|service\s+unavailable|\b503\b|\b529\b|high\s+demand|try\s+again\s+later|temporarily\s+unavailable|throttl(?:ed|ing)|throttlingexception|servicequotaexceededexception|out\s+of\s+extra\s+usage|extra\s+usage\b|claude\s+usage\s+limit\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|usage\s+limit\s+reached|usage\s+cap\s+reached)/i; const CLAUDE_PROVIDER_QUOTA_RE = - /(?:you(?:'|’)ve\s+hit\s+your\s+session\s+limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage\b|claude\s+usage\s+limit\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|usage\s+limit\s+reached|usage\s+cap\s+reached|servicequotaexceededexception)/i; + /(?:you(?:'|’)ve\s+hit\s+your\s+(?:\w+\s+)?limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage\b|claude\s+usage\s+limit\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|usage\s+limit\s+reached|usage\s+cap\s+reached|servicequotaexceededexception)/i; const CLAUDE_MODEL_NOT_FOUND_RE = /(?:\b404\b[\s\S]{0,120})?(?:model[\s_-]*(?:not[\s_-]*found|does not exist|unknown|invalid)|unknown[\s_-]*model)/i; const CLAUDE_EXTRA_USAGE_RESET_RE = - /(?:you(?:'|’)ve\s+hit\s+your\s+session\s+limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage|usage\s+limit\s+reached|usage\s+cap\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|claude\s+usage\s+limit\s+reached)[\s\S]{0,120}?\bresets?\s+(?:at\s+)?([^\n()]+?)(?:\s*\(([^)]+)\))?(?:[.!]|\n|$)/i; + /(?:you(?:'|’)ve\s+hit\s+your\s+(?:\w+\s+)?limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage|usage\s+limit\s+reached|usage\s+cap\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|claude\s+usage\s+limit\s+reached)[\s\S]{0,120}?\bresets?\s+(?:at\s+)?([^\n()]+?)(?:\s*\(([^)]+)\))?(?:[.!]|\n|$)/i; /** * Sum the per-model usage ledger from a Claude CLI result event. The result diff --git a/server/src/services/recovery/provider-failure-classification.test.ts b/server/src/services/recovery/provider-failure-classification.test.ts index 20f5f43e4f..307d38f4fe 100644 --- a/server/src/services/recovery/provider-failure-classification.test.ts +++ b/server/src/services/recovery/provider-failure-classification.test.ts @@ -65,6 +65,22 @@ describe("classifyAdapterFailureForRecovery", () => { }); }); + it("classifies the qualifier-less limit wording and parses the 'resets' clock", () => { + // Current Claude CLI phrasing, as recorded on the run by the adapter. + const now = new Date("2026-08-28T22:30:00.000Z"); + const classification = classifyAdapterFailureForRecovery({ + errorCode: "adapter_failed", + error: "Claude run failed: subtype=success: You've hit your limit · resets 2:30am (UTC)", + resultJson: null, + }, now); + + expect(classification).toEqual({ + kind: "provider_quota", + retryAt: new Date("2026-08-29T02:30:00.000Z"), + parsedResetTime: true, + }); + }); + it.each([ "model_not_found: requested model does not exist", "No API credentials were found for this provider", diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 1693ec9848..1e28caff56 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -288,7 +288,7 @@ function isProviderQuotaRecovery(latestRun: LatestIssueRun) { if (latestRun?.errorCode === "provider_quota") return true; if (readRecoveryRunErrorFamily(latestRun) === "provider_quota") return true; if (latestRun?.errorCode !== "adapter_failed") return false; - return /(?:usage|rate|quota) limit|quota (?:exceeded|reset)|try again after/i.test(latestRun.error ?? ""); + return /(?:usage|rate|quota) limit|you(?:'|’)ve hit your (?:\w+ )?limit|quota (?:exceeded|reset)|try again after/i.test(latestRun.error ?? ""); } function resolveStrandedRecoveryCause( @@ -406,7 +406,7 @@ const CONTINUATION_RECOVERY_TRANSIENT_BASE_BACKOFF_MS = 60_000; export const PROVIDER_QUOTA_RECOVERY_DEFAULT_BACKOFF_MS = 60 * 60 * 1000; const PROVIDER_QUOTA_ERROR_RE = - /(?:you(?:'|’)ve hit your usage limit|usage limit(?: reached| exceeded)?|provider quota|quota (?:limit )?exceeded|model (?:is )?at capacity)/i; + /(?:you(?:'|’)ve hit your (?:\w+ )?limit|usage limit(?: reached| exceeded)?|provider quota|quota (?:limit )?exceeded|model (?:is )?at capacity)/i; const CONFIGURATION_INCOMPLETE_ERROR_RE = /(?:model_not_found|model [^\n]{0,120} not found|missing (?:api )?(?:key|credentials?)|credentials? (?:are |is )?missing|no (?:api )?(?:key|credentials?) (?:was |were )?(?:found|configured|provided)|api key (?:is )?(?:not set|unavailable))/i; @@ -417,7 +417,7 @@ export type AdapterFailureRecoveryClassification = function parseProviderQuotaClockReset(error: string, now: Date) { const match = error.match( - /try again at\s+(\d{1,2})(?::(\d{2}))?\s*(?:([ap])\.?\s*m\.?)?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?/i, + /(?:try again at|resets?(?:\s+at)?)\s+(\d{1,2})(?::(\d{2}))?\s*(?:([ap])\.?\s*m\.?)?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?/i, ); if (!match) return null; diff --git a/server/src/services/recovery/stranded-notice.test.ts b/server/src/services/recovery/stranded-notice.test.ts index 4a2878ac32..e26776496c 100644 --- a/server/src/services/recovery/stranded-notice.test.ts +++ b/server/src/services/recovery/stranded-notice.test.ts @@ -156,6 +156,37 @@ describe("buildStrandedRecoveryEscalationNotice", () => { expect(rows.some((row) => row.label === "Failure summary")).toBe(false); }); + it("leads with the classified run failure code over the generic seed title", () => { + const notice = buildStrandedRecoveryEscalationNotice({ + seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }), + recoveryActionId: actionId, + recoveryOwner: owner, + sourceRun: { + ...sourceRun, + errorCode: "provider_quota", + errorSummary: "You've hit your limit · resets 2:30am (UTC)", + }, + }); + + expect(notice.presentation.title).toBe("Error: usage limit reached"); + expect(allRows(notice.metadata)).toContainEqual({ + type: "key_value", + label: "Failure code", + value: "provider_quota", + }); + }); + + it("titles auth-required run failures as a login error", () => { + expect( + buildStrandedRecoveryEscalationNotice({ + seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }), + recoveryActionId: actionId, + recoveryOwner: null, + sourceRun: { ...sourceRun, errorCode: "claude_auth_required" }, + }).presentation.title, + ).toBe("Error: not logged in to Claude"); + }); + it("is matched by the metadata-based escalation dedupe matcher", () => { const notice = buildStrandedRecoveryEscalationNotice({ seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }), diff --git a/server/src/services/recovery/stranded-notice.ts b/server/src/services/recovery/stranded-notice.ts index 79b78f83dc..75cbaace6f 100644 --- a/server/src/services/recovery/stranded-notice.ts +++ b/server/src/services/recovery/stranded-notice.ts @@ -35,6 +35,17 @@ const STRANDED_RECOVERY_NOTICE_TITLES_BY_CAUSE: Record = { execution_review_participant_recovery: "Review recovery stalled", }; +// Titles keyed by the source run's classified error code. The raw failure text +// never reaches the issue thread (summarizeRunFailureForIssueComment withholds +// it), so the classified code is the only safe, specific cause the collapsed +// notice row can lead with. A mapped code outranks the seed titles because the +// seeds describe the recovery family ("No live execution path"), not the cause. +const STRANDED_RECOVERY_NOTICE_TITLES_BY_RUN_ERROR_CODE: Record = { + provider_quota: "Error: usage limit reached", + claude_auth_required: "Error: not logged in to Claude", + acpx_auth_required: "Error: agent login required", +}; + export function buildImmediateExecutionPathRecoveryNoticeSeed(input: { status: "todo" | "in_progress"; }): StrandedRecoveryNoticeSeed { @@ -112,7 +123,9 @@ export function buildStrandedRecoveryEscalationNotice(input: { }): StrandedRecoveryEscalationNotice { const fallbackBody = input.fallbackBody?.trim(); const body = input.seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY); - const title = input.seed?.title ?? + const title = + STRANDED_RECOVERY_NOTICE_TITLES_BY_RUN_ERROR_CODE[input.sourceRun?.errorCode?.trim() ?? ""] ?? + input.seed?.title ?? STRANDED_RECOVERY_NOTICE_TITLES_BY_CAUSE[input.recoveryCause ?? ""] ?? DEFAULT_STRANDED_RECOVERY_NOTICE_TITLE;