Detect the qualifier-less Claude usage-limit message in quota classification (#12475)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runtime classifies adapter run failures, and the
recovery service uses that classification to decide between automatic
retry, a timed provider-quota wait, and a board escalation
> - The Claude CLI changed its subscription-limit stop message to
"You've hit your limit · resets 2:30am (UTC)", and no quota matcher
knows this qualifier-less wording
> - A limit-hit run therefore classifies as `adapter_failed` (or
`claude_auth_required`), recovery burns its continuation retries against
a hard limit, and the issue blocks with the opaque "No live execution
path" notice instead of waiting for the reset and retrying automatically
> - This pull request teaches the adapter and the recovery service the
new wording, and titles stranded-escalation notices from the classified
run error code so operators see the cause at a glance
> - The benefit is that usage-limit stops self-heal at the provider
reset time, and the notices that do post say "Error: usage limit
reached" or "Error: not logged in to Claude" instead of a generic title
## Linked Issues or Issue Description
No public issue exists; the underlying problem follows the bug template:
**What happened?**
On a staging deployment, an assigned `in_progress` issue hit the Claude
subscription usage limit. The run recorded the error `Claude run failed:
subtype=success: You've hit your limit · resets 2:30am (UTC)`. The
automatic continuation retry failed the same way in 34 seconds with
`errorCode: adapter_failed`. Terminal-run recovery then escalated: the
issue moved to `blocked` with the notice "No live execution path" and a
board-owned recovery action. The notice gave the operator no indication
that the cause was a usage limit with a known reset time.
**Expected behavior**
A usage-limit stop classifies as `provider_quota` with the reset clock
parsed into `retryNotBefore`. The recovery service takes its
provider-quota wait path: a system-owned recovery action that waits for
the reset time and retries the original assignee automatically. If an
escalation notice does post, its title names the classified cause.
**Steps to reproduce**
1. Run a `claude_local` agent on an issue until the Claude subscription
limit is hit, so the CLI result is "You've hit your limit · resets
\<time\> (UTC)".
2. Let terminal-run recovery retry the continuation.
3. Observe the issue block with the "No live execution path" notice
instead of a timed quota wait. `classifyAdapterFailureForRecovery`
returns `null` for the recorded error text; `CLAUDE_PROVIDER_QUOTA_RE`
and `PROVIDER_QUOTA_ERROR_RE` both fail to match it.
## What Changed
- `CLAUDE_PROVIDER_QUOTA_RE` and `CLAUDE_EXTRA_USAGE_RESET_RE`
(claude-local adapter) accept "you've hit your limit" with no qualifier,
alongside the existing "session"/"usage" wordings, so the run classifies
as `provider_quota` and the reset clock lands in `retryNotBefore`.
- `PROVIDER_QUOTA_ERROR_RE` and `isProviderQuotaRecovery` (recovery
service) accept the same wording, so runs recorded before the adapter
fix (errorCode `adapter_failed` with the limit text in the error) also
route to the quota wait.
- `parseProviderQuotaClockReset` parses the "resets 2:30am (UTC)" clock
shape alongside the existing "try again at" shape.
- `buildStrandedRecoveryEscalationNotice` titles the notice from the
source run's classified error code when one is mapped: `provider_quota`
→ "Error: usage limit reached", `claude_auth_required` → "Error: not
logged in to Claude", `acpx_auth_required` → "Error: agent login
required". The raw failure text stays withheld from the issue thread;
only the server-classified code is surfaced. Unmapped codes keep the
existing seed/cause titles.
## Verification
- `pnpm vitest run
packages/adapters/claude-local/src/server/parse.test.ts
server/src/services/recovery/provider-failure-classification.test.ts
server/src/services/recovery/stranded-notice.test.ts` — 72 tests pass,
including 5 new cases that use the exact new CLI message.
- `pnpm vitest run server/src/__tests__/issue-recovery-actions.test.ts
server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts` — 189 tests
pass (no reroute regressions from the widened matchers).
- `tsc --noEmit` clean for `@paperclipai/adapter-claude-local` and
`@paperclipai/server`.
## Risks
- Low risk. The regex widenings are additive; every previously matched
wording still matches, and the existing negative test ("Workspace
storage capacity limit reached." stays unclassified) still passes.
- Behavioral shift, intended: an `adapter_failed` run whose error text
is the new limit wording now routes to the silent system-owned quota
wait instead of a board escalation. This matches how the older limit
wordings already behave.
- The notice title change only affects escalations whose source run
carries one of the three mapped error codes; all other notices render
exactly as before.
## Model Used
Claude Fable 5 (`claude-fable-5`, Claude Code CLI, extended thinking
with tool use).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
parent
25cf079ec5
commit
300a89ec13
|
|
@ -199,6 +199,18 @@ describe("isClaudeTransientUpstreamError", () => {
|
|||
);
|
||||
});
|
||||
|
||||
it("classifies the qualifier-less limit wording as provider quota and extracts the retry time", () => {
|
||||
// Current Claude CLI phrasing: no "session"/"usage" qualifier before "limit".
|
||||
const now = new Date("2026-08-28T22:30:00.000Z");
|
||||
const errorMessage = "You've hit your limit · resets 2:30am (UTC)";
|
||||
|
||||
expect(isClaudeProviderQuotaError({ errorMessage })).toBe(true);
|
||||
expect(isClaudeTransientUpstreamError({ errorMessage })).toBe(false);
|
||||
expect(extractClaudeRetryNotBefore({ errorMessage }, now)?.toISOString()).toBe(
|
||||
"2026-08-29T02:30:00.000Z",
|
||||
);
|
||||
});
|
||||
|
||||
it("classifies Anthropic API rate_limit_error and overloaded_error as transient", () => {
|
||||
expect(
|
||||
isClaudeTransientUpstreamError({
|
||||
|
|
|
|||
|
|
@ -23,11 +23,11 @@ const URL_RE = /(https?:\/\/[^\s'"`<>()[\]{};,!?]+[^\s'"`<>()[\]{};,!.?:]+)/gi;
|
|||
const CLAUDE_TRANSIENT_UPSTREAM_RE =
|
||||
/(?:rate[-\s]?limit(?:ed)?|rate_limit_error|too\s+many\s+requests|\b429\b|overloaded(?:_error)?|server\s+overloaded|service\s+unavailable|\b503\b|\b529\b|high\s+demand|try\s+again\s+later|temporarily\s+unavailable|throttl(?:ed|ing)|throttlingexception|servicequotaexceededexception|out\s+of\s+extra\s+usage|extra\s+usage\b|claude\s+usage\s+limit\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|usage\s+limit\s+reached|usage\s+cap\s+reached)/i;
|
||||
const CLAUDE_PROVIDER_QUOTA_RE =
|
||||
/(?:you(?:'|’)ve\s+hit\s+your\s+session\s+limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage\b|claude\s+usage\s+limit\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|usage\s+limit\s+reached|usage\s+cap\s+reached|servicequotaexceededexception)/i;
|
||||
/(?:you(?:'|’)ve\s+hit\s+your\s+(?:\w+\s+)?limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage\b|claude\s+usage\s+limit\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|usage\s+limit\s+reached|usage\s+cap\s+reached|servicequotaexceededexception)/i;
|
||||
const CLAUDE_MODEL_NOT_FOUND_RE =
|
||||
/(?:\b404\b[\s\S]{0,120})?(?:model[\s_-]*(?:not[\s_-]*found|does not exist|unknown|invalid)|unknown[\s_-]*model)/i;
|
||||
const CLAUDE_EXTRA_USAGE_RESET_RE =
|
||||
/(?:you(?:'|’)ve\s+hit\s+your\s+session\s+limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage|usage\s+limit\s+reached|usage\s+cap\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|claude\s+usage\s+limit\s+reached)[\s\S]{0,120}?\bresets?\s+(?:at\s+)?([^\n()]+?)(?:\s*\(([^)]+)\))?(?:[.!]|\n|$)/i;
|
||||
/(?:you(?:'|’)ve\s+hit\s+your\s+(?:\w+\s+)?limit|session\s+limit\s+(?:reached|exceeded)|out\s+of\s+extra\s+usage|extra\s+usage|usage\s+limit\s+reached|usage\s+cap\s+reached|5[-\s]?hour\s+limit\s+reached|weekly\s+limit\s+reached|claude\s+usage\s+limit\s+reached)[\s\S]{0,120}?\bresets?\s+(?:at\s+)?([^\n()]+?)(?:\s*\(([^)]+)\))?(?:[.!]|\n|$)/i;
|
||||
|
||||
/**
|
||||
* Sum the per-model usage ledger from a Claude CLI result event. The result
|
||||
|
|
|
|||
|
|
@ -65,6 +65,22 @@ describe("classifyAdapterFailureForRecovery", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("classifies the qualifier-less limit wording and parses the 'resets' clock", () => {
|
||||
// Current Claude CLI phrasing, as recorded on the run by the adapter.
|
||||
const now = new Date("2026-08-28T22:30:00.000Z");
|
||||
const classification = classifyAdapterFailureForRecovery({
|
||||
errorCode: "adapter_failed",
|
||||
error: "Claude run failed: subtype=success: You've hit your limit · resets 2:30am (UTC)",
|
||||
resultJson: null,
|
||||
}, now);
|
||||
|
||||
expect(classification).toEqual({
|
||||
kind: "provider_quota",
|
||||
retryAt: new Date("2026-08-29T02:30:00.000Z"),
|
||||
parsedResetTime: true,
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
"model_not_found: requested model does not exist",
|
||||
"No API credentials were found for this provider",
|
||||
|
|
|
|||
|
|
@ -288,7 +288,7 @@ function isProviderQuotaRecovery(latestRun: LatestIssueRun) {
|
|||
if (latestRun?.errorCode === "provider_quota") return true;
|
||||
if (readRecoveryRunErrorFamily(latestRun) === "provider_quota") return true;
|
||||
if (latestRun?.errorCode !== "adapter_failed") return false;
|
||||
return /(?:usage|rate|quota) limit|quota (?:exceeded|reset)|try again after/i.test(latestRun.error ?? "");
|
||||
return /(?:usage|rate|quota) limit|you(?:'|’)ve hit your (?:\w+ )?limit|quota (?:exceeded|reset)|try again after/i.test(latestRun.error ?? "");
|
||||
}
|
||||
|
||||
function resolveStrandedRecoveryCause(
|
||||
|
|
@ -406,7 +406,7 @@ const CONTINUATION_RECOVERY_TRANSIENT_BASE_BACKOFF_MS = 60_000;
|
|||
export const PROVIDER_QUOTA_RECOVERY_DEFAULT_BACKOFF_MS = 60 * 60 * 1000;
|
||||
|
||||
const PROVIDER_QUOTA_ERROR_RE =
|
||||
/(?:you(?:'|’)ve hit your usage limit|usage limit(?: reached| exceeded)?|provider quota|quota (?:limit )?exceeded|model (?:is )?at capacity)/i;
|
||||
/(?:you(?:'|’)ve hit your (?:\w+ )?limit|usage limit(?: reached| exceeded)?|provider quota|quota (?:limit )?exceeded|model (?:is )?at capacity)/i;
|
||||
const CONFIGURATION_INCOMPLETE_ERROR_RE =
|
||||
/(?:model_not_found|model [^\n]{0,120} not found|missing (?:api )?(?:key|credentials?)|credentials? (?:are |is )?missing|no (?:api )?(?:key|credentials?) (?:was |were )?(?:found|configured|provided)|api key (?:is )?(?:not set|unavailable))/i;
|
||||
|
||||
|
|
@ -417,7 +417,7 @@ export type AdapterFailureRecoveryClassification =
|
|||
|
||||
function parseProviderQuotaClockReset(error: string, now: Date) {
|
||||
const match = error.match(
|
||||
/try again at\s+(\d{1,2})(?::(\d{2}))?\s*(?:([ap])\.?\s*m\.?)?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?/i,
|
||||
/(?:try again at|resets?(?:\s+at)?)\s+(\d{1,2})(?::(\d{2}))?\s*(?:([ap])\.?\s*m\.?)?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?/i,
|
||||
);
|
||||
if (!match) return null;
|
||||
|
||||
|
|
|
|||
|
|
@ -156,6 +156,37 @@ describe("buildStrandedRecoveryEscalationNotice", () => {
|
|||
expect(rows.some((row) => row.label === "Failure summary")).toBe(false);
|
||||
});
|
||||
|
||||
it("leads with the classified run failure code over the generic seed title", () => {
|
||||
const notice = buildStrandedRecoveryEscalationNotice({
|
||||
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }),
|
||||
recoveryActionId: actionId,
|
||||
recoveryOwner: owner,
|
||||
sourceRun: {
|
||||
...sourceRun,
|
||||
errorCode: "provider_quota",
|
||||
errorSummary: "You've hit your limit · resets 2:30am (UTC)",
|
||||
},
|
||||
});
|
||||
|
||||
expect(notice.presentation.title).toBe("Error: usage limit reached");
|
||||
expect(allRows(notice.metadata)).toContainEqual({
|
||||
type: "key_value",
|
||||
label: "Failure code",
|
||||
value: "provider_quota",
|
||||
});
|
||||
});
|
||||
|
||||
it("titles auth-required run failures as a login error", () => {
|
||||
expect(
|
||||
buildStrandedRecoveryEscalationNotice({
|
||||
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }),
|
||||
recoveryActionId: actionId,
|
||||
recoveryOwner: null,
|
||||
sourceRun: { ...sourceRun, errorCode: "claude_auth_required" },
|
||||
}).presentation.title,
|
||||
).toBe("Error: not logged in to Claude");
|
||||
});
|
||||
|
||||
it("is matched by the metadata-based escalation dedupe matcher", () => {
|
||||
const notice = buildStrandedRecoveryEscalationNotice({
|
||||
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }),
|
||||
|
|
|
|||
|
|
@ -35,6 +35,17 @@ const STRANDED_RECOVERY_NOTICE_TITLES_BY_CAUSE: Record<string, string> = {
|
|||
execution_review_participant_recovery: "Review recovery stalled",
|
||||
};
|
||||
|
||||
// Titles keyed by the source run's classified error code. The raw failure text
|
||||
// never reaches the issue thread (summarizeRunFailureForIssueComment withholds
|
||||
// it), so the classified code is the only safe, specific cause the collapsed
|
||||
// notice row can lead with. A mapped code outranks the seed titles because the
|
||||
// seeds describe the recovery family ("No live execution path"), not the cause.
|
||||
const STRANDED_RECOVERY_NOTICE_TITLES_BY_RUN_ERROR_CODE: Record<string, string> = {
|
||||
provider_quota: "Error: usage limit reached",
|
||||
claude_auth_required: "Error: not logged in to Claude",
|
||||
acpx_auth_required: "Error: agent login required",
|
||||
};
|
||||
|
||||
export function buildImmediateExecutionPathRecoveryNoticeSeed(input: {
|
||||
status: "todo" | "in_progress";
|
||||
}): StrandedRecoveryNoticeSeed {
|
||||
|
|
@ -112,7 +123,9 @@ export function buildStrandedRecoveryEscalationNotice(input: {
|
|||
}): StrandedRecoveryEscalationNotice {
|
||||
const fallbackBody = input.fallbackBody?.trim();
|
||||
const body = input.seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY);
|
||||
const title = input.seed?.title ??
|
||||
const title =
|
||||
STRANDED_RECOVERY_NOTICE_TITLES_BY_RUN_ERROR_CODE[input.sourceRun?.errorCode?.trim() ?? ""] ??
|
||||
input.seed?.title ??
|
||||
STRANDED_RECOVERY_NOTICE_TITLES_BY_CAUSE[input.recoveryCause ?? ""] ??
|
||||
DEFAULT_STRANDED_RECOVERY_NOTICE_TITLE;
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue