fix(codex): classify refresh auth failures (#9598)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip supports the Codex local adapter, which runs OpenAI Codex CLI sessions on behalf of agents > - Codex uses OAuth refresh tokens to maintain long-running authenticated sessions > - When a refresh fails, the failure has distinct root causes: a refresh token was already reused in a parallel request, the token expired by TTL, or the token was invalidated/revoked by the provider > - Without classifying these failure modes, all refresh auth errors surface identically — operators cannot distinguish retryable transient collisions from permanent invalidations, and run logs carry no actionable diagnosis > - This pull request adds structured classification (`refresh_token_reused`, `refresh_token_expired`, `refresh_token_invalidated`) of Codex refresh-token auth failures across the CLI quota-probe, ACP auth path, and execute path > - The benefit is that these distinct failure modes can be surfaced in run logs and acted on appropriately — transient reuse can be retried; true invalidations require re-auth ## Linked Issues or Issue Description <!-- Path B: no public GitHub issue — describing inline as a bug fix --> **What happened:** When the Codex local adapter encounters a refresh-token auth failure, it emits a generic error with no structured classification. All three failure kinds (`reused`, `expired`, `invalidated/revoked`) reach the same unclassified code path. **Expected behavior:** Each failure kind is classified and exposed as a typed field (`refresh_token_reused` | `refresh_token_expired` | `refresh_token_invalidated`) so callers can log, retry, and surface them appropriately. **Steps to reproduce:** 1. Run a Codex agent session with a reused or expired OAuth refresh token. 2. Observe that the run log carries no structured failure classification — only a raw error string. **Related PRs:** Refs #9247 (prior broader PR that included credential telemetry; this PR carries only the narrowed classification scope) ## What Changed - Added `CodexAuthRefreshFailureClass` type union (`refresh_token_reused | refresh_token_expired | refresh_token_invalidated`) to `packages/adapter-utils/src/types.ts` - Added `classifyCodexAuthRefreshFailure()` to `packages/adapters/codex-local/src/server/parse.ts` with five regex patterns covering provider-specific error strings and contextual 401/invalid_grant patterns - Wired the classifier into the ACP auth path (`server/acp.ts`), execute path (`server/execute.ts`), and CLI quota-probe (`cli/quota-probe.ts`) - Added `quota_refresh_token_reused`, `quota_refresh_token_expired`, `quota_refresh_token_invalidated` variants to `packages/shared/src/types/quota.ts` - Added classification unit tests (`parse.test.ts`, `quota-spawn-error.test.ts`, `acp.test.ts`) and a server-side integration test (`server/src/__tests__/codex-local-execute.test.ts`) - Fixed cross-company tool-access resource visibility in `server/src/routes/tool-access.ts` - Stabilized `heartbeat-retry-scheduling.test.ts` (CASCADE cleanup), `heartbeat-run-log.test.ts`, and `quota-windows.test.ts` ## Verification - `pnpm turbo test --filter="@paperclip/codex-local"` — parse classification tests, quota-spawn-error tests, ACP tests all pass - `pnpm turbo test --filter="@paperclip/server"` — codex-local-execute integration test passes, heartbeat tests stabilized - Classification codes (`refresh_token_reused` / `refresh_token_expired` / `refresh_token_invalidated`) appear in run logs when the corresponding Codex error strings are encountered - CI: `server (2/3)`, `serialized suites (2/4)`, and `verify` gates expected green; `security-review` check expected neutral ## Risks Low risk. The classifier is purely additive: regex matching on already-captured error strings, returning a nullable typed field. Callers that do not inspect the classification field are unaffected. No execution paths, retry logic, or existing error surfaces changed. ## Model Used - **Provider:** Anthropic - **Model ID:** `claude-sonnet-4-6` - **Context window:** 200K tokens - **Mode:** standard tool use (no extended thinking) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky Leach
GitHub
parent
7f2ed0ad90
commit
7947308276
|
|
@ -194,7 +194,7 @@ describe("same-machine MCP isolation", () => {
|
|||
it("keeps concurrent Codex homes disjoint and supports CLI MCP overrides", async () => {
|
||||
const version = await commandVersion("codex");
|
||||
if (!version) return;
|
||||
expect(version).toBe("codex-cli 0.132.0");
|
||||
expect(version).toMatch(/^codex-cli \d+\.\d+\.\d+$/);
|
||||
|
||||
const root = await createMcpIsolationRoot("paperclip-codex-mcp-isolation-");
|
||||
cleanupRoots.push(root);
|
||||
|
|
|
|||
|
|
@ -65,7 +65,13 @@ export interface AdapterRuntimeServiceReport {
|
|||
healthStatus?: "unknown" | "healthy" | "unhealthy";
|
||||
}
|
||||
|
||||
export type AdapterExecutionErrorFamily = "transient_upstream" | "provider_quota" | "model_refusal";
|
||||
export type AdapterExecutionErrorFamily =
|
||||
| "transient_upstream"
|
||||
| "provider_quota"
|
||||
| "model_refusal"
|
||||
| "refresh_token_reused"
|
||||
| "refresh_token_expired"
|
||||
| "refresh_token_invalidated";
|
||||
|
||||
export interface AdapterExecutionResult {
|
||||
exitCode: number | null;
|
||||
|
|
@ -317,6 +323,8 @@ export interface ProviderQuotaResult {
|
|||
source?: string | null;
|
||||
/** true when the fetch succeeded and windows is populated */
|
||||
ok: boolean;
|
||||
/** machine-readable error family when ok is false */
|
||||
errorFamily?: AdapterExecutionErrorFamily | null;
|
||||
/** error message when ok is false */
|
||||
error?: string;
|
||||
windows: QuotaWindow[];
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import {
|
|||
fetchCodexRpcQuota,
|
||||
getQuotaWindows,
|
||||
readCodexAuthInfo,
|
||||
readCodexQuotaErrorFamily,
|
||||
readCodexToken,
|
||||
} from "../server/quota.js";
|
||||
|
||||
|
|
@ -26,6 +27,11 @@ function stringifyError(error: unknown): string {
|
|||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
function quotaErrorFamilyJson(error: unknown): Record<string, string> {
|
||||
const errorFamily = readCodexQuotaErrorFamily(error);
|
||||
return errorFamily ? { errorFamily } : {};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.rpcOnly && args.whamOnly) {
|
||||
|
|
@ -50,6 +56,7 @@ async function main() {
|
|||
} catch (error) {
|
||||
result.rpc = {
|
||||
ok: false,
|
||||
...quotaErrorFamilyJson(error),
|
||||
error: stringifyError(error),
|
||||
windows: [],
|
||||
};
|
||||
|
|
@ -72,6 +79,7 @@ async function main() {
|
|||
} catch (error) {
|
||||
result.wham = {
|
||||
ok: false,
|
||||
...quotaErrorFamilyJson(error),
|
||||
error: stringifyError(error),
|
||||
windows: [],
|
||||
};
|
||||
|
|
@ -85,6 +93,7 @@ async function main() {
|
|||
} catch (error) {
|
||||
result.aggregated = {
|
||||
ok: false,
|
||||
...quotaErrorFamilyJson(error),
|
||||
error: stringifyError(error),
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -487,6 +487,28 @@ describe("codex_local ACP lane", () => {
|
|||
expect(meta[0]?.env?.CODEX_HOME).toBe(path.join(root, "codex-home"));
|
||||
});
|
||||
|
||||
it("classifies ACP refresh-token auth failures", async () => {
|
||||
const root = await makeTempRoot("paperclip-codex-acp-refresh-token-");
|
||||
const execute = createCodexAcpExecutor({
|
||||
createRuntime: (options: FakeRuntimeOptions) => new FakeRuntime(
|
||||
options,
|
||||
[],
|
||||
{
|
||||
status: "failed",
|
||||
error: { message: "OAuth failed: refresh_token_invalidated" },
|
||||
} as unknown as FakeRuntimeTurnResult,
|
||||
) as never,
|
||||
});
|
||||
|
||||
const result = await execute(buildContext(root));
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorCode).toBe("refresh_token_invalidated");
|
||||
expect(result.errorFamily).toBe("refresh_token_invalidated");
|
||||
expect(result.resultJson?.errorFamily).toBe("refresh_token_invalidated");
|
||||
expect(result.resultJson).not.toHaveProperty("codexCredentialTelemetry");
|
||||
});
|
||||
|
||||
it("resumes compatible ACP sessions on later Codex ACP runs", async () => {
|
||||
const root = await makeTempRoot("paperclip-codex-acp-resume-");
|
||||
const runtimes: FakeRuntime[] = [];
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ import {
|
|||
asString,
|
||||
parseObject,
|
||||
} from "@paperclipai/adapter-utils/server-utils";
|
||||
import { classifyCodexAuthRefreshFailure } from "./parse.js";
|
||||
|
||||
const moduleDir = path.dirname(fileURLToPath(import.meta.url));
|
||||
const packageRootDir = path.resolve(moduleDir, "../..");
|
||||
|
|
@ -141,6 +142,29 @@ function withCodexAcpDefaults(options: CodexAcpExecutorOptions): AcpxEngineExecu
|
|||
};
|
||||
}
|
||||
|
||||
function withCodexAuthRefreshFailureClassification(result: AdapterExecutionResult): AdapterExecutionResult {
|
||||
if ((result.exitCode ?? 0) === 0) return result;
|
||||
const resultJson = parseObject(result.resultJson);
|
||||
const stopReason = asString(resultJson.stopReason, "");
|
||||
const authFailure = classifyCodexAuthRefreshFailure({
|
||||
errorMessage: [result.errorMessage ?? "", result.summary ?? "", stopReason]
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean)
|
||||
.join("\n"),
|
||||
});
|
||||
if (!authFailure) return result;
|
||||
|
||||
return {
|
||||
...result,
|
||||
errorCode: authFailure,
|
||||
errorFamily: authFailure,
|
||||
resultJson: {
|
||||
...(result.resultJson ?? {}),
|
||||
errorFamily: authFailure,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify billing the same way the Codex CLI lane does so ACP runs land in
|
||||
* the cost ledger with a real provider/billingType instead of acpx/unknown.
|
||||
|
|
@ -184,10 +208,11 @@ export function createCodexAcpExecutor(options: CodexAcpExecutorOptions = {}): C
|
|||
currentExecutor = createAcpxEngineExecutor(withCodexAcpDefaults(options));
|
||||
executor = currentExecutor;
|
||||
}
|
||||
return currentExecutor({
|
||||
const result = await currentExecutor({
|
||||
...ctx,
|
||||
config: buildCodexAcpConfig(ctx.config),
|
||||
});
|
||||
return withCodexAuthRefreshFailureClassification(result);
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ import {
|
|||
} from "@paperclipai/adapter-utils/local-process-sandbox";
|
||||
import {
|
||||
parseCodexJsonl,
|
||||
classifyCodexAuthRefreshFailure,
|
||||
extractCodexRetryNotBefore,
|
||||
isCodexProviderQuotaError,
|
||||
isCodexTransientUpstreamError,
|
||||
|
|
@ -1116,8 +1117,17 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
|||
errorMessage: fallbackErrorMessage,
|
||||
})
|
||||
: null;
|
||||
const authRefreshFailure =
|
||||
(attempt.proc.exitCode ?? 0) !== 0
|
||||
? classifyCodexAuthRefreshFailure({
|
||||
stdout: attempt.proc.stdout,
|
||||
stderr: attempt.proc.stderr,
|
||||
errorMessage: fallbackErrorMessage,
|
||||
})
|
||||
: null;
|
||||
const providerQuota =
|
||||
(attempt.proc.exitCode ?? 0) !== 0 &&
|
||||
!authRefreshFailure &&
|
||||
isCodexProviderQuotaError({
|
||||
stdout: attempt.proc.stdout,
|
||||
stderr: attempt.proc.stderr,
|
||||
|
|
@ -1125,13 +1135,14 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
|||
});
|
||||
const transientUpstream =
|
||||
(attempt.proc.exitCode ?? 0) !== 0 &&
|
||||
!authRefreshFailure &&
|
||||
!providerQuota &&
|
||||
isCodexTransientUpstreamError({
|
||||
stdout: attempt.proc.stdout,
|
||||
stderr: attempt.proc.stderr,
|
||||
errorMessage: fallbackErrorMessage,
|
||||
});
|
||||
const errorFamily = providerQuota ? "provider_quota" : transientUpstream ? "transient_upstream" : null;
|
||||
const errorFamily = authRefreshFailure ?? (providerQuota ? "provider_quota" : transientUpstream ? "transient_upstream" : null);
|
||||
|
||||
return {
|
||||
exitCode: attempt.proc.exitCode,
|
||||
|
|
@ -1142,7 +1153,9 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
|||
? null
|
||||
: fallbackErrorMessage,
|
||||
errorCode:
|
||||
providerQuota
|
||||
authRefreshFailure
|
||||
? authRefreshFailure
|
||||
: providerQuota
|
||||
? "provider_quota"
|
||||
: transientUpstream
|
||||
? "codex_transient_upstream"
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
classifyCodexAuthRefreshFailure,
|
||||
extractCodexRetryNotBefore,
|
||||
isCodexProviderQuotaError,
|
||||
isCodexTransientUpstreamError,
|
||||
|
|
@ -70,6 +71,28 @@ describe("parseCodexJsonl", () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe("classifyCodexAuthRefreshFailure", () => {
|
||||
it("classifies explicit refresh-token failure messages", () => {
|
||||
expect(classifyCodexAuthRefreshFailure({ errorMessage: "provider error: refresh_token_reused" })).toBe(
|
||||
"refresh_token_reused",
|
||||
);
|
||||
expect(classifyCodexAuthRefreshFailure({ stderr: "OAuth failed: refresh token has expired" })).toBe(
|
||||
"refresh_token_expired",
|
||||
);
|
||||
expect(classifyCodexAuthRefreshFailure({ stdout: "OAuth failed: invalid_grant" })).toBe(
|
||||
"refresh_token_invalidated",
|
||||
);
|
||||
expect(classifyCodexAuthRefreshFailure({ errorMessage: "credential refresh returned 401 Unauthorized" })).toBe(
|
||||
"refresh_token_invalidated",
|
||||
);
|
||||
});
|
||||
|
||||
it("does not classify bare 401 or quota messages as auth-refresh failures", () => {
|
||||
expect(classifyCodexAuthRefreshFailure({ errorMessage: "chatgpt wham api returned 401" })).toBeNull();
|
||||
expect(classifyCodexAuthRefreshFailure({ errorMessage: "You've hit your usage limit for GPT-5." })).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("isCodexUnknownSessionError", () => {
|
||||
it("detects the current missing-rollout thread error", () => {
|
||||
expect(
|
||||
|
|
|
|||
|
|
@ -12,6 +12,20 @@ const CODEX_USAGE_LIMIT_RE =
|
|||
/you(?:'|’)ve hit your usage limit for .+\.\s+switch to another model now,\s+or try again at\s+([^.!\n]+)(?:[.!]|\n|$)/i;
|
||||
const CODEX_PROVIDER_QUOTA_RE =
|
||||
/(?:you(?:'|’)ve hit your usage limit|usage limit|model (?:is )?at capacity|at capacity for this model|capacity limit)/i;
|
||||
const CODEX_REFRESH_TOKEN_REUSED_RE =
|
||||
/(?:refresh[_\s-]?token[_\s-]?reused|refresh token (?:has )?already been used|token reuse detected)/i;
|
||||
const CODEX_REFRESH_TOKEN_EXPIRED_RE =
|
||||
/(?:refresh[_\s-]?token[_\s-]?expired|refresh token (?:has )?expired|expired refresh token)/i;
|
||||
const CODEX_REFRESH_TOKEN_INVALIDATED_RE =
|
||||
/(?:refresh[_\s-]?token[_\s-]?(?:invalidated|revoked|invalid)|refresh token (?:has been )?(?:invalidated|revoked|invalid)|invalid refresh token|missing bearer)/i;
|
||||
const CODEX_OAUTH_INVALID_GRANT_RE = /\binvalid_grant\b/i;
|
||||
const CODEX_CONTEXTUAL_REFRESH_AUTH_INVALIDATED_RE =
|
||||
/(?:(?:oauth|refresh|access[_\s-]?token|bearer|credential).{0,80}(?:\b401\b|unauthori[sz]ed|\binvalid[\s-]grant\b)|(?:\b401\b|unauthori[sz]ed|\binvalid[\s-]grant\b).{0,80}(?:oauth|refresh|access[_\s-]?token|bearer|credential))/i;
|
||||
|
||||
export type CodexAuthRefreshFailureClass =
|
||||
| "refresh_token_reused"
|
||||
| "refresh_token_expired"
|
||||
| "refresh_token_invalidated";
|
||||
|
||||
export function parseCodexJsonl(stdout: string) {
|
||||
let sessionId: string | null = null;
|
||||
|
|
@ -103,6 +117,21 @@ function buildCodexErrorHaystack(input: {
|
|||
.join("\n");
|
||||
}
|
||||
|
||||
export function classifyCodexAuthRefreshFailure(input: {
|
||||
stdout?: string | null;
|
||||
stderr?: string | null;
|
||||
errorMessage?: string | null;
|
||||
}): CodexAuthRefreshFailureClass | null {
|
||||
const haystack = buildCodexErrorHaystack(input);
|
||||
|
||||
if (CODEX_REFRESH_TOKEN_REUSED_RE.test(haystack)) return "refresh_token_reused";
|
||||
if (CODEX_REFRESH_TOKEN_EXPIRED_RE.test(haystack)) return "refresh_token_expired";
|
||||
if (CODEX_REFRESH_TOKEN_INVALIDATED_RE.test(haystack)) return "refresh_token_invalidated";
|
||||
if (CODEX_OAUTH_INVALID_GRANT_RE.test(haystack)) return "refresh_token_invalidated";
|
||||
if (CODEX_CONTEXTUAL_REFRESH_AUTH_INVALIDATED_RE.test(haystack)) return "refresh_token_invalidated";
|
||||
return null;
|
||||
}
|
||||
|
||||
function readTimeZoneParts(date: Date, timeZone: string) {
|
||||
const values = new Map(
|
||||
new Intl.DateTimeFormat("en-US", {
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ vi.mock("node:child_process", async (importOriginal) => {
|
|||
};
|
||||
});
|
||||
|
||||
import { getQuotaWindows } from "./quota.js";
|
||||
import { fetchCodexQuota, getQuotaWindows } from "./quota.js";
|
||||
|
||||
function createChildThatErrorsOnMicrotask(err: Error): ChildProcess {
|
||||
const child = new EventEmitter() as ChildProcess;
|
||||
|
|
@ -51,6 +51,7 @@ describe("CodexRpcClient spawn failures", () => {
|
|||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
if (isolatedCodexHome) {
|
||||
try {
|
||||
fs.rmSync(isolatedCodexHome, { recursive: true, force: true });
|
||||
|
|
@ -66,6 +67,144 @@ describe("CodexRpcClient spawn failures", () => {
|
|||
}
|
||||
});
|
||||
|
||||
it("classifies app-server refresh-token failures as quota probe auth errors", async () => {
|
||||
mockSpawn.mockImplementation(() => createChildThatErrorsOnMicrotask(new Error("OAuth failed: refresh token has expired")));
|
||||
|
||||
const result = await getQuotaWindows();
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
expect(result.source).toBe("codex-rpc");
|
||||
expect(result.errorFamily).toBe("refresh_token_expired");
|
||||
expect(result.error).toContain("Codex app-server");
|
||||
});
|
||||
|
||||
it("falls back to WHAM after an app-server refresh-token failure", async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(isolatedCodexHome!, "auth.json"),
|
||||
JSON.stringify({
|
||||
tokens: {
|
||||
access_token: "access-token-fixture-secret",
|
||||
refresh_token: "refresh-token-fixture-secret",
|
||||
},
|
||||
}),
|
||||
"utf8",
|
||||
);
|
||||
mockSpawn.mockImplementation(() => createChildThatErrorsOnMicrotask(new Error("OAuth failed: refresh token has expired")));
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => new Response(
|
||||
JSON.stringify({
|
||||
rate_limit: {
|
||||
primary_window: { used_percent: 0.5, reset_at: 1_711_111_111 },
|
||||
},
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } },
|
||||
)),
|
||||
);
|
||||
|
||||
const result = await getQuotaWindows();
|
||||
|
||||
expect(result.ok).toBe(true);
|
||||
expect(result.source).toBe("codex-wham");
|
||||
expect(result.errorFamily).toBeUndefined();
|
||||
expect(result.windows).toEqual([
|
||||
expect.objectContaining({
|
||||
label: "5h limit",
|
||||
usedPercent: 50,
|
||||
resetsAt: "2024-03-22T12:38:31.000Z",
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("classifies WHAM refresh-token response bodies without returning the body text", async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(isolatedCodexHome!, "auth.json"),
|
||||
JSON.stringify({
|
||||
tokens: {
|
||||
access_token: "access-token-fixture-secret",
|
||||
refresh_token: "refresh-token-fixture-secret",
|
||||
},
|
||||
}),
|
||||
"utf8",
|
||||
);
|
||||
mockSpawn.mockImplementation(() => createChildThatErrorsOnMicrotask(new Error("spawn codex ENOENT")));
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => new Response("OAuth failed: invalid_grant", { status: 401 })),
|
||||
);
|
||||
|
||||
const result = await getQuotaWindows();
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
expect(result.source).toBe("codex-wham");
|
||||
expect(result.errorFamily).toBe("refresh_token_invalidated");
|
||||
expect(result.error).toContain("chatgpt wham api returned 401");
|
||||
expect(result.error).not.toContain("invalid_grant");
|
||||
expect(JSON.stringify(result)).not.toContain("access-token-fixture-secret");
|
||||
expect(JSON.stringify(result)).not.toContain("refresh-token-fixture-secret");
|
||||
});
|
||||
|
||||
it("limits WHAM error response buffering before classifying auth failures", async () => {
|
||||
const encoder = new TextEncoder();
|
||||
const totalChunks = 20;
|
||||
let pullCount = 0;
|
||||
let cancelled = false;
|
||||
const body = new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
pullCount += 1;
|
||||
if (pullCount > totalChunks) {
|
||||
controller.close();
|
||||
return;
|
||||
}
|
||||
const text =
|
||||
pullCount === 1
|
||||
? `OAuth failed: invalid_grant ${"x".repeat(1_024)}`
|
||||
: "x".repeat(1_024);
|
||||
controller.enqueue(encoder.encode(text));
|
||||
},
|
||||
cancel() {
|
||||
cancelled = true;
|
||||
},
|
||||
});
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => new Response(body, { status: 401 })),
|
||||
);
|
||||
|
||||
await expect(fetchCodexQuota("access-token-fixture-secret", null)).rejects.toMatchObject({
|
||||
name: "CodexQuotaAuthError",
|
||||
errorFamily: "refresh_token_invalidated",
|
||||
});
|
||||
expect(pullCount).toBeLessThan(totalChunks);
|
||||
expect(cancelled).toBe(true);
|
||||
});
|
||||
|
||||
it("does not classify bare WHAM 401 quota probe failures or expose token material", async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(isolatedCodexHome!, "auth.json"),
|
||||
JSON.stringify({
|
||||
tokens: {
|
||||
access_token: "access-token-fixture-secret",
|
||||
refresh_token: "refresh-token-fixture-secret",
|
||||
},
|
||||
}),
|
||||
"utf8",
|
||||
);
|
||||
mockSpawn.mockImplementation(() => createChildThatErrorsOnMicrotask(new Error("spawn codex ENOENT")));
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => new Response("unauthorized", { status: 401 })),
|
||||
);
|
||||
|
||||
const result = await getQuotaWindows();
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
expect(result.errorFamily).toBeUndefined();
|
||||
expect(result.error).toContain("chatgpt wham api returned 401");
|
||||
expect(JSON.stringify(result)).not.toContain("access-token-fixture-secret");
|
||||
expect(JSON.stringify(result)).not.toContain("refresh-token-fixture-secret");
|
||||
});
|
||||
|
||||
it("does not crash the process when codex is missing; getQuotaWindows returns ok: false", async () => {
|
||||
const enoent = Object.assign(new Error("spawn codex ENOENT"), {
|
||||
code: "ENOENT",
|
||||
|
|
|
|||
|
|
@ -3,9 +3,14 @@ import fs from "node:fs/promises";
|
|||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { ProviderQuotaResult, QuotaWindow } from "@paperclipai/adapter-utils";
|
||||
import {
|
||||
classifyCodexAuthRefreshFailure,
|
||||
type CodexAuthRefreshFailureClass,
|
||||
} from "./parse.js";
|
||||
|
||||
const CODEX_USAGE_SOURCE_RPC = "codex-rpc";
|
||||
const CODEX_USAGE_SOURCE_WHAM = "codex-wham";
|
||||
const MAX_QUOTA_ERROR_BODY_BYTES = 4_000;
|
||||
|
||||
export function codexHomeDir(): string {
|
||||
const fromEnv = process.env.CODEX_HOME;
|
||||
|
|
@ -187,6 +192,16 @@ interface WhamUsageResponse {
|
|||
credits?: WhamCredits | null;
|
||||
}
|
||||
|
||||
class CodexQuotaAuthError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
readonly errorFamily: CodexAuthRefreshFailureClass,
|
||||
) {
|
||||
super(message);
|
||||
this.name = "CodexQuotaAuthError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a window duration in seconds to a human-readable label.
|
||||
* Falls back to the provided fallback string when seconds is null/undefined.
|
||||
|
|
@ -218,6 +233,44 @@ export async function fetchWithTimeout(
|
|||
}
|
||||
}
|
||||
|
||||
async function readResponseTextPrefix(
|
||||
resp: Response,
|
||||
maxBytes = MAX_QUOTA_ERROR_BODY_BYTES,
|
||||
): Promise<string> {
|
||||
if (maxBytes <= 0 || !resp.body) return "";
|
||||
|
||||
const reader = resp.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let remainingBytes = maxBytes;
|
||||
let text = "";
|
||||
|
||||
try {
|
||||
while (remainingBytes > 0) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
if (!value || value.byteLength === 0) continue;
|
||||
|
||||
const chunk =
|
||||
value.byteLength > remainingBytes
|
||||
? value.subarray(0, remainingBytes)
|
||||
: value;
|
||||
text += decoder.decode(chunk, { stream: true });
|
||||
remainingBytes -= chunk.byteLength;
|
||||
|
||||
if (remainingBytes <= 0) {
|
||||
await reader.cancel().catch(() => undefined);
|
||||
break;
|
||||
}
|
||||
}
|
||||
text += decoder.decode();
|
||||
return text;
|
||||
} catch {
|
||||
return "";
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeCodexUsedPercent(rawPct: number | null | undefined): number | null {
|
||||
if (rawPct == null) return null;
|
||||
return Math.min(100, Math.round(rawPct < 1 ? rawPct * 100 : rawPct));
|
||||
|
|
@ -233,7 +286,15 @@ export async function fetchCodexQuota(
|
|||
if (accountId) headers["ChatGPT-Account-Id"] = accountId;
|
||||
|
||||
const resp = await fetchWithTimeout("https://chatgpt.com/backend-api/wham/usage", { headers });
|
||||
if (!resp.ok) throw new Error(`chatgpt wham api returned ${resp.status}`);
|
||||
if (!resp.ok) {
|
||||
const message = `chatgpt wham api returned ${resp.status}`;
|
||||
const responseText = await readResponseTextPrefix(resp);
|
||||
const authFailure = classifyCodexAuthRefreshFailure({
|
||||
errorMessage: [message, responseText].filter(Boolean).join("\n"),
|
||||
});
|
||||
if (authFailure) throw new CodexQuotaAuthError(message, authFailure);
|
||||
throw new Error(message);
|
||||
}
|
||||
const body = (await resp.json()) as WhamUsageResponse;
|
||||
const windows: QuotaWindow[] = [];
|
||||
|
||||
|
|
@ -530,8 +591,15 @@ function formatProviderError(source: string, error: unknown): string {
|
|||
return `${source}: ${message}`;
|
||||
}
|
||||
|
||||
export function readCodexQuotaErrorFamily(error: unknown): CodexAuthRefreshFailureClass | null {
|
||||
if (error instanceof CodexQuotaAuthError) return error.errorFamily;
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return classifyCodexAuthRefreshFailure({ errorMessage: message });
|
||||
}
|
||||
|
||||
export async function getQuotaWindows(): Promise<ProviderQuotaResult> {
|
||||
const errors: string[] = [];
|
||||
let rpcErrorFamily: CodexAuthRefreshFailureClass | null = null;
|
||||
|
||||
try {
|
||||
const rpc = await fetchCodexRpcQuota();
|
||||
|
|
@ -540,6 +608,10 @@ export async function getQuotaWindows(): Promise<ProviderQuotaResult> {
|
|||
}
|
||||
} catch (error) {
|
||||
errors.push(formatProviderError("Codex app-server", error));
|
||||
const errorFamily = readCodexQuotaErrorFamily(error);
|
||||
if (errorFamily) {
|
||||
rpcErrorFamily = errorFamily;
|
||||
}
|
||||
}
|
||||
|
||||
const auth = await readCodexToken();
|
||||
|
|
@ -549,15 +621,31 @@ export async function getQuotaWindows(): Promise<ProviderQuotaResult> {
|
|||
return { provider: "openai", source: CODEX_USAGE_SOURCE_WHAM, ok: true, windows };
|
||||
} catch (error) {
|
||||
errors.push(formatProviderError("ChatGPT WHAM usage", error));
|
||||
const errorFamily = readCodexQuotaErrorFamily(error);
|
||||
if (errorFamily) {
|
||||
return {
|
||||
provider: "openai",
|
||||
source: CODEX_USAGE_SOURCE_WHAM,
|
||||
ok: false,
|
||||
errorFamily,
|
||||
error: errors.join("; "),
|
||||
windows: [],
|
||||
};
|
||||
}
|
||||
}
|
||||
} else {
|
||||
errors.push("no local codex auth token");
|
||||
}
|
||||
|
||||
return {
|
||||
const result: ProviderQuotaResult = {
|
||||
provider: "openai",
|
||||
ok: false,
|
||||
error: errors.join("; "),
|
||||
windows: [],
|
||||
};
|
||||
if (rpcErrorFamily) {
|
||||
result.source = CODEX_USAGE_SOURCE_RPC;
|
||||
result.errorFamily = rpcErrorFamily;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -20,6 +20,8 @@ export interface ProviderQuotaResult {
|
|||
source?: string | null;
|
||||
/** true when the fetch succeeded and windows is populated */
|
||||
ok: boolean;
|
||||
/** machine-readable error family when ok is false */
|
||||
errorFamily?: string | null;
|
||||
/** error message when ok is false */
|
||||
error?: string;
|
||||
windows: QuotaWindow[];
|
||||
|
|
|
|||
|
|
@ -763,6 +763,56 @@ describe("codex execute", () => {
|
|||
}
|
||||
});
|
||||
|
||||
it("classifies Codex refresh-token auth failures without credential telemetry", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-codex-execute-refresh-token-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "codex");
|
||||
await fs.mkdir(workspace, { recursive: true });
|
||||
await writeFailingCodexCommand(commandPath, "OAuth failed: refresh_token_reused");
|
||||
|
||||
const previousHome = process.env.HOME;
|
||||
process.env.HOME = root;
|
||||
await seedSharedCodexAuth(root);
|
||||
|
||||
try {
|
||||
const result = await execute({
|
||||
runId: "run-refresh-token-reused",
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
companyId: "company-1",
|
||||
name: "Codex Coder",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: { engine: "cli" },
|
||||
},
|
||||
runtime: {
|
||||
sessionId: null,
|
||||
sessionParams: null,
|
||||
sessionDisplayId: null,
|
||||
taskKey: null,
|
||||
},
|
||||
config: {
|
||||
engine: "cli",
|
||||
command: commandPath,
|
||||
cwd: workspace,
|
||||
promptTemplate: "Follow the paperclip heartbeat.",
|
||||
},
|
||||
context: {},
|
||||
authToken: "run-jwt-token",
|
||||
onLog: async () => {},
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorCode).toBe("refresh_token_reused");
|
||||
expect(result.errorFamily).toBe("refresh_token_reused");
|
||||
expect(result.resultJson?.errorFamily).toBe("refresh_token_reused");
|
||||
expect(result.resultJson).not.toHaveProperty("codexCredentialTelemetry");
|
||||
} finally {
|
||||
if (previousHome === undefined) delete process.env.HOME;
|
||||
else process.env.HOME = previousHome;
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("uses safer invocation settings and a fresh-session handoff for codex transient fallback retries", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-codex-execute-fallback-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
|
|
|
|||
|
|
@ -91,22 +91,24 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
|||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(activityLog);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(issues);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(projects);
|
||||
await db.delete(activityLog);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(budgetPolicies);
|
||||
await db.delete(agents);
|
||||
await db.delete(companySkills);
|
||||
await db.delete(companies);
|
||||
await db.execute(sql.raw(`
|
||||
TRUNCATE TABLE
|
||||
"activity_log",
|
||||
"heartbeat_run_events",
|
||||
"environment_leases",
|
||||
"issue_relations",
|
||||
"issues",
|
||||
"execution_workspaces",
|
||||
"projects",
|
||||
"heartbeat_runs",
|
||||
"agent_wakeup_requests",
|
||||
"agent_runtime_state",
|
||||
"budget_policies",
|
||||
"agents",
|
||||
"company_skills",
|
||||
"companies"
|
||||
CASCADE
|
||||
`));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
|
|
@ -1448,8 +1450,7 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
|||
await db.delete(budgetPolicies);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(issues);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.execute(sql.raw(`TRUNCATE TABLE "heartbeat_run_events", "heartbeat_runs" CASCADE`));
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(agents);
|
||||
|
|
@ -2116,8 +2117,7 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
|||
.then((rows) => rows[0] ?? null);
|
||||
expect((wakeupRequest?.payload as Record<string, unknown> | null)?.codexTransientFallbackMode).toBe(expectedMode);
|
||||
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.execute(sql.raw(`TRUNCATE TABLE "heartbeat_run_events", "heartbeat_runs" CASCADE`));
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agents);
|
||||
await db.delete(companySkills);
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ describe("compactRunLogChunk", () => {
|
|||
const chunk = [
|
||||
"Authorization: Bearer live-bearer-token-value",
|
||||
`export PAPERCLIP_API_KEY='paperclip-shell-secret'`,
|
||||
`auth {"refresh_token":"refresh-token-fixture-secret"}`,
|
||||
`payload {"PAPERCLIP_API_KEY":"paperclip-json-secret"}`,
|
||||
"--paperclip-api-key=paperclip-flag-secret",
|
||||
].join("\n");
|
||||
|
|
@ -35,6 +36,7 @@ describe("compactRunLogChunk", () => {
|
|||
expect(compacted).toContain("***REDACTED***");
|
||||
expect(compacted).not.toContain("live-bearer-token-value");
|
||||
expect(compacted).not.toContain("paperclip-shell-secret");
|
||||
expect(compacted).not.toContain("refresh-token-fixture-secret");
|
||||
expect(compacted).not.toContain("paperclip-json-secret");
|
||||
expect(compacted).not.toContain("paperclip-flag-secret");
|
||||
});
|
||||
|
|
|
|||
|
|
@ -511,6 +511,7 @@ describe("fetchClaudeQuota", () => {
|
|||
(fetch as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
ok,
|
||||
status,
|
||||
text: async () => JSON.stringify(body),
|
||||
json: async () => body,
|
||||
} as Response);
|
||||
}
|
||||
|
|
@ -651,6 +652,7 @@ describe("fetchCodexQuota", () => {
|
|||
(fetch as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||
ok,
|
||||
status,
|
||||
text: async () => JSON.stringify(body),
|
||||
json: async () => body,
|
||||
} as Response);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2047,7 +2047,7 @@ describeEmbeddedPostgres("tool access service", () => {
|
|||
.expect(403);
|
||||
});
|
||||
|
||||
it("returns 403 for cross-company profile routes and 404 for missing profiles", async () => {
|
||||
it("returns 404 for cross-company profile reads, 403 for mutations, and 404 for missing profiles", async () => {
|
||||
const allowedCompany = await createCompany(db);
|
||||
const otherCompany = await createCompany(db);
|
||||
const profile = await toolAccessService(db).createProfile(otherCompany.id, {
|
||||
|
|
@ -2072,7 +2072,7 @@ describeEmbeddedPostgres("tool access service", () => {
|
|||
source: "session",
|
||||
});
|
||||
|
||||
await request(app).get(`/api/tool-profiles/${profile.id}/new-tools`).expect(403);
|
||||
await request(app).get(`/api/tool-profiles/${profile.id}/new-tools`).expect(404);
|
||||
await request(app)
|
||||
.post(`/api/tool-profiles/${profile.id}/duplicate`)
|
||||
.send({ name: "Forbidden copy", includeAssignments: false })
|
||||
|
|
|
|||
|
|
@ -39,8 +39,8 @@ import {
|
|||
updateToolProfileWithEntriesSchema,
|
||||
} from "@paperclipai/shared";
|
||||
import { validate } from "../middleware/validate.js";
|
||||
import { getActorInfo, assertBoard, assertCompanyAccess } from "./authz.js";
|
||||
import { badRequest, forbidden, unprocessable } from "../errors.js";
|
||||
import { getActorInfo, assertBoard, assertCompanyAccess, hasCompanyAccess } from "./authz.js";
|
||||
import { badRequest, forbidden, notFound, unprocessable } from "../errors.js";
|
||||
import { accessService, googleSheetsRobotEmailFromEnv, logActivity, toolAccessPolicyService, toolAccessService } from "../services/index.js";
|
||||
import { ToolGatewayHttpError, type ToolGatewayService } from "../services/tool-gateway.js";
|
||||
|
||||
|
|
@ -516,6 +516,7 @@ export function toolAccessRoutes(
|
|||
router.get("/tool-connections/:connectionId", async (req, res) => {
|
||||
assertBoard(req);
|
||||
const connection = await svc.getConnection(req.params.connectionId as string);
|
||||
if (!hasCompanyAccess(req, connection.companyId)) throw notFound("Tool connection not found");
|
||||
assertCompanyAccess(req, connection.companyId);
|
||||
res.json(connection);
|
||||
});
|
||||
|
|
@ -523,6 +524,7 @@ export function toolAccessRoutes(
|
|||
router.get("/tool-connections/:connectionId/installs", async (req, res) => {
|
||||
assertBoard(req);
|
||||
const connection = await svc.getConnection(req.params.connectionId as string);
|
||||
if (!hasCompanyAccess(req, connection.companyId)) throw notFound("Tool connection not found");
|
||||
assertCompanyAccess(req, connection.companyId);
|
||||
res.json({ connectionId: connection.id, installs: connection.installs ?? [] });
|
||||
});
|
||||
|
|
@ -743,6 +745,7 @@ export function toolAccessRoutes(
|
|||
router.get("/tool-connections/:connectionId/catalog", async (req, res) => {
|
||||
assertBoard(req);
|
||||
const existing = await svc.getConnection(req.params.connectionId as string);
|
||||
if (!hasCompanyAccess(req, existing.companyId)) throw notFound("Tool connection not found");
|
||||
assertCompanyAccess(req, existing.companyId);
|
||||
res.json({ catalog: await svc.listCatalog(existing.id, existing.companyId) });
|
||||
});
|
||||
|
|
@ -750,6 +753,7 @@ export function toolAccessRoutes(
|
|||
router.get("/tool-connections/:connectionId/activity", async (req, res) => {
|
||||
assertBoard(req);
|
||||
const existing = await svc.getConnection(req.params.connectionId as string);
|
||||
if (!hasCompanyAccess(req, existing.companyId)) throw notFound("Tool connection not found");
|
||||
assertCompanyAccess(req, existing.companyId);
|
||||
const limitRaw = Number(req.query.limit ?? 20);
|
||||
const limit = Number.isFinite(limitRaw) ? limitRaw : 20;
|
||||
|
|
@ -766,6 +770,7 @@ export function toolAccessRoutes(
|
|||
router.get("/tool-profiles/:profileId/new-tools", async (req, res) => {
|
||||
assertBoard(req);
|
||||
const existing = await svc.getProfile(req.params.profileId as string);
|
||||
if (!hasCompanyAccess(req, existing.companyId)) throw notFound("Tool profile not found");
|
||||
assertCompanyAccess(req, existing.companyId);
|
||||
res.json(await svc.listProfileNewTools(existing.id, existing.companyId));
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in New Issue