fix(runner): align direct eval provider setup with qualified runtimes (#12945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner direct live evals test semantic tools against a mock control
plane.
> - The first complete AWS campaign exercised 358 cells.
> - It exposed setup differences from the working full-stack harness.
> - This pull request corrects those direct-harness differences.
> - It preserves production permission defaults and the full-stack
workflow.
## Linked Issues or Issue Description
**What happened?**
Native Codex cells could not find a global codex executable. ACPX denied
unattended tool requests and lost valid provider usage receipts.
AgentCore hit a 30-second facade timeout while its worker allows 120
seconds for delivery. Several models reported a native run result
without updating the separate mock task state.
**What did you expect?**
The direct harness should use the pinned executable, explicit test
permissions, and a timeout compatible with the provider delivery
contract. Its instructions should explain which operation changes mock
task state.
**Steps to reproduce**
Run the full Runner Direct Live Protocol Evals workflow. Baseline
campaign:
https://github.com/paperclipai/paperclip/actions/runs/34059009921.
**Paperclip version**
Master at b55ce03e86. Related public
change: #12932. No duplicate fix was found.
## What Changed
- Resolve native Codex from the pinned Codex ACP dependency, as the
full-stack launcher does.
- Select explicit unattended ACPX permissions only in the isolated
direct eval harness.
- Add a configurable bounded turn-admission wait. AgentCore direct evals
use up to 125 seconds, capped by their turn budget. Other callers retain
the current default waits.
- Explain mock task-state operations separately from native run-result
reporting. Keep all scoring assertions unchanged.
- Bind the direct turn's scope to the current request, rather than stale
shared fixture notes. A provider's end-of-turn result does not authorize
an unrequested mock task completion or extra completion comment.
- Normalize the qualified Claude/Codex usage semantics without
double-counting reasoning or inventing unknown billable categories.
Forward ACPX's persisted terminal prompt-response usage for the exact
current turn; reject stale or ambiguous receipts.
- Preserve the known numeric ACPX token-counter aliases through durable
redaction. Continue redacting strings and credential-shaped values. A
regression test reproduces the previously redacted usage before
normalization.
- Add regression tests and operating documentation.
## Verification
- Passed Runner TypeScript build.
- Passed 88 focused tests across the eval request contract, provider
setup, and Runner transport.
- Passed 93 focused ACPX adapter, sidecar lifecycle, and
usage-accounting tests, including persisted receipt identity and
missing-field regressions.
- Passed 97 additional runtime-host, in-process ACPX driver, OpenCode
proxy, and MCP bridge tests.
- Reproduced the ACPX counter-redaction failure, then passed 36 Rust
decoder/normalizer and durable-redaction tests after the fix.
Credential-shaped strings and objects remain redacted.
- Native Codex get-task-context passed a real provider probe with no
global CLI dependency.
- Local ACPX probes reached a platform startup rejection on macOS;
qualification therefore used the intended AWS Linux fleet. Complete
campaign 34060573948 reached 341/358 passing with zero infrastructure
failures (up from 250/358 and 100 infrastructure failures). ACPX Claude
reached 34/35, ACPX Codex 7/8, native Codex 34/35, and AgentCore 35/35.
Remaining failures were retained in the canonical report:
https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34060573948-1/.
- Final complete campaign 34062394019 exercises the current-request
scope clarification with all 358 cells; results pending.
- Passed git diff --check.
- Repo-wide typecheck, build, and tests are delegated to PR CI. They
were not repeated on this machine.
## Risks
- ACPX approval is scoped to the operator-requested isolated fixture
harness. Production defaults do not change.
- AgentCore admission can wait longer, but remains within the whole-turn
budget.
- The mock fixture has no production task lifecycle service. Explicit
task-state instructions describe that boundary; they do not relax
scoring.
- This change does not modify the browser full-stack workflow or its
test fixtures.
- ACPX receipt corrections apply to all native ACPX sessions. They rely
on the pinned qualified server contracts; unknown or ambiguous receipts
stay unknown, and a final accounting-read error never overwrites the
provider's terminal result.
## Model Used
- OpenAI GPT-6 (`gpt-6-astra`) through Codex, with reasoning, tool use,
and code execution. The session does not expose an exact context-window
limit.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
GitHub
parent
d293dd3d14
commit
83987210d6
|
|
@ -62,6 +62,43 @@ same `paperclip.native-execution-input.v3` contract as production, including
|
|||
the AgentCore HarnessSkill upload path, without borrowing any browser E2E
|
||||
setup.
|
||||
|
||||
The direct CLI resolves native Codex from the pinned Codex ACP dependency, as
|
||||
the browser E2E launcher does; no globally installed `codex` is required.
|
||||
Its isolated fixture sessions explicitly use unattended ACPX `approve-all`
|
||||
permissions, matching the full-stack ACPX test profiles. OpenCode retains its
|
||||
existing semantic-only permission boundary.
|
||||
The seeded mock control plane still enforces tool grants, company boundaries,
|
||||
and governance checks. These test-only choices do not change production
|
||||
permission defaults and are not a qualification of interactive approval UI.
|
||||
|
||||
AgentCore eval admission waits up to 125 seconds, capped by the configured
|
||||
whole-turn timeout. Its network worker already allows 120 seconds for cold
|
||||
invocation delivery; the generic 30-second facade wait must not cut that
|
||||
delivery short during parallel fleet startup. Other providers retain their
|
||||
existing admission timeout.
|
||||
|
||||
The runtime instructions distinguish the fixture's `finish_task`/`block_task`
|
||||
state changes from native `paperclip_finish`/`paperclip_block` run-result
|
||||
reporting. This layer has no production server to apply a reported run result
|
||||
to a task. Its completion cases still require the actual semantic operation
|
||||
and durable mock-state change; reporting success alone does not pass them.
|
||||
The current turn request defines the requested work. Shared seed descriptions,
|
||||
old eval notes, or past accepted interactions are background state, not a
|
||||
replacement objective or proof that a newly requested action is already done.
|
||||
Finishing the provider turn does not authorize an unrequested mock task-state
|
||||
change or completion comment. These harness instructions keep single-operation
|
||||
cases bounded while leaving their operation and state-effect assertions intact.
|
||||
|
||||
ACPX accounting uses the qualified server's billable token semantics: Claude
|
||||
and Codex already include reasoning in output, and Codex has no cache-write
|
||||
billing category. Other missing categories remain unknown. ACPX stores Codex's
|
||||
terminal prompt-response usage without streaming it, so the sidecar forwards
|
||||
the single newly persisted receipt bound to that exact turn before its terminal
|
||||
event. It never substitutes a previous turn's receipt or context occupancy for
|
||||
billable usage. These receipt corrections also apply to non-eval ACPX sessions.
|
||||
The durable redactor preserves these known numeric token-counter fields;
|
||||
string/object values under the same names remain redacted as sensitive data.
|
||||
|
||||
`all` means the maintained enabled campaign, not every matching file in the
|
||||
roster directory. A missing campaign file fails closed. Within an enabled
|
||||
lane, a missing remote profile or unavailable provider is retained as an
|
||||
|
|
|
|||
|
|
@ -1135,6 +1135,12 @@ fn sensitive_key(key: &str, value: &Value) -> bool {
|
|||
| "outputtokens"
|
||||
| "cachereadtokens"
|
||||
| "cachewritetokens"
|
||||
// The qualified ACPX sidecar uses these numeric billing aliases.
|
||||
// Strings under the same names remain credentials, not counters.
|
||||
| "cachedreadtokens"
|
||||
| "cachedwritetokens"
|
||||
| "thoughttokens"
|
||||
| "totaltokens"
|
||||
| "pretokens"
|
||||
| "posttokens"
|
||||
) {
|
||||
|
|
@ -1825,6 +1831,24 @@ mod tests {
|
|||
sanitize_value(&json!({"inputTokens": "plain-provider-secret"})),
|
||||
json!({"inputTokens": "[REDACTED]"})
|
||||
);
|
||||
for key in [
|
||||
"cachedReadTokens",
|
||||
"cachedWriteTokens",
|
||||
"thoughtTokens",
|
||||
"totalTokens",
|
||||
] {
|
||||
assert_eq!(sanitize_value(&json!({key: 12})), json!({key: 12}));
|
||||
for value in [
|
||||
json!("plain-provider-secret"),
|
||||
json!({"value":"secret"}),
|
||||
json!(["secret"]),
|
||||
] {
|
||||
assert_eq!(
|
||||
sanitize_value(&json!({key: value})),
|
||||
json!({key: "[REDACTED]"})
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -201,6 +201,49 @@ fn maps_bounded_plan_and_completion_state() {
|
|||
assert_eq!(events[0].priority, EventPriority::P1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preserves_usage_counters_across_sidecar_payload_redaction() {
|
||||
let mut scope = AcpxEventScope::new("run-1").unwrap();
|
||||
scope.bind_turn("turn-1").unwrap();
|
||||
let decoded = decode_acpx_event(
|
||||
&scope,
|
||||
&AcpxSidecarEvent {
|
||||
sequence: 1,
|
||||
event_type: GeneratedAcpxSidecarEventType::RuntimeEvent,
|
||||
run_id: Some("run-1".to_owned()),
|
||||
turn_id: Some("turn-1".to_owned()),
|
||||
payload: json!({
|
||||
"type":"status", "tag":"usage_update",
|
||||
"breakdown":{
|
||||
"inputTokens":12, "outputTokens":7, "thoughtTokens":0,
|
||||
"cachedReadTokens":2, "cachedWriteTokens":0, "totalTokens":21
|
||||
},
|
||||
"accessToken":"provider-secret",
|
||||
"cost":{"amount":0.25,"currency":"USD"}
|
||||
}),
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
let AcpxEventPayload::Runtime {
|
||||
kind,
|
||||
tool_operation,
|
||||
payload,
|
||||
..
|
||||
} = decoded
|
||||
else {
|
||||
panic!("runtime usage must decode as a runtime payload");
|
||||
};
|
||||
assert_eq!(payload["accessToken"], "[REDACTED]");
|
||||
assert_eq!(payload["breakdown"]["totalTokens"], 21);
|
||||
let events =
|
||||
normalize_acpx_runtime_event(kind, &payload, tool_operation, "event-7", "turn-1", 3);
|
||||
assert_eq!(events[0].payload["runDeltaAvailable"], true);
|
||||
assert_eq!(events[0].payload["runDelta"]["inputTokens"], 12);
|
||||
assert_eq!(events[0].payload["runDelta"]["outputTokens"], 7);
|
||||
assert_eq!(events[0].payload["runDelta"]["cacheReadTokens"], 2);
|
||||
assert_eq!(events[0].payload["runDelta"]["cacheWriteTokens"], 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn maps_usage_and_review_status_but_ignores_inventory_updates() {
|
||||
let usage = normalize(
|
||||
|
|
|
|||
|
|
@ -51,6 +51,10 @@ import {
|
|||
type AcpxSidecarResponse,
|
||||
} from "../drivers/acpx/sidecar-protocol.js";
|
||||
import { safeAcpxLocations } from "./acpx-sidecar-locations.js";
|
||||
import {
|
||||
persistedAcpxTurnUsage,
|
||||
qualifiedAcpxUsageBreakdown,
|
||||
} from "../drivers/acpx/usage-accounting.js";
|
||||
import { validatePrpStructuredRunResult } from "../protocol/replay-contract.js";
|
||||
import type { RunnerToolCall } from "../drivers/runner-tool-bridge.js";
|
||||
import {
|
||||
|
|
@ -314,7 +318,9 @@ async function dispatch(
|
|||
const currentTurnId = boundedIdentity(request.params.turnId, "turnId");
|
||||
turnId = currentTurnId;
|
||||
let runtimeTurn: AcpxRuntimeTurn;
|
||||
let usageBefore: unknown;
|
||||
try {
|
||||
usageBefore = await readSidecarHostStatusWithin(activeHost);
|
||||
runtimeTurn = activeHost.startTurn({
|
||||
requestId: `${runId}:${currentTurnId}`,
|
||||
text: boundedText(request.params.message, "message", 1024 * 1024),
|
||||
|
|
@ -325,7 +331,7 @@ async function dispatch(
|
|||
turnId = null;
|
||||
throw error;
|
||||
}
|
||||
void pumpTurn(currentTurnId, runtimeTurn);
|
||||
void pumpTurn(currentTurnId, runtimeTurn, activeHost, usageBefore);
|
||||
return { turnId: currentTurnId };
|
||||
}
|
||||
if (request.command === "turn.cancel") {
|
||||
|
|
@ -479,6 +485,8 @@ async function dispatch(
|
|||
async function pumpTurn(
|
||||
currentTurnId: string,
|
||||
runtimeTurn: AcpxRuntimeTurn,
|
||||
activeHost: AcpxRuntimeHost,
|
||||
usageBefore: unknown,
|
||||
): Promise<void> {
|
||||
let terminal: Record<string, unknown>;
|
||||
try {
|
||||
|
|
@ -496,6 +504,24 @@ async function pumpTurn(
|
|||
);
|
||||
}
|
||||
const result = await runtimeTurn.result;
|
||||
try {
|
||||
const usage = persistedAcpxTurnUsage(
|
||||
usageBefore,
|
||||
await readSidecarHostStatusWithin(activeHost),
|
||||
runtimeTurn.requestId,
|
||||
);
|
||||
if (usage) {
|
||||
emit(
|
||||
"runtime.event",
|
||||
sanitizeRuntimeEvent(usage as unknown as AcpRuntimeEvent),
|
||||
currentTurnId,
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
// Missing usage must stay unknown, but an accounting read failure must
|
||||
// not replace the provider's authoritative completed/cancelled result.
|
||||
diagnostic("acpx_terminal_usage_unavailable", safeMessage(error));
|
||||
}
|
||||
terminal = boundedSidecarValue(result);
|
||||
} catch (error) {
|
||||
terminal = {
|
||||
|
|
@ -868,7 +894,9 @@ function sanitizeRuntimeStatus(value: unknown): Record<string, unknown> {
|
|||
|
||||
function safeUsage(cost: unknown, breakdown: unknown): Record<string, unknown> {
|
||||
const nativeCost = record(cost);
|
||||
const nativeBreakdown = record(breakdown);
|
||||
const nativeBreakdown = record(
|
||||
qualifiedAcpxUsageBreakdown(initializedAgent, breakdown),
|
||||
);
|
||||
return {
|
||||
cost:
|
||||
cost === undefined || cost === null
|
||||
|
|
|
|||
|
|
@ -0,0 +1,29 @@
|
|||
import { existsSync } from "node:fs";
|
||||
import { isAbsolute } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { evalProviderTransportOptions } from "./eval-provider-runtime.js";
|
||||
|
||||
describe("direct eval provider runtime", () => {
|
||||
it("uses the installed pinned Codex dependency without requiring a global CLI", () => {
|
||||
const options = evalProviderTransportOptions("codex");
|
||||
expect(isAbsolute(options.codexCommand!)).toBe(true);
|
||||
expect(existsSync(options.codexCommand!)).toBe(true);
|
||||
expect(options.codexCommand).toMatch(/@openai\/codex\/bin\/codex\.js$/);
|
||||
expect(options.acpxPermissionMode).toBeUndefined();
|
||||
});
|
||||
|
||||
it("uses explicit unattended permissions only for isolated local eval providers", () => {
|
||||
expect(evalProviderTransportOptions("acpx")).toEqual({
|
||||
acpxPermissionMode: "approve-all",
|
||||
acpxPermissionModePinned: true,
|
||||
});
|
||||
expect(evalProviderTransportOptions("opencode")).toEqual({});
|
||||
expect(evalProviderTransportOptions("claude_managed")).toEqual({});
|
||||
});
|
||||
|
||||
it("bounds AgentCore admission by both the turn budget and delivery timeout", () => {
|
||||
expect(evalProviderTransportOptions("aws_agentcore", 180_000)).toEqual({ turnStartTimeoutMs: 125_000 });
|
||||
expect(evalProviderTransportOptions("aws_agentcore", 10_000)).toEqual({ turnStartTimeoutMs: 10_000 });
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
import type { CapabilityRunnerdCodexTransportOptions } from "../live/runnerd-codex-transport.js";
|
||||
|
||||
/** The direct fixture harness uses the same unattended policy as Runner E2E. */
|
||||
export function evalProviderTransportOptions(
|
||||
provider: "codex" | "opencode" | "acpx" | "claude_managed" | "aws_agentcore",
|
||||
turnTimeoutMs = 120_000,
|
||||
): Pick<CapabilityRunnerdCodexTransportOptions,
|
||||
"codexCommand" | "acpxPermissionMode" | "acpxPermissionModePinned" | "turnStartTimeoutMs"
|
||||
> {
|
||||
if (provider === "aws_agentcore") {
|
||||
// AgentCore's worker permits 120 seconds for invocation delivery. Cold
|
||||
// starts under fleet load must not hit the facade's shorter 30s default.
|
||||
return { turnStartTimeoutMs: Math.min(turnTimeoutMs, 125_000) };
|
||||
}
|
||||
if (provider === "acpx") {
|
||||
// The operator requested this isolated mock-control-plane campaign. This
|
||||
// is an eval policy, not a change to production permission defaults.
|
||||
return { acpxPermissionMode: "approve-all", acpxPermissionModePinned: true };
|
||||
}
|
||||
if (provider !== "codex") return {};
|
||||
const runnerRequire = createRequire(import.meta.url);
|
||||
const codexRequire = createRequire(
|
||||
runnerRequire.resolve("@agentclientprotocol/codex-acp/package.json"),
|
||||
);
|
||||
const manifestPath = codexRequire.resolve("@openai/codex/package.json");
|
||||
const manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as {
|
||||
bin?: string | Record<string, string>;
|
||||
};
|
||||
const executable = typeof manifest.bin === "string" ? manifest.bin : manifest.bin?.codex;
|
||||
if (!executable) throw new Error("Pinned Codex dependency does not expose its executable");
|
||||
return { codexCommand: resolve(dirname(manifestPath), executable) };
|
||||
}
|
||||
|
|
@ -78,6 +78,9 @@ describe("eval-session request contract", () => {
|
|||
)).toContain("Paperclip direct live evaluation");
|
||||
const systemInstructions = evalRuntimeSystemInstructions(context);
|
||||
expect(systemInstructions).toContain("Paperclip direct live evaluation");
|
||||
expect(systemInstructions).toContain("Task-state changes in this mock control plane use finish_task and block_task");
|
||||
expect(systemInstructions).toContain("The current user request defines the work for this turn");
|
||||
expect(systemInstructions).toContain("Do not finish or block the mock task unless the current request asks for that state change");
|
||||
expect(systemInstructions).toContain(
|
||||
`Read-only instruction sibling root: ${context.instructions.bundle.rootPath}`,
|
||||
);
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ import {
|
|||
type EvalSessionRequest,
|
||||
type EvalSessionUsage,
|
||||
} from "./eval-session-contract.js";
|
||||
import { evalProviderTransportOptions } from "./eval-provider-runtime.js";
|
||||
|
||||
interface EvalSessionCliOptions {
|
||||
requestPath: string;
|
||||
|
|
@ -68,6 +69,9 @@ const EVAL_RUNTIME_INSTRUCTIONS = [
|
|||
"",
|
||||
"Use the provided Paperclip semantic tools to inspect and act on the assigned task.",
|
||||
"Treat the seeded control-plane state as authoritative and keep every action within the requested scope.",
|
||||
"The current user request defines the work for this turn. Seeded task descriptions, notes, and past interaction results are background context; they do not supersede that request or establish that a newly requested action has already been performed.",
|
||||
"Task-state changes in this mock control plane use finish_task and block_task. Native paperclip_finish and paperclip_block report the provider run result but do not update the mock task. When asked to finish or block the assigned task, use its task-state semantic operation before reporting the run result.",
|
||||
"Do not finish or block the mock task unless the current request asks for that state change. Ending the provider turn after another requested action does not authorize additional task-state changes or completion comments.",
|
||||
"",
|
||||
].join("\n");
|
||||
|
||||
|
|
@ -293,6 +297,7 @@ export async function runEvalSessionCli(
|
|||
const service = options.serviceFactory?.(runnerdPath) ??
|
||||
new CapabilityLiveSessionService({
|
||||
transportOptions: {
|
||||
...evalProviderTransportOptions(requestedProvider, request.limits.turnTimeoutMs),
|
||||
runnerBinary: runnerdPath,
|
||||
runtimeContext,
|
||||
baseInstructions: evalRuntimeSystemInstructions(runtimeContext),
|
||||
|
|
|
|||
|
|
@ -206,6 +206,9 @@ describe("Codex ACPX runtime adapter", () => {
|
|||
acpxRecordId: "record-1",
|
||||
acpSessionId: "backend-1",
|
||||
agentSessionId: "agent-1",
|
||||
lastRequestId: "run:turn-1",
|
||||
request_token_usage: { prompt: { input_tokens: 12, output_tokens: 30 } },
|
||||
cumulative_cost: { amount: 0.1, currency: "USD" },
|
||||
acpx: {
|
||||
current_model_id: "gpt-5.6-sol",
|
||||
available_models: ["gpt-5.6-sol"],
|
||||
|
|
@ -225,6 +228,9 @@ describe("Codex ACPX runtime adapter", () => {
|
|||
acpxRecordId: "record-1",
|
||||
backendSessionId: "backend-1",
|
||||
agentSessionId: "agent-1",
|
||||
lastRequestId: "run:turn-1",
|
||||
requestTokenUsage: { prompt: { input_tokens: 12, output_tokens: 30 } },
|
||||
usageCost: { amount: 0.1, currency: "USD" },
|
||||
models: {
|
||||
currentModelId: "gpt-5.6-sol",
|
||||
availableModelIds: ["gpt-5.6-sol"],
|
||||
|
|
|
|||
|
|
@ -1148,6 +1148,9 @@ async function persistedRuntimeStatus(
|
|||
acpxRecordId: record.acpxRecordId,
|
||||
backendSessionId: record.acpSessionId,
|
||||
agentSessionId: persistedAgentSessionId,
|
||||
lastRequestId: record.lastRequestId,
|
||||
requestTokenUsage: structuredClone(record.request_token_usage ?? {}),
|
||||
usageCost: structuredClone(record.cumulative_cost),
|
||||
...(currentModelId === undefined && !availableModelIds?.length
|
||||
? {}
|
||||
: {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,132 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
persistedAcpxTurnUsage,
|
||||
qualifiedAcpxUsageBreakdown,
|
||||
} from "./usage-accounting.js";
|
||||
|
||||
describe("qualified ACPX usage", () => {
|
||||
it("accepts Claude's four-field aggregate without inventing extra reasoning", () => {
|
||||
expect(
|
||||
qualifiedAcpxUsageBreakdown("claude", {
|
||||
inputTokens: 12,
|
||||
outputTokens: 30,
|
||||
cachedReadTokens: 40,
|
||||
cachedWriteTokens: 50,
|
||||
}),
|
||||
).toEqual({
|
||||
inputTokens: 12,
|
||||
outputTokens: 30,
|
||||
cachedReadTokens: 40,
|
||||
cachedWriteTokens: 50,
|
||||
thoughtTokens: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not count Codex reasoning twice and knows cache writes are inapplicable", () => {
|
||||
expect(
|
||||
qualifiedAcpxUsageBreakdown("codex", {
|
||||
inputTokens: 12,
|
||||
outputTokens: 30,
|
||||
cachedReadTokens: 40,
|
||||
thoughtTokens: 20,
|
||||
}),
|
||||
).toEqual({
|
||||
inputTokens: 12,
|
||||
outputTokens: 30,
|
||||
cachedReadTokens: 40,
|
||||
cachedWriteTokens: 0,
|
||||
thoughtTokens: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves unknown billable fields and explicit invalid values", () => {
|
||||
expect(qualifiedAcpxUsageBreakdown("claude", { inputTokens: 12 })).toEqual({
|
||||
inputTokens: 12,
|
||||
thoughtTokens: 0,
|
||||
});
|
||||
expect(
|
||||
qualifiedAcpxUsageBreakdown("codex", {
|
||||
inputTokens: null,
|
||||
cachedWriteTokens: null,
|
||||
}),
|
||||
).toEqual({ inputTokens: null, cachedWriteTokens: null, thoughtTokens: 0 });
|
||||
expect(qualifiedAcpxUsageBreakdown("codex", undefined)).toBeUndefined();
|
||||
expect(qualifiedAcpxUsageBreakdown("claude", null)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not infer accounting semantics for Pi or an uninitialized agent", () => {
|
||||
const usage = { inputTokens: 12, outputTokens: 30, thoughtTokens: 20 };
|
||||
expect(qualifiedAcpxUsageBreakdown("pi", usage)).toEqual(usage);
|
||||
expect(qualifiedAcpxUsageBreakdown(null, usage)).toEqual(usage);
|
||||
});
|
||||
});
|
||||
|
||||
describe("persisted terminal ACPX usage", () => {
|
||||
const before = { requestTokenUsage: { previous: { input_tokens: 999 } } };
|
||||
const receipt = {
|
||||
input_tokens: 10,
|
||||
output_tokens: 20,
|
||||
cache_read_input_tokens: 30,
|
||||
thought_tokens: 5,
|
||||
total_tokens: 60,
|
||||
};
|
||||
const after = {
|
||||
lastRequestId: "run:turn-2",
|
||||
usageCost: { amount: 0.5, currency: "USD" },
|
||||
requestTokenUsage: { ...before.requestTokenUsage, current: receipt },
|
||||
};
|
||||
|
||||
it("recovers only the newly persisted prompt receipt, including prompt-response-only usage", () => {
|
||||
expect(persistedAcpxTurnUsage(before, after, "run:turn-2")).toEqual({
|
||||
type: "status",
|
||||
tag: "usage_update",
|
||||
text: "terminal prompt usage",
|
||||
cost: { amount: 0.5, currency: "USD" },
|
||||
breakdown: {
|
||||
inputTokens: 10,
|
||||
outputTokens: 20,
|
||||
cachedReadTokens: 30,
|
||||
cachedWriteTokens: undefined,
|
||||
thoughtTokens: 5,
|
||||
totalTokens: 60,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects old, ambiguous, or differently bound receipts", () => {
|
||||
expect(persistedAcpxTurnUsage(before, after, "run:other")).toBeNull();
|
||||
expect(persistedAcpxTurnUsage(after, after, "run:turn-2")).toBeNull();
|
||||
expect(
|
||||
persistedAcpxTurnUsage(
|
||||
before,
|
||||
{
|
||||
...after,
|
||||
requestTokenUsage: { current: receipt, extra: receipt },
|
||||
},
|
||||
"run:turn-2",
|
||||
),
|
||||
).toBeNull();
|
||||
expect(
|
||||
persistedAcpxTurnUsage(undefined, undefined, "run:turn-2"),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it("works for the first turn without prior usage and retains missing fields", () => {
|
||||
expect(
|
||||
persistedAcpxTurnUsage(
|
||||
{},
|
||||
{
|
||||
lastRequestId: "first",
|
||||
requestTokenUsage: { current: { input_tokens: 10 } },
|
||||
},
|
||||
"first",
|
||||
),
|
||||
).toMatchObject({
|
||||
breakdown: {
|
||||
inputTokens: 10,
|
||||
outputTokens: undefined,
|
||||
cachedReadTokens: undefined,
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
import type { QualifiedAcpxAgent } from "./qualified-profiles.js";
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: {};
|
||||
}
|
||||
|
||||
/** Normalize only semantics established by the pinned, qualified ACP servers. */
|
||||
export function qualifiedAcpxUsageBreakdown(
|
||||
agent: QualifiedAcpxAgent | null,
|
||||
value: unknown,
|
||||
): unknown {
|
||||
if (value === null || value === undefined) return value;
|
||||
const breakdown = record(value);
|
||||
if (agent !== "claude" && agent !== "codex") return breakdown;
|
||||
// Claude SDK aggregate output and Codex ACP toPromptUsage.outputTokens both
|
||||
// INCLUDE reasoning. PRP folds thought into output, so its additive component
|
||||
// is zero here, not the provider's diagnostic reasoning-token subset.
|
||||
return {
|
||||
...breakdown,
|
||||
thoughtTokens: 0,
|
||||
// Codex ACP 1.6.2 has no cache-write billing category. Do not apply this
|
||||
// provider-specific zero to Claude/Pi or to an explicitly invalid value.
|
||||
...(agent === "codex" && breakdown.cachedWriteTokens === undefined
|
||||
? { cachedWriteTokens: 0 }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* ACPX persists terminal prompt-response usage but does not stream it. Recover
|
||||
* exactly the new prompt receipt belonging to this turn, never a prior receipt
|
||||
* or the misleadingly named cumulative_token_usage (which is last-write-wins).
|
||||
*/
|
||||
export function persistedAcpxTurnUsage(
|
||||
before: unknown,
|
||||
after: unknown,
|
||||
requestId: string,
|
||||
): Record<string, unknown> | null {
|
||||
const current = record(after);
|
||||
if (current.lastRequestId !== requestId) return null;
|
||||
const previousReceipts = record(record(before).requestTokenUsage);
|
||||
const receipts = record(current.requestTokenUsage);
|
||||
const added = Object.keys(receipts).filter(
|
||||
(key) => !Object.hasOwn(previousReceipts, key),
|
||||
);
|
||||
if (added.length !== 1) return null;
|
||||
const usage = record(receipts[added[0]!]);
|
||||
return {
|
||||
type: "status",
|
||||
tag: "usage_update",
|
||||
text: "terminal prompt usage",
|
||||
cost: current.usageCost,
|
||||
breakdown: {
|
||||
inputTokens: usage.input_tokens,
|
||||
outputTokens: usage.output_tokens,
|
||||
cachedReadTokens: usage.cache_read_input_tokens,
|
||||
cachedWriteTokens: usage.cache_creation_input_tokens,
|
||||
thoughtTokens: usage.thought_tokens,
|
||||
totalTokens: usage.total_tokens,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -940,6 +940,8 @@ export interface CapabilityRunnerdCodexTransportOptions {
|
|||
/** Provider system instructions supplied by a native execution caller. */
|
||||
baseInstructions?: string;
|
||||
closeGraceMs?: number;
|
||||
/** Bounded provider turn-admission wait; defaults to 30 seconds. */
|
||||
turnStartTimeoutMs?: number;
|
||||
onDiagnostic?: (message: string) => void;
|
||||
onEvidence?: (evidence: Readonly<CapabilityRunnerdProcessEvidence>) => void;
|
||||
stateDirectory?: string;
|
||||
|
|
@ -3737,6 +3739,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
|||
async #startTurn(
|
||||
params: Record<string, unknown>,
|
||||
): Promise<Record<string, unknown>> {
|
||||
const turnStartTimeoutMs = this.options.turnStartTimeoutMs ?? 30_000;
|
||||
if (!Number.isSafeInteger(turnStartTimeoutMs) || turnStartTimeoutMs <= 0) {
|
||||
throw new Error("turnStartTimeoutMs must be a positive safe integer");
|
||||
}
|
||||
const commandDeadline = Date.now() + turnStartTimeoutMs;
|
||||
const input = Array.isArray(params.input) ? params.input.map(record) : [];
|
||||
const message = input
|
||||
.map((item) => (typeof item.text === "string" ? item.text : ""))
|
||||
|
|
@ -3755,7 +3762,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
|||
const startResult = await this.#commandResult("turn.start", {
|
||||
text: message,
|
||||
turnId: pendingTurnId,
|
||||
});
|
||||
}, commandDeadline);
|
||||
const expectedProviderTurnId =
|
||||
typeof startResult.providerTurnId === "string" &&
|
||||
startResult.providerTurnId.length > 0
|
||||
|
|
@ -3788,7 +3795,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
|||
// command's durable result so a delayed prior-turn event cannot satisfy
|
||||
// the new response fence. ACPX echoes the requested identity, while
|
||||
// Codex and OpenCode return their provider-assigned identity.
|
||||
const deadline = Date.now() + 30_000;
|
||||
const deadline = this.options.turnStartTimeoutMs === undefined
|
||||
? Date.now() + 30_000
|
||||
: commandDeadline;
|
||||
const providerTurnStarted = () =>
|
||||
turnStartResponseReady({
|
||||
responseEpoch,
|
||||
|
|
|
|||
Loading…
Reference in New Issue