Add TypeScript PRP replay contracts (#12091)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner needs one typed interpretation of the language-neutral PRP contract. > - The JSON Schemas and fixtures now exist, but TypeScript consumers cannot validate or replay them yet. > - A deterministic reducer must define how duplicate delivery and source gaps affect the projected session. > - Result and question contracts must also validate untrusted provider and user input before later runtime code uses it. > - This pull request adds those TypeScript contracts and replay oracles without adding a process, provider, endpoint, or production behavior. > - The benefit is a reviewable and testable TypeScript foundation for the local runner and transport pull requests. ## Linked Issues or Issue Description **Subsystem affected** This change affects the private `@paperclipai/paperclip-runner` package. It does not change an existing server or adapter execution path. **Problem or motivation** The PRP v1 schemas do not yet provide TypeScript types, runtime validators, normalized result handling, or a deterministic session projection. Later Rust, transport, provider, and server work needs one tested TypeScript oracle instead of separate interpretations. **Proposed solution** Generate a checked-in TypeScript schema bundle from the PRP v1 sources. Add derived types, AJV validation, result and question validation, deterministic replay, a reducer, and generated golden snapshots. Export only these implemented root-package surfaces. **Alternatives considered** The combined runner branch adds the TypeScript contracts together with Rust, providers, semantic authorization, SDKs, labs, and server behavior. That delta is too large for normal review. Handwritten duplicate protocol types would also create a drift risk. **Roadmap alignment** This work supports the governed tool and control-plane direction in `ROADMAP.md`. It does not enable a new production adapter or endpoint. **Additional context** Refs #12087 and #11962. This pull request was prepared on #12087, then rebased onto its squash merge before opening. The current delta against `master` is 37 files. ## What Changed - Added JSON-Schema-derived PRP v1 types and AJV runtime validation. - Added fail-closed required-version checks and cross-envelope binding checks. - Added provider-neutral completion-result and structured-question contracts. - Added normalization for accepted legacy provider result aliases before strict validation. - Added a deterministic session reducer for replay, duplicate delivery, source gaps, requests, items, results, and terminal state. - Added generated replay snapshots and compact parity summaries for six accepted fixtures. - Added schema-bundle, manifest, and replay-golden drift gates. - Added only the root package export. Deferred testing, SDK, evaluation, lab, provider, and browser entry points remain unavailable. ## Verification - `pnpm --filter @paperclipai/paperclip-runner test` passed with 8 protocol tests and 44 TypeScript tests. - `pnpm --filter @paperclipai/paperclip-runner typecheck` passed. - `pnpm --filter @paperclipai/paperclip-runner check:replay-goldens` passed. - `pnpm -r typecheck` passed. - `pnpm build` passed. - `pnpm check:token-gates` passed. - `git diff --check` passed. - The delta against its declared base is 37 files. - `pnpm test:run` was executed locally. The package tests pass, while the macOS repository run retains the unchanged local-environment failures documented on #12087. The complete Linux CI matrix must pass on this commit. - A scoped scan found no secret-like values, internal references, or deferred-provider file names. - Greptile found an unbounded sequence-gap allocation. Commit `4a405c17` caps detailed missing IDs at 256, records the full missing count and truncation state, and rejects sequence values above the exact JavaScript integer range. The focused tests, workspace typecheck, build, and token gates pass after this fix. ## Risks Low production risk. The package remains private. This change adds no process, network endpoint, provider bridge, server integration, database change, or execution selection. The main risk is protocol interpretation drift. Generated schema and replay gates detect that drift. Browser and CSP-specific validator packaging remains deferred to its later package boundary. I checked `ROADMAP.md`. This change defines contracts for planned control-plane work and does not add overlapping product behavior. ## Model Used OpenAI Codex with GPT-5 was used. The exact serving model ID and context size were not exposed. The model used high reasoning, repository tools, GitHub tools, and local code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
3708779501
commit
b2d1673b9e
|
|
@ -2,26 +2,31 @@
|
|||
|
||||
This private workspace package contains the staged Paperclip Runner work.
|
||||
|
||||
This change adds only the language-neutral PRP v1 contract. It does not add a
|
||||
runtime, a server adapter, or a public package export. The schemas and fixtures
|
||||
are safe to review and merge before production behavior exists.
|
||||
The package currently exposes only the language-neutral PRP v1 TypeScript
|
||||
contract, provider-neutral structured questions and responses, deterministic
|
||||
fixture validation/replay, structured-result normalization, and the session
|
||||
reducer oracle. It does not add a runner process, provider transport, server
|
||||
adapter, semantic-tool authorization, or production Paperclip behavior.
|
||||
|
||||
The first provider scope is Codex. The protocol contains provider-neutral event
|
||||
and semantic receipt shapes, but their presence does not authorize a tool or
|
||||
enable a provider.
|
||||
|
||||
Run the protocol gate with:
|
||||
The root export is intentionally narrow. The `./testing` entry point and package
|
||||
release boundary will arrive with the later package-boundary change.
|
||||
|
||||
Run the complete contract gate with:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/paperclip-runner check:protocol
|
||||
```
|
||||
|
||||
Use `generate:protocol-manifest` after a schema or fixture change. Commit the
|
||||
updated manifest with its source change. Do not edit the manifest by hand.
|
||||
The gate compiles every schema with AJV 2020-12. It validates accepted replay,
|
||||
question, and cross-language conformance fixtures against the declared schemas.
|
||||
It also proves that the unsupported required-version fixture is rejected.
|
||||
Use `generate:protocol-manifest` after a schema or fixture change,
|
||||
`generate:protocol-types` after a schema change, and
|
||||
`generate:replay-goldens` after an intentional reducer change. Commit generated
|
||||
outputs with their sources; do not edit them by hand.
|
||||
|
||||
The next pull request will add TypeScript validation and replay behavior. The
|
||||
public root and `./testing` exports will arrive with the package boundary pull
|
||||
request.
|
||||
The gate compiles every schema with AJV 2020-12, validates accepted fixtures,
|
||||
rejects unsupported required versions, checks generated TypeScript schema
|
||||
drift, runs the TypeScript contract tests, and compares reducer snapshots and
|
||||
parity summaries byte-for-byte with their checked-in golden files.
|
||||
|
|
|
|||
|
|
@ -7,15 +7,37 @@
|
|||
"node": ">=24.11.0"
|
||||
},
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"import": "./dist/index.js"
|
||||
}
|
||||
},
|
||||
"files": [
|
||||
"dist",
|
||||
"protocol",
|
||||
"README.md"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "pnpm run check:protocol-manifest",
|
||||
"typecheck": "node --check scripts/protocol-contract.mjs && node --check scripts/generate-protocol-manifest.mjs",
|
||||
"test": "node --test test/protocol-contract.test.mjs",
|
||||
"build": "pnpm run check:protocol-manifest && pnpm run build:typescript && node scripts/generate-replay-goldens.mjs --check",
|
||||
"build:typescript": "pnpm run check:protocol-types && tsc -p tsconfig.json",
|
||||
"typecheck": "node --check scripts/protocol-contract.mjs && node --check scripts/generate-protocol-manifest.mjs && node --check scripts/generate-protocol-schema-module.mjs && node --check scripts/generate-replay-goldens.mjs && pnpm run check:protocol-types && tsc -p tsconfig.json --noEmit",
|
||||
"test": "node --test test/protocol-contract.test.mjs && vitest run",
|
||||
"generate:protocol-manifest": "node scripts/generate-protocol-manifest.mjs",
|
||||
"check:protocol-manifest": "node scripts/generate-protocol-manifest.mjs --check",
|
||||
"check:protocol": "pnpm run typecheck && pnpm run check:protocol-manifest && pnpm test"
|
||||
"generate:protocol-types": "node scripts/generate-protocol-schema-module.mjs",
|
||||
"check:protocol-types": "node scripts/generate-protocol-schema-module.mjs --check",
|
||||
"generate:replay-goldens": "pnpm run build:typescript && node scripts/generate-replay-goldens.mjs",
|
||||
"check:replay-goldens": "pnpm run build:typescript && node scripts/generate-replay-goldens.mjs --check",
|
||||
"check:protocol": "pnpm run typecheck && pnpm run check:protocol-manifest && pnpm test && pnpm run check:replay-goldens"
|
||||
},
|
||||
"dependencies": {
|
||||
"ajv": "^8.20.0",
|
||||
"json-schema-to-ts": "^3.1.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"ajv": "^8.20.0"
|
||||
"@types/node": "^24.0.0",
|
||||
"typescript": "^5.7.3",
|
||||
"vitest": "^4.1.10"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -30,3 +30,8 @@ The conformance manifest records every source file and its SHA-256 digest. Run
|
|||
runs the same generator with `--check` to reject drift. This check also compiles
|
||||
the JSON Schemas and validates every replay, question, and cross-language
|
||||
conformance fixture against its declared schema.
|
||||
|
||||
The files in `fixtures/replay/golden/` are deterministic reducer oracles. Each
|
||||
accepted replay fixture has a complete session snapshot and a compact parity
|
||||
summary. `pnpm generate:replay-goldens` updates them after an intentional
|
||||
reducer change; package build and CI fail when they drift.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,130 @@
|
|||
{
|
||||
"schema": "paperclip.prp.session-snapshot.v1",
|
||||
"fixtureName": "Duplicate event",
|
||||
"identity": {
|
||||
"schema": "paperclip.prp.identity.v1",
|
||||
"companyId": "company_replay",
|
||||
"issueId": "issue_replay_duplicate",
|
||||
"runId": "run_replay_duplicate",
|
||||
"environmentLeaseId": "lease_replay_duplicate",
|
||||
"runnerInstanceId": "runner_replay",
|
||||
"normalizedSessionId": "session_replay_duplicate",
|
||||
"driverSessionId": "driver_replay_duplicate"
|
||||
},
|
||||
"capabilities": {
|
||||
"schema": "paperclip.prp.capabilities.v1",
|
||||
"sessionReusePolicy": "new_per_run",
|
||||
"driver": {
|
||||
"kind": "fake",
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"steer": false,
|
||||
"interrupt": true,
|
||||
"resume": true,
|
||||
"runtimeRequests": false,
|
||||
"structuredResult": true,
|
||||
"typedEvents": true
|
||||
},
|
||||
"runPhase": "terminal",
|
||||
"sessionState": "running",
|
||||
"turnState": "completed",
|
||||
"activeTurnId": null,
|
||||
"items": [
|
||||
{
|
||||
"itemId": "item_replay_duplicate",
|
||||
"kind": "assistant_message",
|
||||
"status": "completed",
|
||||
"text": "Exactly one projection."
|
||||
}
|
||||
],
|
||||
"requests": [],
|
||||
"proposedResult": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "done",
|
||||
"summary": "Duplicate delivery was idempotent.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "replay-contract-v1",
|
||||
"objectiveSatisfied": true,
|
||||
"criteria": [],
|
||||
"remainingWork": []
|
||||
},
|
||||
"evidence": [],
|
||||
"verification": [],
|
||||
"attentionRequests": [],
|
||||
"artifacts": []
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded",
|
||||
"reportedWorkDisposition": "done"
|
||||
},
|
||||
"timeline": [
|
||||
{
|
||||
"position": 1,
|
||||
"sourceEventId": "event_duplicate_01",
|
||||
"sourceSeq": 1,
|
||||
"eventType": "session.started",
|
||||
"emittedAt": "2026-08-07T12:30:00.010Z",
|
||||
"summary": "session started"
|
||||
},
|
||||
{
|
||||
"position": 2,
|
||||
"sourceEventId": "event_duplicate_02",
|
||||
"sourceSeq": 2,
|
||||
"eventType": "turn.started",
|
||||
"emittedAt": "2026-08-07T12:30:00.020Z",
|
||||
"summary": "turn started"
|
||||
},
|
||||
{
|
||||
"position": 3,
|
||||
"sourceEventId": "event_duplicate_03",
|
||||
"sourceSeq": 3,
|
||||
"eventType": "item.completed",
|
||||
"emittedAt": "2026-08-07T12:30:00.030Z",
|
||||
"itemId": "item_replay_duplicate",
|
||||
"summary": "Completed assistant_message"
|
||||
},
|
||||
{
|
||||
"position": 4,
|
||||
"sourceEventId": "event_duplicate_04",
|
||||
"sourceSeq": 4,
|
||||
"eventType": "run.result.proposed",
|
||||
"emittedAt": "2026-08-07T12:30:00.040Z",
|
||||
"summary": "Duplicate delivery was idempotent."
|
||||
},
|
||||
{
|
||||
"position": 5,
|
||||
"sourceEventId": "event_duplicate_05",
|
||||
"sourceSeq": 5,
|
||||
"eventType": "turn.completed",
|
||||
"emittedAt": "2026-08-07T12:30:00.050Z",
|
||||
"summary": "turn completed"
|
||||
},
|
||||
{
|
||||
"position": 6,
|
||||
"sourceEventId": "event_duplicate_06",
|
||||
"sourceSeq": 6,
|
||||
"eventType": "run.terminal",
|
||||
"emittedAt": "2026-08-07T12:30:00.060Z",
|
||||
"summary": "Run succeeded"
|
||||
}
|
||||
],
|
||||
"sourceCursors": {
|
||||
"runner:runner_replay": 6
|
||||
},
|
||||
"processedEventIds": [
|
||||
"event_duplicate_01",
|
||||
"event_duplicate_02",
|
||||
"event_duplicate_03",
|
||||
"event_duplicate_04",
|
||||
"event_duplicate_05",
|
||||
"event_duplicate_06"
|
||||
],
|
||||
"duplicateEventIds": [
|
||||
"event_duplicate_03"
|
||||
],
|
||||
"outOfOrderEventIds": [],
|
||||
"gaps": [],
|
||||
"integrity": "complete"
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
{
|
||||
"runId": "run_replay_duplicate",
|
||||
"integrity": "complete",
|
||||
"timelineCount": 6,
|
||||
"duplicateEventIds": [
|
||||
"event_duplicate_03"
|
||||
],
|
||||
"gaps": [],
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded"
|
||||
}
|
||||
|
|
@ -0,0 +1,143 @@
|
|||
{
|
||||
"schema": "paperclip.prp.session-snapshot.v1",
|
||||
"fixtureName": "Failed run",
|
||||
"identity": {
|
||||
"schema": "paperclip.prp.identity.v1",
|
||||
"companyId": "company_replay",
|
||||
"issueId": "issue_replay_failed",
|
||||
"runId": "run_replay_failed",
|
||||
"environmentLeaseId": "lease_replay_failed",
|
||||
"runnerInstanceId": "runner_replay",
|
||||
"normalizedSessionId": "session_replay_failed",
|
||||
"driverSessionId": "driver_replay_failed"
|
||||
},
|
||||
"capabilities": {
|
||||
"schema": "paperclip.prp.capabilities.v1",
|
||||
"sessionReusePolicy": "new_per_run",
|
||||
"driver": {
|
||||
"kind": "fake",
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"steer": true,
|
||||
"interrupt": true,
|
||||
"resume": true,
|
||||
"runtimeRequests": true,
|
||||
"structuredResult": true,
|
||||
"typedEvents": true
|
||||
},
|
||||
"runPhase": "terminal",
|
||||
"sessionState": "running",
|
||||
"turnState": "failed",
|
||||
"activeTurnId": null,
|
||||
"items": [
|
||||
{
|
||||
"itemId": "item_failed_command",
|
||||
"kind": "command",
|
||||
"status": "failed",
|
||||
"text": ""
|
||||
}
|
||||
],
|
||||
"requests": [],
|
||||
"proposedResult": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "yielded",
|
||||
"summary": "The scripted command failed and needs a retry.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "replay-contract-v1",
|
||||
"objectiveSatisfied": false,
|
||||
"criteria": [],
|
||||
"remainingWork": [
|
||||
{
|
||||
"description": "Retry the failed command",
|
||||
"blocksCompletion": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"evidence": [],
|
||||
"verification": [
|
||||
{
|
||||
"commandOrCheck": "fake command",
|
||||
"status": "failed"
|
||||
}
|
||||
],
|
||||
"attentionRequests": [],
|
||||
"artifacts": [],
|
||||
"continuation": {
|
||||
"kind": "retry",
|
||||
"summary": "Retry after correcting the command",
|
||||
"idempotencyKey": "retry_replay_failed"
|
||||
}
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "failed",
|
||||
"runTerminalState": "failed",
|
||||
"reportedWorkDisposition": "yielded"
|
||||
},
|
||||
"timeline": [
|
||||
{
|
||||
"position": 1,
|
||||
"sourceEventId": "event_failed_01",
|
||||
"sourceSeq": 1,
|
||||
"eventType": "session.started",
|
||||
"emittedAt": "2026-08-07T12:10:00.020Z",
|
||||
"summary": "session started"
|
||||
},
|
||||
{
|
||||
"position": 2,
|
||||
"sourceEventId": "event_failed_02",
|
||||
"sourceSeq": 2,
|
||||
"eventType": "turn.started",
|
||||
"emittedAt": "2026-08-07T12:10:00.030Z",
|
||||
"summary": "turn started"
|
||||
},
|
||||
{
|
||||
"position": 3,
|
||||
"sourceEventId": "event_failed_03",
|
||||
"sourceSeq": 3,
|
||||
"eventType": "item.failed",
|
||||
"emittedAt": "2026-08-07T12:10:00.040Z",
|
||||
"itemId": "item_failed_command",
|
||||
"summary": "Command exited with code 1"
|
||||
},
|
||||
{
|
||||
"position": 4,
|
||||
"sourceEventId": "event_failed_04",
|
||||
"sourceSeq": 4,
|
||||
"eventType": "run.result.proposed",
|
||||
"emittedAt": "2026-08-07T12:10:00.050Z",
|
||||
"summary": "The scripted command failed and needs a retry."
|
||||
},
|
||||
{
|
||||
"position": 5,
|
||||
"sourceEventId": "event_failed_05",
|
||||
"sourceSeq": 5,
|
||||
"eventType": "turn.failed",
|
||||
"emittedAt": "2026-08-07T12:10:00.060Z",
|
||||
"summary": "turn failed"
|
||||
},
|
||||
{
|
||||
"position": 6,
|
||||
"sourceEventId": "event_failed_06",
|
||||
"sourceSeq": 6,
|
||||
"eventType": "run.terminal",
|
||||
"emittedAt": "2026-08-07T12:10:00.070Z",
|
||||
"summary": "Run failed"
|
||||
}
|
||||
],
|
||||
"sourceCursors": {
|
||||
"runner:runner_replay": 6
|
||||
},
|
||||
"processedEventIds": [
|
||||
"event_failed_01",
|
||||
"event_failed_02",
|
||||
"event_failed_03",
|
||||
"event_failed_04",
|
||||
"event_failed_05",
|
||||
"event_failed_06"
|
||||
],
|
||||
"duplicateEventIds": [],
|
||||
"outOfOrderEventIds": [],
|
||||
"gaps": [],
|
||||
"integrity": "complete"
|
||||
}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"runId": "run_replay_failed",
|
||||
"integrity": "complete",
|
||||
"timelineCount": 6,
|
||||
"duplicateEventIds": [],
|
||||
"gaps": [],
|
||||
"turnTerminalState": "failed",
|
||||
"runTerminalState": "failed"
|
||||
}
|
||||
|
|
@ -0,0 +1,166 @@
|
|||
{
|
||||
"schema": "paperclip.prp.session-snapshot.v1",
|
||||
"fixtureName": "Happy path",
|
||||
"identity": {
|
||||
"schema": "paperclip.prp.identity.v1",
|
||||
"companyId": "company_replay",
|
||||
"issueId": "issue_replay_happy",
|
||||
"runId": "run_replay_happy",
|
||||
"environmentLeaseId": "lease_replay_happy",
|
||||
"runnerInstanceId": "runner_replay",
|
||||
"normalizedSessionId": "session_replay_happy",
|
||||
"driverSessionId": "driver_replay_happy",
|
||||
"providerSessionId": "provider_replay_happy"
|
||||
},
|
||||
"capabilities": {
|
||||
"schema": "paperclip.prp.capabilities.v1",
|
||||
"sessionReusePolicy": "new_per_run",
|
||||
"driver": {
|
||||
"kind": "fake",
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"steer": true,
|
||||
"interrupt": true,
|
||||
"resume": true,
|
||||
"runtimeRequests": true,
|
||||
"structuredResult": true,
|
||||
"typedEvents": true,
|
||||
"unsupported": []
|
||||
},
|
||||
"runPhase": "terminal",
|
||||
"sessionState": "running",
|
||||
"turnState": "completed",
|
||||
"activeTurnId": null,
|
||||
"items": [
|
||||
{
|
||||
"itemId": "item_replay_message",
|
||||
"kind": "assistant_message",
|
||||
"status": "completed",
|
||||
"text": "Validated fixture. Replay complete."
|
||||
}
|
||||
],
|
||||
"requests": [],
|
||||
"proposedResult": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "done",
|
||||
"summary": "The scripted run completed successfully.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "replay-contract-v1",
|
||||
"objectiveSatisfied": true,
|
||||
"criteria": [],
|
||||
"remainingWork": []
|
||||
},
|
||||
"evidence": [],
|
||||
"verification": [
|
||||
{
|
||||
"commandOrCheck": "fixture replay",
|
||||
"status": "passed"
|
||||
}
|
||||
],
|
||||
"attentionRequests": [],
|
||||
"artifacts": []
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded",
|
||||
"reportedWorkDisposition": "done",
|
||||
"workAssessmentId": "assessment_replay_happy",
|
||||
"statusDecisionId": "decision_replay_happy"
|
||||
},
|
||||
"timeline": [
|
||||
{
|
||||
"position": 1,
|
||||
"sourceEventId": "event_happy_01",
|
||||
"sourceSeq": 1,
|
||||
"eventType": "runtime.phase.changed",
|
||||
"emittedAt": "2026-08-07T12:00:00.030Z",
|
||||
"summary": "Phase changed to workspace preparing"
|
||||
},
|
||||
{
|
||||
"position": 2,
|
||||
"sourceEventId": "event_happy_02",
|
||||
"sourceSeq": 2,
|
||||
"eventType": "session.started",
|
||||
"emittedAt": "2026-08-07T12:00:00.040Z",
|
||||
"summary": "session started"
|
||||
},
|
||||
{
|
||||
"position": 3,
|
||||
"sourceEventId": "event_happy_03",
|
||||
"sourceSeq": 3,
|
||||
"eventType": "turn.started",
|
||||
"emittedAt": "2026-08-07T12:00:00.050Z",
|
||||
"summary": "turn started"
|
||||
},
|
||||
{
|
||||
"position": 4,
|
||||
"sourceEventId": "event_happy_04",
|
||||
"sourceSeq": 4,
|
||||
"eventType": "item.started",
|
||||
"emittedAt": "2026-08-07T12:00:00.060Z",
|
||||
"itemId": "item_replay_message",
|
||||
"summary": "Started assistant_message"
|
||||
},
|
||||
{
|
||||
"position": 5,
|
||||
"sourceEventId": "event_happy_05",
|
||||
"sourceSeq": 5,
|
||||
"eventType": "item.delta",
|
||||
"emittedAt": "2026-08-07T12:00:00.070Z",
|
||||
"itemId": "item_replay_message",
|
||||
"summary": "Validated fixture. "
|
||||
},
|
||||
{
|
||||
"position": 6,
|
||||
"sourceEventId": "event_happy_06",
|
||||
"sourceSeq": 6,
|
||||
"eventType": "item.completed",
|
||||
"emittedAt": "2026-08-07T12:00:00.080Z",
|
||||
"itemId": "item_replay_message",
|
||||
"summary": "Completed assistant_message"
|
||||
},
|
||||
{
|
||||
"position": 7,
|
||||
"sourceEventId": "event_happy_07",
|
||||
"sourceSeq": 7,
|
||||
"eventType": "run.result.proposed",
|
||||
"emittedAt": "2026-08-07T12:00:00.090Z",
|
||||
"summary": "The scripted run completed successfully."
|
||||
},
|
||||
{
|
||||
"position": 8,
|
||||
"sourceEventId": "event_happy_08",
|
||||
"sourceSeq": 8,
|
||||
"eventType": "turn.completed",
|
||||
"emittedAt": "2026-08-07T12:00:00.100Z",
|
||||
"summary": "turn completed"
|
||||
},
|
||||
{
|
||||
"position": 9,
|
||||
"sourceEventId": "event_happy_09",
|
||||
"sourceSeq": 9,
|
||||
"eventType": "run.terminal",
|
||||
"emittedAt": "2026-08-07T12:00:00.110Z",
|
||||
"summary": "Run succeeded"
|
||||
}
|
||||
],
|
||||
"sourceCursors": {
|
||||
"runner:runner_replay": 9
|
||||
},
|
||||
"processedEventIds": [
|
||||
"event_happy_01",
|
||||
"event_happy_02",
|
||||
"event_happy_03",
|
||||
"event_happy_04",
|
||||
"event_happy_05",
|
||||
"event_happy_06",
|
||||
"event_happy_07",
|
||||
"event_happy_08",
|
||||
"event_happy_09"
|
||||
],
|
||||
"duplicateEventIds": [],
|
||||
"outOfOrderEventIds": [],
|
||||
"gaps": [],
|
||||
"integrity": "complete"
|
||||
}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"runId": "run_replay_happy",
|
||||
"integrity": "complete",
|
||||
"timelineCount": 9,
|
||||
"duplicateEventIds": [],
|
||||
"gaps": [],
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded"
|
||||
}
|
||||
|
|
@ -0,0 +1,138 @@
|
|||
{
|
||||
"schema": "paperclip.prp.session-snapshot.v1",
|
||||
"fixtureName": "Interrupted run",
|
||||
"identity": {
|
||||
"schema": "paperclip.prp.identity.v1",
|
||||
"companyId": "company_replay",
|
||||
"issueId": "issue_replay_interrupted",
|
||||
"runId": "run_replay_interrupted",
|
||||
"environmentLeaseId": "lease_replay_interrupted",
|
||||
"runnerInstanceId": "runner_replay",
|
||||
"normalizedSessionId": "session_replay_interrupted",
|
||||
"driverSessionId": "driver_replay_interrupted"
|
||||
},
|
||||
"capabilities": {
|
||||
"schema": "paperclip.prp.capabilities.v1",
|
||||
"sessionReusePolicy": "reuse_per_issue",
|
||||
"driver": {
|
||||
"kind": "fake",
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"steer": true,
|
||||
"interrupt": true,
|
||||
"resume": true,
|
||||
"runtimeRequests": true,
|
||||
"structuredResult": true,
|
||||
"typedEvents": true
|
||||
},
|
||||
"runPhase": "terminal",
|
||||
"sessionState": "running",
|
||||
"turnState": "interrupted",
|
||||
"activeTurnId": null,
|
||||
"items": [],
|
||||
"requests": [
|
||||
{
|
||||
"requestId": "request_interrupted_permission",
|
||||
"requestKind": "runtime",
|
||||
"type": "permission",
|
||||
"status": "pending",
|
||||
"prompt": "Allow the fake command?"
|
||||
}
|
||||
],
|
||||
"proposedResult": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "yielded",
|
||||
"summary": "The turn was interrupted.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "replay-contract-v1",
|
||||
"objectiveSatisfied": false,
|
||||
"criteria": [],
|
||||
"remainingWork": [
|
||||
{
|
||||
"description": "Resume the interrupted turn",
|
||||
"blocksCompletion": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"evidence": [],
|
||||
"verification": [],
|
||||
"attentionRequests": [],
|
||||
"artifacts": [],
|
||||
"continuation": {
|
||||
"kind": "same_agent",
|
||||
"summary": "Resume in the same normalized session",
|
||||
"idempotencyKey": "resume_replay_interrupted"
|
||||
}
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "interrupted",
|
||||
"runTerminalState": "cancelled",
|
||||
"reportedWorkDisposition": "yielded"
|
||||
},
|
||||
"timeline": [
|
||||
{
|
||||
"position": 1,
|
||||
"sourceEventId": "event_interrupted_01",
|
||||
"sourceSeq": 1,
|
||||
"eventType": "session.started",
|
||||
"emittedAt": "2026-08-07T12:20:00.020Z",
|
||||
"summary": "session started"
|
||||
},
|
||||
{
|
||||
"position": 2,
|
||||
"sourceEventId": "event_interrupted_02",
|
||||
"sourceSeq": 2,
|
||||
"eventType": "turn.started",
|
||||
"emittedAt": "2026-08-07T12:20:00.030Z",
|
||||
"summary": "turn started"
|
||||
},
|
||||
{
|
||||
"position": 3,
|
||||
"sourceEventId": "event_interrupted_03",
|
||||
"sourceSeq": 3,
|
||||
"eventType": "runtime_request.created",
|
||||
"emittedAt": "2026-08-07T12:20:00.040Z",
|
||||
"summary": "permission: Allow the fake command?"
|
||||
},
|
||||
{
|
||||
"position": 4,
|
||||
"sourceEventId": "event_interrupted_04",
|
||||
"sourceSeq": 4,
|
||||
"eventType": "turn.interrupted",
|
||||
"emittedAt": "2026-08-07T12:20:00.060Z",
|
||||
"summary": "turn interrupted"
|
||||
},
|
||||
{
|
||||
"position": 5,
|
||||
"sourceEventId": "event_interrupted_05",
|
||||
"sourceSeq": 5,
|
||||
"eventType": "run.result.proposed",
|
||||
"emittedAt": "2026-08-07T12:20:00.070Z",
|
||||
"summary": "The turn was interrupted."
|
||||
},
|
||||
{
|
||||
"position": 6,
|
||||
"sourceEventId": "event_interrupted_06",
|
||||
"sourceSeq": 6,
|
||||
"eventType": "run.terminal",
|
||||
"emittedAt": "2026-08-07T12:20:00.080Z",
|
||||
"summary": "Run cancelled"
|
||||
}
|
||||
],
|
||||
"sourceCursors": {
|
||||
"runner:runner_replay": 6
|
||||
},
|
||||
"processedEventIds": [
|
||||
"event_interrupted_01",
|
||||
"event_interrupted_02",
|
||||
"event_interrupted_03",
|
||||
"event_interrupted_04",
|
||||
"event_interrupted_05",
|
||||
"event_interrupted_06"
|
||||
],
|
||||
"duplicateEventIds": [],
|
||||
"outOfOrderEventIds": [],
|
||||
"gaps": [],
|
||||
"integrity": "complete"
|
||||
}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"runId": "run_replay_interrupted",
|
||||
"integrity": "complete",
|
||||
"timelineCount": 6,
|
||||
"duplicateEventIds": [],
|
||||
"gaps": [],
|
||||
"turnTerminalState": "interrupted",
|
||||
"runTerminalState": "cancelled"
|
||||
}
|
||||
|
|
@ -0,0 +1,127 @@
|
|||
{
|
||||
"schema": "paperclip.prp.session-snapshot.v1",
|
||||
"fixtureName": "Source sequence gap",
|
||||
"identity": {
|
||||
"schema": "paperclip.prp.identity.v1",
|
||||
"companyId": "company_replay",
|
||||
"issueId": "issue_replay_gap",
|
||||
"runId": "run_replay_gap",
|
||||
"environmentLeaseId": "lease_replay_gap",
|
||||
"runnerInstanceId": "runner_replay",
|
||||
"normalizedSessionId": "session_replay_gap",
|
||||
"driverSessionId": "driver_replay_gap"
|
||||
},
|
||||
"capabilities": {
|
||||
"schema": "paperclip.prp.capabilities.v1",
|
||||
"sessionReusePolicy": "new_per_run",
|
||||
"driver": {
|
||||
"kind": "fake",
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"steer": false,
|
||||
"interrupt": false,
|
||||
"resume": true,
|
||||
"runtimeRequests": false,
|
||||
"structuredResult": true,
|
||||
"typedEvents": true
|
||||
},
|
||||
"runPhase": "terminal",
|
||||
"sessionState": "running",
|
||||
"turnState": "completed",
|
||||
"activeTurnId": null,
|
||||
"items": [],
|
||||
"requests": [],
|
||||
"proposedResult": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "needs_review",
|
||||
"summary": "The source gap requires reconciliation.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "replay-contract-v1",
|
||||
"objectiveSatisfied": false,
|
||||
"criteria": [],
|
||||
"remainingWork": [
|
||||
{
|
||||
"description": "Replay missing source event",
|
||||
"blocksCompletion": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"evidence": [],
|
||||
"verification": [],
|
||||
"attentionRequests": [],
|
||||
"artifacts": []
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded",
|
||||
"reportedWorkDisposition": "needs_review"
|
||||
},
|
||||
"timeline": [
|
||||
{
|
||||
"position": 1,
|
||||
"sourceEventId": "event_gap_01",
|
||||
"sourceSeq": 1,
|
||||
"eventType": "session.started",
|
||||
"emittedAt": "2026-08-07T12:40:00.010Z",
|
||||
"summary": "session started"
|
||||
},
|
||||
{
|
||||
"position": 2,
|
||||
"sourceEventId": "event_gap_02",
|
||||
"sourceSeq": 2,
|
||||
"eventType": "turn.started",
|
||||
"emittedAt": "2026-08-07T12:40:00.020Z",
|
||||
"summary": "turn started"
|
||||
},
|
||||
{
|
||||
"position": 3,
|
||||
"sourceEventId": "event_gap_04",
|
||||
"sourceSeq": 4,
|
||||
"eventType": "run.result.proposed",
|
||||
"emittedAt": "2026-08-07T12:40:00.040Z",
|
||||
"summary": "The source gap requires reconciliation."
|
||||
},
|
||||
{
|
||||
"position": 4,
|
||||
"sourceEventId": "event_gap_05",
|
||||
"sourceSeq": 5,
|
||||
"eventType": "turn.completed",
|
||||
"emittedAt": "2026-08-07T12:40:00.050Z",
|
||||
"summary": "turn completed"
|
||||
},
|
||||
{
|
||||
"position": 5,
|
||||
"sourceEventId": "event_gap_06",
|
||||
"sourceSeq": 6,
|
||||
"eventType": "run.terminal",
|
||||
"emittedAt": "2026-08-07T12:40:00.060Z",
|
||||
"summary": "Run succeeded"
|
||||
}
|
||||
],
|
||||
"sourceCursors": {
|
||||
"runner:runner_replay": 6
|
||||
},
|
||||
"processedEventIds": [
|
||||
"event_gap_01",
|
||||
"event_gap_02",
|
||||
"event_gap_04",
|
||||
"event_gap_05",
|
||||
"event_gap_06"
|
||||
],
|
||||
"duplicateEventIds": [],
|
||||
"outOfOrderEventIds": [],
|
||||
"gaps": [
|
||||
{
|
||||
"sourceKey": "runner:runner_replay",
|
||||
"expected": 3,
|
||||
"received": 4,
|
||||
"missingCount": 1,
|
||||
"missing": [
|
||||
3
|
||||
],
|
||||
"truncated": false
|
||||
}
|
||||
],
|
||||
"integrity": "gap_detected"
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"runId": "run_replay_gap",
|
||||
"integrity": "gap_detected",
|
||||
"timelineCount": 5,
|
||||
"duplicateEventIds": [],
|
||||
"gaps": [
|
||||
{
|
||||
"sourceKey": "runner:runner_replay",
|
||||
"expected": 3,
|
||||
"received": 4,
|
||||
"missingCount": 1,
|
||||
"missing": [
|
||||
3
|
||||
],
|
||||
"truncated": false
|
||||
}
|
||||
],
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded"
|
||||
}
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"schema": "paperclip.prp.session-snapshot.v1",
|
||||
"fixtureName": "Unknown optional fields",
|
||||
"identity": {
|
||||
"schema": "paperclip.prp.identity.v1",
|
||||
"companyId": "company_replay",
|
||||
"issueId": "issue_replay_future",
|
||||
"runId": "run_replay_future",
|
||||
"environmentLeaseId": "lease_replay_future",
|
||||
"runnerInstanceId": "runner_replay",
|
||||
"normalizedSessionId": "session_replay_future",
|
||||
"driverSessionId": "driver_replay_future",
|
||||
"futureIdentityLabel": "preserved"
|
||||
},
|
||||
"capabilities": {
|
||||
"schema": "paperclip.prp.capabilities.v1",
|
||||
"sessionReusePolicy": "new_per_run",
|
||||
"driver": {
|
||||
"kind": "fake",
|
||||
"version": "1.1.0",
|
||||
"futureDriverField": true
|
||||
},
|
||||
"steer": true,
|
||||
"interrupt": true,
|
||||
"resume": true,
|
||||
"runtimeRequests": true,
|
||||
"structuredResult": true,
|
||||
"typedEvents": true,
|
||||
"futureCapability": {
|
||||
"mode": "optional"
|
||||
}
|
||||
},
|
||||
"runPhase": "terminal",
|
||||
"sessionState": "running",
|
||||
"turnState": "completed",
|
||||
"activeTurnId": null,
|
||||
"items": [],
|
||||
"requests": [],
|
||||
"proposedResult": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "done",
|
||||
"summary": "Unknown optional fields did not change the v1 projection.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "replay-contract-v1",
|
||||
"objectiveSatisfied": true,
|
||||
"criteria": [],
|
||||
"remainingWork": []
|
||||
},
|
||||
"evidence": [],
|
||||
"verification": [],
|
||||
"attentionRequests": [],
|
||||
"artifacts": [],
|
||||
"futureResultAnnotation": {
|
||||
"safe": true
|
||||
}
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded",
|
||||
"reportedWorkDisposition": "done",
|
||||
"futureTerminalReason": "optional"
|
||||
},
|
||||
"timeline": [
|
||||
{
|
||||
"position": 1,
|
||||
"sourceEventId": "event_future_01",
|
||||
"sourceSeq": 1,
|
||||
"eventType": "session.started",
|
||||
"emittedAt": "2026-08-07T12:50:00.010Z",
|
||||
"summary": "session started"
|
||||
},
|
||||
{
|
||||
"position": 2,
|
||||
"sourceEventId": "event_future_02",
|
||||
"sourceSeq": 2,
|
||||
"eventType": "run.result.proposed",
|
||||
"emittedAt": "2026-08-07T12:50:00.020Z",
|
||||
"summary": "Unknown optional fields did not change the v1 projection."
|
||||
},
|
||||
{
|
||||
"position": 3,
|
||||
"sourceEventId": "event_future_03",
|
||||
"sourceSeq": 3,
|
||||
"eventType": "turn.completed",
|
||||
"emittedAt": "2026-08-07T12:50:00.030Z",
|
||||
"summary": "turn completed"
|
||||
},
|
||||
{
|
||||
"position": 4,
|
||||
"sourceEventId": "event_future_04",
|
||||
"sourceSeq": 4,
|
||||
"eventType": "run.terminal",
|
||||
"emittedAt": "2026-08-07T12:50:00.040Z",
|
||||
"summary": "Run succeeded"
|
||||
}
|
||||
],
|
||||
"sourceCursors": {
|
||||
"runner:runner_replay": 4
|
||||
},
|
||||
"processedEventIds": [
|
||||
"event_future_01",
|
||||
"event_future_02",
|
||||
"event_future_03",
|
||||
"event_future_04"
|
||||
],
|
||||
"duplicateEventIds": [],
|
||||
"outOfOrderEventIds": [],
|
||||
"gaps": [],
|
||||
"integrity": "complete"
|
||||
}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"runId": "run_replay_future",
|
||||
"integrity": "complete",
|
||||
"timelineCount": 4,
|
||||
"duplicateEventIds": [],
|
||||
"gaps": [],
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded"
|
||||
}
|
||||
|
|
@ -30,7 +30,7 @@
|
|||
{
|
||||
"path": "schemas/event.schema.json",
|
||||
"id": "https://paperclip.dev/schemas/prp/v1/event.schema.json",
|
||||
"sha256": "b930af4a187c26312b4fff66813498b95332cb4e972f8b9282cdceafdf0de32c"
|
||||
"sha256": "d69ac4acfd9cd265f9dcb6a6580e949a67a50347093d0a7c12c2000a2da6d482"
|
||||
},
|
||||
{
|
||||
"path": "schemas/fixture.schema.json",
|
||||
|
|
@ -139,6 +139,78 @@
|
|||
"expectation": "accept",
|
||||
"compatibilityCase": "canonical"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/duplicate-event.snapshot.json",
|
||||
"sha256": "24a6c221f9d46824128cde009b277dae2ea8672a608cf7fa2d7e4a714e38bc92",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/duplicate-event.summary.json",
|
||||
"sha256": "dcbb2dae5810649cc59c6c0fe136e1033131d9510ebcbcc5f3435bfa998b95c9",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/failed-run.snapshot.json",
|
||||
"sha256": "a2bb468e3ca94b092a86afb8a57d2d05e0ce3cd6e9f87927dea81aef5aa32dd1",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/failed-run.summary.json",
|
||||
"sha256": "d7ac19d40b5d1d6106e2232cd2436ba1c4232bc6daa8a71957f12300ba851f65",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/happy-path.snapshot.json",
|
||||
"sha256": "037c114df8f205a13fa9070b07e6f9707218b8249bd196925246b054a7690853",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/happy-path.summary.json",
|
||||
"sha256": "00d2a033518c7b3f51ecb10aa64f7456922f6b3f930d853b3db35e72eaf55325",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/interrupted-run.snapshot.json",
|
||||
"sha256": "dc216831c437cc8fd2c37cee6f40f342767b21e01a511f2235a443be0d0b7b0c",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/interrupted-run.summary.json",
|
||||
"sha256": "753f226607b2d97301736d7f54b281ab7157397f3f32015c644787d370526332",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/source-gap.snapshot.json",
|
||||
"sha256": "7db7f25e423a721e60adfcdfee02db6d5e6fcd4a644ee180f5e29810ebbc0d78",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/source-gap.summary.json",
|
||||
"sha256": "861d0ee3fafe13bf8bf0fb71b7c1ef9a62cb26efba4dc8db889f36d27a14a813",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/unknown-optional-fields.snapshot.json",
|
||||
"sha256": "cf5b13a4b8d7ae73d1632e0539ec50d2ebcee8fbb6a2f33b28db7ff786fa90ff",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/golden/unknown-optional-fields.summary.json",
|
||||
"sha256": "7d983bcad0439b7884e70d07f8070b53ed594fba2da3e8ac6998570e238e84dc",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "deterministic-replay-oracle"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/replay/happy-path.json",
|
||||
"sha256": "dfe0aeb4e7a217f8d375afe6e6d1ab0151bb8e81921b07ae0aaa6c4d0b8f8a1d",
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@
|
|||
"properties": {
|
||||
"schema": { "const": "paperclip.prp.event.v1" },
|
||||
"sourceEventId": { "type": "string", "minLength": 1, "maxLength": 160 },
|
||||
"sourceSeq": { "type": "integer", "minimum": 1 },
|
||||
"sourceSeq": { "type": "integer", "minimum": 1, "maximum": 9007199254740991 },
|
||||
"sourceInstanceId": { "type": "string", "minLength": 1, "maxLength": 160 },
|
||||
"sourceKind": { "enum": ["runner", "control_plane"] },
|
||||
"runId": { "type": "string", "minLength": 1, "maxLength": 160 },
|
||||
|
|
|
|||
|
|
@ -22,7 +22,8 @@ const protocolRoot = resolve(packageRoot, "protocol");
|
|||
const schemaDirectory = resolve(protocolRoot, "schemas");
|
||||
const fixtureDirectory = resolve(protocolRoot, "fixtures");
|
||||
const outputPath = resolve(protocolRoot, "manifest.json");
|
||||
const expectedRejectedFixture = "fixtures/replay/unsupported-required-version.json";
|
||||
const expectedRejectedFixture =
|
||||
"fixtures/replay/unsupported-required-version.json";
|
||||
|
||||
export async function buildProtocolManifest() {
|
||||
const schemas = await loadSchemaCatalog(schemaDirectory);
|
||||
|
|
@ -38,7 +39,9 @@ export async function buildProtocolManifest() {
|
|||
let expectation = "accept";
|
||||
let compatibilityCase = "canonical";
|
||||
|
||||
if (relativePath.startsWith("fixtures/replay/")) {
|
||||
if (relativePath.startsWith("fixtures/replay/golden/")) {
|
||||
compatibilityCase = "deterministic-replay-oracle";
|
||||
} else if (relativePath.startsWith("fixtures/replay/")) {
|
||||
if (relativePath === expectedRejectedFixture) {
|
||||
expectation = "reject";
|
||||
compatibilityCase = "unknown-required-version";
|
||||
|
|
@ -46,7 +49,10 @@ export async function buildProtocolManifest() {
|
|||
assertReplayFixtureCompatibility(value);
|
||||
throw new Error(`${relativePath} did not fail closed`);
|
||||
} catch (error) {
|
||||
if (!String(error.message).startsWith("unsupported_required_version:")) throw error;
|
||||
if (
|
||||
!String(error.message).startsWith("unsupported_required_version:")
|
||||
)
|
||||
throw error;
|
||||
}
|
||||
assertSchemaInstance(validators.fixture, value, relativePath, false);
|
||||
} else {
|
||||
|
|
@ -58,7 +64,11 @@ export async function buildProtocolManifest() {
|
|||
}
|
||||
} else if (relativePath === "fixtures/questions/codex.json") {
|
||||
assertCodexQuestionFixture(value);
|
||||
assertSchemaInstance(validators.questionAdapterFixture, value, relativePath);
|
||||
assertSchemaInstance(
|
||||
validators.questionAdapterFixture,
|
||||
value,
|
||||
relativePath,
|
||||
);
|
||||
compatibilityCase = "codex-structured-input";
|
||||
} else if (relativePath === "fixtures/conformance-minimal-run.json") {
|
||||
assertSchemaInstance(validators.conformanceFixture, value, relativePath);
|
||||
|
|
@ -100,10 +110,14 @@ async function main() {
|
|||
if (process.argv.includes("--check")) {
|
||||
const current = await readFile(outputPath, "utf8").catch(() => "");
|
||||
if (current !== encoded) {
|
||||
process.stderr.write("The generated PRP contract manifest is stale. Run pnpm generate:protocol-manifest.\n");
|
||||
process.stderr.write(
|
||||
"The generated PRP contract manifest is stale. Run pnpm generate:protocol-manifest.\n",
|
||||
);
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
process.stdout.write("The generated PRP contract manifest matches its sources.\n");
|
||||
process.stdout.write(
|
||||
"The generated PRP contract manifest matches its sources.\n",
|
||||
);
|
||||
}
|
||||
} else {
|
||||
await writeFile(outputPath, encoded);
|
||||
|
|
@ -111,4 +125,5 @@ async function main() {
|
|||
}
|
||||
}
|
||||
|
||||
if (resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) await main();
|
||||
if (resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url))
|
||||
await main();
|
||||
|
|
|
|||
|
|
@ -0,0 +1,70 @@
|
|||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const schemaDirectory = resolve(packageRoot, "protocol/schemas");
|
||||
const outputPath = resolve(
|
||||
packageRoot,
|
||||
"src/protocol/generated/schema-bundle.ts",
|
||||
);
|
||||
const schemaNames = [
|
||||
"identity",
|
||||
"capabilities",
|
||||
"command",
|
||||
"provider-descriptor",
|
||||
"provider-event",
|
||||
"workspace-diff",
|
||||
"workspace-file-reference",
|
||||
"semantic-tool",
|
||||
"usage",
|
||||
"stop-reason",
|
||||
"terminal",
|
||||
"question-set",
|
||||
"question-response",
|
||||
"question-adapter-fixture",
|
||||
"request",
|
||||
"result",
|
||||
"event",
|
||||
"fixture",
|
||||
];
|
||||
|
||||
const schemas = await Promise.all(
|
||||
schemaNames.map(async (name) => ({
|
||||
name,
|
||||
value: JSON.parse(
|
||||
await readFile(resolve(schemaDirectory, `${name}.schema.json`), "utf8"),
|
||||
),
|
||||
})),
|
||||
);
|
||||
|
||||
const identifier = (name) =>
|
||||
name.replace(/-([a-z])/g, (_match, letter) => letter.toUpperCase());
|
||||
const declarations = schemas
|
||||
.map(
|
||||
({ name, value }) =>
|
||||
`export const ${identifier(name)}Schema = ${JSON.stringify(value, null, 2)} as const;`,
|
||||
)
|
||||
.join("\n\n");
|
||||
const bundle = schemaNames
|
||||
.map((name) => ` ${JSON.stringify(name)}: ${identifier(name)}Schema,`)
|
||||
.join("\n");
|
||||
const generated = `// Generated by scripts/generate-protocol-schema-module.mjs. Do not edit.\n\n${declarations}\n\nexport const prpSchemaBundle = {\n${bundle}\n} as const;\n`;
|
||||
|
||||
if (process.argv.includes("--check")) {
|
||||
const current = await readFile(outputPath, "utf8").catch(() => "");
|
||||
if (current !== generated) {
|
||||
process.stderr.write(
|
||||
"Generated PRP schema module is stale. Run pnpm generate:protocol-types.\n",
|
||||
);
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
process.stdout.write(
|
||||
"Generated PRP schema module matches the JSON Schema sources.\n",
|
||||
);
|
||||
}
|
||||
} else {
|
||||
await mkdir(dirname(outputPath), { recursive: true });
|
||||
await writeFile(outputPath, generated);
|
||||
process.stdout.write("Generated PRP schema module.\n");
|
||||
}
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import {
|
||||
parsePrpFixtureText,
|
||||
replayParitySummary,
|
||||
reducePrpFixture,
|
||||
} from "../dist/index.js";
|
||||
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const fixtureDirectory = resolve(packageRoot, "protocol/fixtures/replay");
|
||||
const goldenDirectory = resolve(fixtureDirectory, "golden");
|
||||
const fixtureNames = [
|
||||
"happy-path",
|
||||
"failed-run",
|
||||
"interrupted-run",
|
||||
"duplicate-event",
|
||||
"source-gap",
|
||||
"unknown-optional-fields",
|
||||
];
|
||||
const check = process.argv.includes("--check");
|
||||
const stale = [];
|
||||
|
||||
await mkdir(goldenDirectory, { recursive: true });
|
||||
for (const fixtureName of fixtureNames) {
|
||||
const source = await readFile(
|
||||
resolve(fixtureDirectory, `${fixtureName}.json`),
|
||||
"utf8",
|
||||
);
|
||||
const validation = parsePrpFixtureText(source);
|
||||
if (!validation.ok) {
|
||||
throw new Error(
|
||||
`${fixtureName}.json failed validation: ${validation.issues.map((issue) => issue.message).join("; ")}`,
|
||||
);
|
||||
}
|
||||
const snapshot = reducePrpFixture(validation.fixture);
|
||||
const outputs = [
|
||||
["snapshot", snapshot],
|
||||
["summary", replayParitySummary(snapshot)],
|
||||
];
|
||||
for (const [kind, value] of outputs) {
|
||||
const path = resolve(goldenDirectory, `${fixtureName}.${kind}.json`);
|
||||
const generated = `${JSON.stringify(value, null, 2)}\n`;
|
||||
if (check) {
|
||||
if ((await readFile(path, "utf8").catch(() => "")) !== generated) {
|
||||
stale.push(`${fixtureName}.${kind}.json`);
|
||||
}
|
||||
} else {
|
||||
await writeFile(path, generated);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (stale.length > 0) {
|
||||
process.stderr.write(
|
||||
`Replay golden files are stale:\n- ${stale.join("\n- ")}\n`,
|
||||
);
|
||||
process.exitCode = 1;
|
||||
} else if (check) {
|
||||
process.stdout.write(
|
||||
"Replay golden snapshots and parity summaries are current.\n",
|
||||
);
|
||||
} else {
|
||||
process.stdout.write(
|
||||
"Generated Replay golden snapshots and parity summaries.\n",
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
import Ajv2020 from "ajv/dist/2020.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
PRP_COMPLETION_RESULT_OUTPUT_SCHEMA,
|
||||
PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA,
|
||||
} from "./completion-result.js";
|
||||
|
||||
const baseResult = {
|
||||
schema: "paperclip.run_result.v1",
|
||||
reportedWorkDisposition: "done",
|
||||
summary: "Completed the requested work.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{ criterionId: "objective", status: "satisfied", evidenceRefs: [] },
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [],
|
||||
attentionRequests: [],
|
||||
artifacts: [],
|
||||
};
|
||||
|
||||
describe("provider-neutral completion result schema", () => {
|
||||
const validate = new Ajv2020({ allErrors: true, strict: false }).compile(
|
||||
PRP_COMPLETION_RESULT_OUTPUT_SCHEMA,
|
||||
);
|
||||
|
||||
it("allows done with no verification and no actionable attention", () => {
|
||||
expect(validate(structuredClone(baseResult))).toBe(true);
|
||||
});
|
||||
|
||||
it("allows provider tool callers to omit the constant schema discriminator", () => {
|
||||
const providerValidate = new Ajv2020({
|
||||
allErrors: true,
|
||||
strict: false,
|
||||
}).compile(PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA);
|
||||
const providerResult = structuredClone(baseResult) as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
delete providerResult.schema;
|
||||
expect(providerValidate(providerResult)).toBe(true);
|
||||
});
|
||||
|
||||
it("admits known smaller-model aliases at the provider boundary for canonical normalization", () => {
|
||||
const providerValidate = new Ajv2020({
|
||||
allErrors: true,
|
||||
strict: false,
|
||||
}).compile(PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA);
|
||||
const providerResult = structuredClone(baseResult);
|
||||
providerResult.schema = "paperclip_paperclip_finish";
|
||||
providerResult.reportedWorkDisposition = "completed";
|
||||
providerResult.completionClaim.criteria[0]!.status = "passed";
|
||||
providerResult.verification = [
|
||||
{ commandOrCheck: "model check", status: "pass" } as never,
|
||||
];
|
||||
expect(providerValidate(providerResult)).toBe(true);
|
||||
});
|
||||
|
||||
it("requires a reason code for verification that was not run", () => {
|
||||
const result = structuredClone(baseResult);
|
||||
result.verification = [
|
||||
{ commandOrCheck: "Run tests", status: "not_run" } as never,
|
||||
];
|
||||
expect(validate(result)).toBe(false);
|
||||
|
||||
result.verification = [
|
||||
{
|
||||
commandOrCheck: "Run tests",
|
||||
status: "not_run",
|
||||
reasonCode: "tool_unavailable",
|
||||
} as never,
|
||||
];
|
||||
expect(validate(result)).toBe(true);
|
||||
});
|
||||
|
||||
it("requires needs_review for actionable attention", () => {
|
||||
const result = structuredClone(baseResult);
|
||||
result.attentionRequests = [
|
||||
{
|
||||
kind: "review",
|
||||
summary: "Confirm the external result.",
|
||||
ownerClass: "human",
|
||||
} as never,
|
||||
];
|
||||
expect(validate(result)).toBe(false);
|
||||
|
||||
result.reportedWorkDisposition = "needs_review";
|
||||
expect(validate(result)).toBe(true);
|
||||
});
|
||||
|
||||
it("requires agent-owned attention to identify its target agent", () => {
|
||||
const result = structuredClone(baseResult);
|
||||
result.reportedWorkDisposition = "needs_review";
|
||||
result.attentionRequests = [
|
||||
{
|
||||
kind: "agent_handoff",
|
||||
summary: "Ask the deployment agent to continue.",
|
||||
ownerClass: "agent",
|
||||
} as never,
|
||||
];
|
||||
expect(validate(result)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,387 @@
|
|||
/** Provider-neutral semantic completion tools and their strict model-facing schemas. */
|
||||
export const PRP_COMPLETION_TOOL_NAME = "paperclip_finish" as const;
|
||||
export const PRP_BLOCK_TOOL_NAME = "paperclip_block" as const;
|
||||
export const PRP_SEMANTIC_TOOL_NAMES = [
|
||||
PRP_COMPLETION_TOOL_NAME,
|
||||
PRP_BLOCK_TOOL_NAME,
|
||||
] as const;
|
||||
|
||||
export const PRP_VERIFICATION_REASON_CODES = [
|
||||
"environment_unavailable",
|
||||
"tool_unavailable",
|
||||
"dependency_missing",
|
||||
"permission_denied",
|
||||
"credential_missing",
|
||||
"policy_restricted",
|
||||
"external_service_unavailable",
|
||||
"budget_exhausted",
|
||||
"other",
|
||||
] as const;
|
||||
|
||||
export const PRP_ACTIONABLE_ATTENTION_KINDS = [
|
||||
"review",
|
||||
"human_input",
|
||||
"approval",
|
||||
"credential",
|
||||
"external_action",
|
||||
"agent_handoff",
|
||||
] as const;
|
||||
|
||||
export const PRP_ATTENTION_OWNER_CLASSES = [
|
||||
"human",
|
||||
"agent",
|
||||
"external_system",
|
||||
] as const;
|
||||
|
||||
const completionClaimSchema = {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: [
|
||||
"contractRevision",
|
||||
"objectiveSatisfied",
|
||||
"criteria",
|
||||
"remainingWork",
|
||||
],
|
||||
properties: {
|
||||
contractRevision: { type: "string", minLength: 1 },
|
||||
objectiveSatisfied: { type: "boolean" },
|
||||
criteria: {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["criterionId", "status", "evidenceRefs"],
|
||||
properties: {
|
||||
criterionId: { type: "string", minLength: 1 },
|
||||
status: { enum: ["satisfied", "not_satisfied", "unknown"] },
|
||||
evidenceRefs: { type: "array", items: { type: "string" } },
|
||||
},
|
||||
},
|
||||
},
|
||||
remainingWork: {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["description", "blocksCompletion"],
|
||||
properties: {
|
||||
description: { type: "string", minLength: 1 },
|
||||
blocksCompletion: { type: "boolean" },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
|
||||
const evidenceSchema = {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["ref"],
|
||||
properties: { ref: { type: "string", minLength: 1 } },
|
||||
},
|
||||
} as const;
|
||||
|
||||
const verificationSchema = {
|
||||
type: "array",
|
||||
description:
|
||||
"Checks used to verify the work. Use failed only when the check actually ran and found a defect in the work. If a check could not run or complete because the environment, tool, dependency, permission, credential, policy, external service, or budget was unavailable, use not_run with reasonCode even when the attempted command exited non-zero.",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["commandOrCheck", "status"],
|
||||
properties: {
|
||||
commandOrCheck: { type: "string", minLength: 1 },
|
||||
status: {
|
||||
enum: ["passed", "failed", "not_run"],
|
||||
description:
|
||||
"passed: the check ran and succeeded. failed: the check ran to a meaningful verdict and found the work incorrect. not_run: the check had no meaningful verdict because it was not attempted or an environment/tool/dependency/permission/credential/policy/service/budget limitation prevented it from completing; command launch or setup failures are not_run.",
|
||||
},
|
||||
reasonCode: {
|
||||
enum: [...PRP_VERIFICATION_REASON_CODES],
|
||||
description:
|
||||
"Required for not_run; identify why the check had no meaningful verdict.",
|
||||
},
|
||||
detail: { type: "string" },
|
||||
artifactRef: { type: "string", minLength: 1 },
|
||||
},
|
||||
allOf: [
|
||||
{
|
||||
if: {
|
||||
properties: { status: { const: "not_run" } },
|
||||
required: ["status"],
|
||||
},
|
||||
then: { required: ["reasonCode"] },
|
||||
},
|
||||
],
|
||||
},
|
||||
} as const;
|
||||
|
||||
const attentionRequestsSchema = {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["kind", "summary", "ownerClass"],
|
||||
properties: {
|
||||
kind: { enum: [...PRP_ACTIONABLE_ATTENTION_KINDS] },
|
||||
summary: { type: "string", minLength: 1 },
|
||||
ownerClass: { enum: [...PRP_ATTENTION_OWNER_CLASSES] },
|
||||
targetAgentId: { type: "string", minLength: 1 },
|
||||
},
|
||||
allOf: [
|
||||
{
|
||||
if: {
|
||||
properties: { ownerClass: { const: "agent" } },
|
||||
required: ["ownerClass"],
|
||||
},
|
||||
then: { required: ["targetAgentId"] },
|
||||
},
|
||||
],
|
||||
},
|
||||
} as const;
|
||||
|
||||
const artifactsSchema = {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["kind", "ref"],
|
||||
properties: {
|
||||
kind: { type: "string", minLength: 1 },
|
||||
ref: { type: "string", minLength: 1 },
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
|
||||
const commonResultProperties = {
|
||||
schema: { type: "string", const: "paperclip.run_result.v1" },
|
||||
summary: { type: "string", minLength: 1 },
|
||||
completionClaim: completionClaimSchema,
|
||||
evidence: evidenceSchema,
|
||||
verification: verificationSchema,
|
||||
attentionRequests: attentionRequestsSchema,
|
||||
artifacts: artifactsSchema,
|
||||
} as const;
|
||||
|
||||
export const PRP_COMPLETION_RESULT_OUTPUT_SCHEMA = {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: [
|
||||
"reportedWorkDisposition",
|
||||
"summary",
|
||||
"completionClaim",
|
||||
"evidence",
|
||||
"verification",
|
||||
"attentionRequests",
|
||||
"artifacts",
|
||||
],
|
||||
properties: {
|
||||
...commonResultProperties,
|
||||
reportedWorkDisposition: { enum: ["done", "needs_review"] },
|
||||
},
|
||||
allOf: [
|
||||
{
|
||||
if: {
|
||||
properties: { reportedWorkDisposition: { const: "done" } },
|
||||
required: ["reportedWorkDisposition"],
|
||||
},
|
||||
then: { properties: { attentionRequests: { maxItems: 0 } } },
|
||||
},
|
||||
{
|
||||
if: {
|
||||
properties: { reportedWorkDisposition: { const: "needs_review" } },
|
||||
required: ["reportedWorkDisposition"],
|
||||
},
|
||||
then: { properties: { attentionRequests: { minItems: 1 } } },
|
||||
},
|
||||
],
|
||||
} as const;
|
||||
|
||||
export const PRP_BLOCK_RESULT_OUTPUT_SCHEMA = {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: [
|
||||
"reportedWorkDisposition",
|
||||
"summary",
|
||||
"completionClaim",
|
||||
"evidence",
|
||||
"verification",
|
||||
"attentionRequests",
|
||||
"artifacts",
|
||||
"blocker",
|
||||
],
|
||||
properties: {
|
||||
...commonResultProperties,
|
||||
reportedWorkDisposition: { type: "string", const: "blocked" },
|
||||
attentionRequests: {
|
||||
type: "array",
|
||||
maxItems: 0,
|
||||
items: attentionRequestsSchema.items,
|
||||
},
|
||||
blocker: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["reasonCode", "owner", "unblockAction", "scope"],
|
||||
properties: {
|
||||
reasonCode: { type: "string", minLength: 1 },
|
||||
owner: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["kind", "name"],
|
||||
properties: {
|
||||
kind: { enum: ["agent", "user", "system", "external"] },
|
||||
name: { type: "string", minLength: 1 },
|
||||
},
|
||||
},
|
||||
unblockAction: { type: "string", minLength: 1 },
|
||||
scope: { enum: ["current_track", "task_wide"] },
|
||||
},
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
|
||||
const providerVerificationCompatibilitySchema = {
|
||||
type: "array",
|
||||
description:
|
||||
"Checks used to verify the work. Use failed only when the check actually ran and found a defect in the work. If a check could not run or complete because the environment, tool, dependency, permission, credential, policy, external service, or budget was unavailable, use not_run with reasonCode even when the attempted command exited non-zero.",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: true,
|
||||
required: ["status"],
|
||||
properties: {
|
||||
commandOrCheck: { type: "string", minLength: 1 },
|
||||
command: { type: "string", minLength: 1 },
|
||||
status: {
|
||||
enum: [
|
||||
"passed",
|
||||
"failed",
|
||||
"not_run",
|
||||
"blocked",
|
||||
"skipped",
|
||||
"pass",
|
||||
"success",
|
||||
"succeeded",
|
||||
"fail",
|
||||
],
|
||||
description:
|
||||
"passed: the check ran and succeeded. failed: the check ran to a meaningful verdict and found the work incorrect. not_run: no meaningful verdict because the check was not attempted or could not complete. Prefer not_run over legacy blocked/skipped, and include reasonCode for any unavailable check.",
|
||||
},
|
||||
reasonCode: {
|
||||
enum: [...PRP_VERIFICATION_REASON_CODES],
|
||||
description:
|
||||
"Required for not_run; identify why the check had no meaningful verdict.",
|
||||
},
|
||||
detail: { type: "string" },
|
||||
result: { type: "string" },
|
||||
cwd: { type: "string" },
|
||||
artifactRef: { type: "string", minLength: 1 },
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
|
||||
const providerCompletionClaimCompatibilitySchema = {
|
||||
...completionClaimSchema,
|
||||
properties: {
|
||||
...completionClaimSchema.properties,
|
||||
criteria: {
|
||||
...completionClaimSchema.properties.criteria,
|
||||
items: {
|
||||
...completionClaimSchema.properties.criteria.items,
|
||||
properties: {
|
||||
...completionClaimSchema.properties.criteria.items.properties,
|
||||
status: {
|
||||
enum: [
|
||||
"satisfied",
|
||||
"not_satisfied",
|
||||
"unknown",
|
||||
"pass",
|
||||
"passed",
|
||||
"complete",
|
||||
"completed",
|
||||
"fail",
|
||||
"failed",
|
||||
"incomplete",
|
||||
],
|
||||
description:
|
||||
"Use satisfied when the criterion is met, not_satisfied when it is not met, or unknown when no verdict is available. Common provider aliases are accepted and normalized.",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
|
||||
const providerCommonResultProperties = {
|
||||
...commonResultProperties,
|
||||
schema: {
|
||||
enum: [
|
||||
"paperclip.run_result.v1",
|
||||
"paperclip_finish",
|
||||
"paperclip_paperclip_finish",
|
||||
"paperclip_finish.v1",
|
||||
],
|
||||
},
|
||||
completionClaim: providerCompletionClaimCompatibilitySchema,
|
||||
} as const;
|
||||
|
||||
const providerAttentionCompatibilitySchema = {
|
||||
type: "array",
|
||||
items: {
|
||||
type: "object",
|
||||
additionalProperties: true,
|
||||
required: ["kind", "summary"],
|
||||
properties: {
|
||||
kind: { type: "string", minLength: 1 },
|
||||
summary: { type: "string", minLength: 1 },
|
||||
ownerClass: { enum: [...PRP_ATTENTION_OWNER_CLASSES] },
|
||||
targetAgentId: { type: "string", minLength: 1 },
|
||||
requestedCapability: { type: "string", minLength: 1 },
|
||||
requiredAuthority: { type: "string", minLength: 1 },
|
||||
target: { type: "object", additionalProperties: true },
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* Provider-facing compatibility schemas. Canonical PRP validation still uses
|
||||
* the strict schemas above after legacy aliases have been normalized.
|
||||
*/
|
||||
export const PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA = {
|
||||
type: "object",
|
||||
additionalProperties: true,
|
||||
required: [
|
||||
"reportedWorkDisposition",
|
||||
"summary",
|
||||
"completionClaim",
|
||||
"evidence",
|
||||
"verification",
|
||||
],
|
||||
properties: {
|
||||
...providerCommonResultProperties,
|
||||
reportedWorkDisposition: { enum: ["done", "needs_review", "completed"] },
|
||||
verification: providerVerificationCompatibilitySchema,
|
||||
attentionRequests: providerAttentionCompatibilitySchema,
|
||||
},
|
||||
} as const;
|
||||
|
||||
export const PRP_BLOCK_RESULT_PROVIDER_INPUT_SCHEMA = {
|
||||
type: "object",
|
||||
additionalProperties: true,
|
||||
required: [
|
||||
"reportedWorkDisposition",
|
||||
"summary",
|
||||
"completionClaim",
|
||||
"evidence",
|
||||
"verification",
|
||||
"blocker",
|
||||
],
|
||||
properties: {
|
||||
...providerCommonResultProperties,
|
||||
reportedWorkDisposition: { type: "string", const: "blocked" },
|
||||
verification: providerVerificationCompatibilitySchema,
|
||||
attentionRequests: providerAttentionCompatibilitySchema,
|
||||
blocker: PRP_BLOCK_RESULT_OUTPUT_SCHEMA.properties.blocker,
|
||||
},
|
||||
} as const;
|
||||
|
|
@ -0,0 +1,108 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
PAPERCLIP_QUESTION_SET_SCHEMA,
|
||||
parsePaperclipQuestionResponse,
|
||||
parsePaperclipQuestionSet,
|
||||
type PaperclipQuestionSet,
|
||||
} from "./question-set.js";
|
||||
|
||||
const questionSet: PaperclipQuestionSet = {
|
||||
schema: PAPERCLIP_QUESTION_SET_SCHEMA,
|
||||
title: "Release input",
|
||||
questions: [
|
||||
{
|
||||
id: "environment",
|
||||
prompt: "Where should we deploy?",
|
||||
required: true,
|
||||
answerMode: "single_select",
|
||||
options: [
|
||||
{ id: "staging", label: "Staging" },
|
||||
{ id: "production", label: "Production" },
|
||||
],
|
||||
customAnswer: { enabled: true, label: "Other" },
|
||||
},
|
||||
{
|
||||
id: "replicas",
|
||||
prompt: "How many replicas?",
|
||||
required: true,
|
||||
answerMode: "text",
|
||||
textValidation: { inputType: "integer", minimum: 1, maximum: 20 },
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
describe("Paperclip question-set contract", () => {
|
||||
it("round-trips the portable presentation model", () => {
|
||||
expect(parsePaperclipQuestionSet(questionSet)).toEqual(questionSet);
|
||||
expect(
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: {
|
||||
environment: { selectedOptionIds: ["staging"] },
|
||||
replicas: { text: "3" },
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: {
|
||||
environment: { selectedOptionIds: ["staging"] },
|
||||
replicas: { text: "3" },
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects missing, unknown, and provider-shaped answers", () => {
|
||||
expect(() =>
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: {
|
||||
environment: { selectedOptionIds: ["unknown"] },
|
||||
replicas: { text: "3" },
|
||||
},
|
||||
}),
|
||||
).toThrow(/unknown option/);
|
||||
expect(() =>
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: { environment: { selectedOptionIds: ["staging"] } },
|
||||
}),
|
||||
).toThrow(/replicas.*required/);
|
||||
expect(() =>
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
answers: { environment: { answers: ["Staging"] } },
|
||||
}),
|
||||
).toThrow(/paperclip.question_response.v1/);
|
||||
expect(() =>
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: {
|
||||
environment: { answers: ["Staging"] },
|
||||
replicas: { text: "3" },
|
||||
},
|
||||
}),
|
||||
).toThrow(/canonical response contract/);
|
||||
});
|
||||
|
||||
it("applies typed numeric validation before an adapter sees the answer", () => {
|
||||
expect(() =>
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: {
|
||||
environment: { customText: "Canary" },
|
||||
replicas: { text: "3.5" },
|
||||
},
|
||||
}),
|
||||
).toThrow(/valid integer/);
|
||||
expect(() =>
|
||||
parsePaperclipQuestionResponse(questionSet, {
|
||||
schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA,
|
||||
answers: {
|
||||
environment: { customText: "Canary" },
|
||||
replicas: { text: "21" },
|
||||
},
|
||||
}),
|
||||
).toThrow(/at most 20/);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,664 @@
|
|||
export const PAPERCLIP_QUESTION_SET_SCHEMA =
|
||||
"paperclip.question_set.v1" as const;
|
||||
export const PAPERCLIP_QUESTION_RESPONSE_SCHEMA =
|
||||
"paperclip.question_response.v1" as const;
|
||||
export const PAPERCLIP_RUNTIME_REQUEST_SCHEMA_V2 =
|
||||
"paperclip.runtime_request.v2" as const;
|
||||
|
||||
export type PaperclipQuestionAnswerMode =
|
||||
"single_select" | "multi_select" | "text";
|
||||
|
||||
export interface PaperclipQuestionOption {
|
||||
id: string;
|
||||
label: string;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
export interface PaperclipQuestionCustomAnswer {
|
||||
enabled: true;
|
||||
label?: string;
|
||||
placeholder?: string;
|
||||
}
|
||||
|
||||
export interface PaperclipQuestionTextValidation {
|
||||
minLength?: number;
|
||||
maxLength?: number;
|
||||
pattern?: string;
|
||||
inputType?: "text" | "number" | "integer";
|
||||
minimum?: number;
|
||||
maximum?: number;
|
||||
}
|
||||
|
||||
export interface PaperclipQuestion {
|
||||
id: string;
|
||||
header?: string;
|
||||
prompt: string;
|
||||
helpText?: string;
|
||||
required: boolean;
|
||||
answerMode: PaperclipQuestionAnswerMode;
|
||||
options?: PaperclipQuestionOption[];
|
||||
customAnswer?: PaperclipQuestionCustomAnswer;
|
||||
textValidation?: PaperclipQuestionTextValidation;
|
||||
}
|
||||
|
||||
export interface PaperclipQuestionSet {
|
||||
schema: typeof PAPERCLIP_QUESTION_SET_SCHEMA;
|
||||
title?: string;
|
||||
description?: string;
|
||||
submitLabel?: string;
|
||||
questions: PaperclipQuestion[];
|
||||
}
|
||||
|
||||
export interface PaperclipQuestionAnswer {
|
||||
selectedOptionIds?: string[];
|
||||
text?: string;
|
||||
customText?: string;
|
||||
}
|
||||
|
||||
export interface PaperclipQuestionResponse {
|
||||
schema: typeof PAPERCLIP_QUESTION_RESPONSE_SCHEMA;
|
||||
answers: Record<string, PaperclipQuestionAnswer>;
|
||||
}
|
||||
|
||||
export interface PaperclipRuntimeRequestOrigin {
|
||||
adapter: string;
|
||||
provider?: string;
|
||||
method?: string;
|
||||
}
|
||||
|
||||
export interface PaperclipRuntimeInputRequest {
|
||||
schema: typeof PAPERCLIP_RUNTIME_REQUEST_SCHEMA_V2;
|
||||
requestKind: "runtime";
|
||||
requestId: string;
|
||||
type: "input";
|
||||
status: "pending" | "resolved" | "expired" | "cancelled";
|
||||
prompt: string;
|
||||
input: PaperclipQuestionSet;
|
||||
origin?: PaperclipRuntimeRequestOrigin;
|
||||
turnId?: string;
|
||||
itemId?: string;
|
||||
}
|
||||
|
||||
export class PaperclipQuestionValidationError extends Error {
|
||||
readonly code = "invalid_question_response" as const;
|
||||
readonly path: string;
|
||||
|
||||
constructor(path: string, detail: string) {
|
||||
super(`${path}: ${detail}`);
|
||||
this.name = "PaperclipQuestionValidationError";
|
||||
this.path = path;
|
||||
}
|
||||
}
|
||||
|
||||
function record(value: unknown): Record<string, unknown> | null {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: null;
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(
|
||||
value: Record<string, unknown>,
|
||||
allowed: readonly string[],
|
||||
path: string,
|
||||
): void {
|
||||
const allowedKeys = new Set(allowed);
|
||||
const unknown = Object.keys(value).find((key) => !allowedKeys.has(key));
|
||||
if (unknown !== undefined) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/${unknown}`,
|
||||
"is not part of the canonical response contract",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function requiredText(value: unknown, path: string, maxLength = 4_000): string {
|
||||
if (
|
||||
typeof value !== "string" ||
|
||||
value.length === 0 ||
|
||||
value.length > maxLength
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must be a non-empty string of at most ${maxLength} characters`,
|
||||
);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function optionalText(
|
||||
value: unknown,
|
||||
path: string,
|
||||
maxLength = 4_000,
|
||||
): string | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
if (typeof value !== "string" || value.length > maxLength) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must be a string of at most ${maxLength} characters`,
|
||||
);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function optionalFiniteNumber(
|
||||
value: unknown,
|
||||
path: string,
|
||||
): number | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
if (typeof value !== "number" || !Number.isFinite(value)) {
|
||||
throw new PaperclipQuestionValidationError(path, "must be a finite number");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Parse and sanitize the provider-neutral presentation contract at an adapter boundary. */
|
||||
export function parsePaperclipQuestionSet(
|
||||
value: unknown,
|
||||
): PaperclipQuestionSet {
|
||||
const candidate = record(value);
|
||||
if (
|
||||
candidate === null ||
|
||||
candidate.schema !== PAPERCLIP_QUESTION_SET_SCHEMA
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
"/input",
|
||||
`must use ${PAPERCLIP_QUESTION_SET_SCHEMA}`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
!Array.isArray(candidate.questions) ||
|
||||
candidate.questions.length === 0 ||
|
||||
candidate.questions.length > 64
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
"/input/questions",
|
||||
"must contain between 1 and 64 questions",
|
||||
);
|
||||
}
|
||||
const questionIds = new Set<string>();
|
||||
const questions = candidate.questions.map(
|
||||
(rawQuestion, questionIndex): PaperclipQuestion => {
|
||||
const path = `/input/questions/${questionIndex}`;
|
||||
const question = record(rawQuestion);
|
||||
if (question === null)
|
||||
throw new PaperclipQuestionValidationError(path, "must be an object");
|
||||
const id = requiredText(question.id, `${path}/id`, 160);
|
||||
if (questionIds.has(id))
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/id`,
|
||||
"must be unique",
|
||||
);
|
||||
questionIds.add(id);
|
||||
const answerMode = question.answerMode;
|
||||
if (
|
||||
answerMode !== "single_select" &&
|
||||
answerMode !== "multi_select" &&
|
||||
answerMode !== "text"
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/answerMode`,
|
||||
"must be single_select, multi_select, or text",
|
||||
);
|
||||
}
|
||||
if (typeof question.required !== "boolean") {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/required`,
|
||||
"must be boolean",
|
||||
);
|
||||
}
|
||||
if (Array.isArray(question.options) && question.options.length > 128) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/options`,
|
||||
"cannot contain more than 128 options",
|
||||
);
|
||||
}
|
||||
const options: PaperclipQuestionOption[] | undefined = Array.isArray(
|
||||
question.options,
|
||||
)
|
||||
? question.options.map((rawOption, optionIndex) => {
|
||||
const optionPath = `${path}/options/${optionIndex}`;
|
||||
const option = record(rawOption);
|
||||
if (option === null)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
optionPath,
|
||||
"must be an object",
|
||||
);
|
||||
return {
|
||||
id: requiredText(option.id, `${optionPath}/id`, 160),
|
||||
label: requiredText(option.label, `${optionPath}/label`, 1_000),
|
||||
...(optionalText(
|
||||
option.description,
|
||||
`${optionPath}/description`,
|
||||
) !== undefined
|
||||
? {
|
||||
description: optionalText(
|
||||
option.description,
|
||||
`${optionPath}/description`,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
})
|
||||
: undefined;
|
||||
if (
|
||||
options !== undefined &&
|
||||
new Set(options.map((option) => option.id)).size !== options.length
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/options`,
|
||||
"option IDs must be unique within a question",
|
||||
);
|
||||
}
|
||||
if (answerMode !== "text" && (!options || options.length === 0)) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/options`,
|
||||
"select questions require at least one option",
|
||||
);
|
||||
}
|
||||
if (
|
||||
answerMode === "text" &&
|
||||
options !== undefined &&
|
||||
options.length > 0
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/options`,
|
||||
"text questions cannot define options",
|
||||
);
|
||||
}
|
||||
const custom = record(question.customAnswer);
|
||||
const customAnswer =
|
||||
custom === null
|
||||
? undefined
|
||||
: {
|
||||
enabled: true as const,
|
||||
...(optionalText(
|
||||
custom.label,
|
||||
`${path}/customAnswer/label`,
|
||||
1_000,
|
||||
) !== undefined
|
||||
? {
|
||||
label: optionalText(
|
||||
custom.label,
|
||||
`${path}/customAnswer/label`,
|
||||
1_000,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
...(optionalText(
|
||||
custom.placeholder,
|
||||
`${path}/customAnswer/placeholder`,
|
||||
1_000,
|
||||
) !== undefined
|
||||
? {
|
||||
placeholder: optionalText(
|
||||
custom.placeholder,
|
||||
`${path}/customAnswer/placeholder`,
|
||||
1_000,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
if (custom !== null && custom.enabled !== true) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/customAnswer/enabled`,
|
||||
"must be true when customAnswer is present",
|
||||
);
|
||||
}
|
||||
if (answerMode === "text" && customAnswer !== undefined) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/customAnswer`,
|
||||
"text questions do not use a separate custom answer",
|
||||
);
|
||||
}
|
||||
const validation = record(question.textValidation);
|
||||
const textValidation: PaperclipQuestionTextValidation | undefined =
|
||||
validation === null
|
||||
? undefined
|
||||
: {
|
||||
...(typeof validation.minLength === "number"
|
||||
? { minLength: validation.minLength }
|
||||
: {}),
|
||||
...(typeof validation.maxLength === "number"
|
||||
? { maxLength: validation.maxLength }
|
||||
: {}),
|
||||
...(optionalText(
|
||||
validation.pattern,
|
||||
`${path}/textValidation/pattern`,
|
||||
1_000,
|
||||
) !== undefined
|
||||
? {
|
||||
pattern: optionalText(
|
||||
validation.pattern,
|
||||
`${path}/textValidation/pattern`,
|
||||
1_000,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
...(validation.inputType === "number" ||
|
||||
validation.inputType === "integer" ||
|
||||
validation.inputType === "text"
|
||||
? { inputType: validation.inputType }
|
||||
: {}),
|
||||
...(optionalFiniteNumber(
|
||||
validation.minimum,
|
||||
`${path}/textValidation/minimum`,
|
||||
) !== undefined
|
||||
? {
|
||||
minimum: optionalFiniteNumber(
|
||||
validation.minimum,
|
||||
`${path}/textValidation/minimum`,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
...(optionalFiniteNumber(
|
||||
validation.maximum,
|
||||
`${path}/textValidation/maximum`,
|
||||
) !== undefined
|
||||
? {
|
||||
maximum: optionalFiniteNumber(
|
||||
validation.maximum,
|
||||
`${path}/textValidation/maximum`,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
if (validation !== null) {
|
||||
for (const key of ["minLength", "maxLength"] as const) {
|
||||
const raw = validation[key];
|
||||
if (
|
||||
raw !== undefined &&
|
||||
(!Number.isSafeInteger(raw) ||
|
||||
(raw as number) < 0 ||
|
||||
(raw as number) > 100_000)
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/textValidation/${key}`,
|
||||
"must be an integer from 0 through 100000",
|
||||
);
|
||||
}
|
||||
}
|
||||
if (
|
||||
validation.inputType !== undefined &&
|
||||
!["text", "number", "integer"].includes(String(validation.inputType))
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/textValidation/inputType`,
|
||||
"must be text, number, or integer",
|
||||
);
|
||||
}
|
||||
if (
|
||||
textValidation?.minLength !== undefined &&
|
||||
textValidation.maxLength !== undefined &&
|
||||
textValidation.minLength > textValidation.maxLength
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/textValidation`,
|
||||
"minLength cannot exceed maxLength",
|
||||
);
|
||||
}
|
||||
if (
|
||||
textValidation?.minimum !== undefined &&
|
||||
textValidation.maximum !== undefined &&
|
||||
textValidation.minimum > textValidation.maximum
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/textValidation`,
|
||||
"minimum cannot exceed maximum",
|
||||
);
|
||||
}
|
||||
if (textValidation?.pattern !== undefined) {
|
||||
try {
|
||||
new RegExp(textValidation.pattern);
|
||||
} catch {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/textValidation/pattern`,
|
||||
"must be a valid regular expression",
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
return {
|
||||
id,
|
||||
...(optionalText(question.header, `${path}/header`, 1_000) !== undefined
|
||||
? { header: optionalText(question.header, `${path}/header`, 1_000) }
|
||||
: {}),
|
||||
prompt: requiredText(question.prompt, `${path}/prompt`),
|
||||
...(optionalText(question.helpText, `${path}/helpText`) !== undefined
|
||||
? { helpText: optionalText(question.helpText, `${path}/helpText`) }
|
||||
: {}),
|
||||
required: question.required,
|
||||
answerMode,
|
||||
...(options !== undefined ? { options } : {}),
|
||||
...(customAnswer !== undefined ? { customAnswer } : {}),
|
||||
...(textValidation !== undefined ? { textValidation } : {}),
|
||||
};
|
||||
},
|
||||
);
|
||||
return {
|
||||
schema: PAPERCLIP_QUESTION_SET_SCHEMA,
|
||||
...(optionalText(candidate.title, "/input/title", 1_000) !== undefined
|
||||
? { title: optionalText(candidate.title, "/input/title", 1_000) }
|
||||
: {}),
|
||||
...(optionalText(candidate.description, "/input/description") !== undefined
|
||||
? {
|
||||
description: optionalText(
|
||||
candidate.description,
|
||||
"/input/description",
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
...(optionalText(candidate.submitLabel, "/input/submitLabel", 200) !==
|
||||
undefined
|
||||
? {
|
||||
submitLabel: optionalText(
|
||||
candidate.submitLabel,
|
||||
"/input/submitLabel",
|
||||
200,
|
||||
),
|
||||
}
|
||||
: {}),
|
||||
questions,
|
||||
};
|
||||
}
|
||||
|
||||
function answerHasValue(answer: PaperclipQuestionAnswer): boolean {
|
||||
return Boolean(
|
||||
answer.text?.trim() ||
|
||||
answer.customText?.trim() ||
|
||||
answer.selectedOptionIds?.length,
|
||||
);
|
||||
}
|
||||
|
||||
/** Revalidate untrusted UI input against the persisted question set. */
|
||||
export function parsePaperclipQuestionResponse(
|
||||
questionSetValue: unknown,
|
||||
responseValue: unknown,
|
||||
): PaperclipQuestionResponse {
|
||||
const questionSet = parsePaperclipQuestionSet(questionSetValue);
|
||||
const response = record(responseValue);
|
||||
if (
|
||||
response === null ||
|
||||
response.schema !== PAPERCLIP_QUESTION_RESPONSE_SCHEMA
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
"/response",
|
||||
`must use ${PAPERCLIP_QUESTION_RESPONSE_SCHEMA}`,
|
||||
);
|
||||
}
|
||||
rejectUnknownKeys(response, ["schema", "answers"], "/response");
|
||||
const rawAnswers = record(response.answers);
|
||||
if (rawAnswers === null)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
"/response/answers",
|
||||
"must be an object keyed by question ID",
|
||||
);
|
||||
const questions = new Map(
|
||||
questionSet.questions.map((question) => [question.id, question]),
|
||||
);
|
||||
for (const questionId of Object.keys(rawAnswers)) {
|
||||
if (!questions.has(questionId))
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`/response/answers/${questionId}`,
|
||||
"does not match a question in the persisted set",
|
||||
);
|
||||
}
|
||||
const answers: Record<string, PaperclipQuestionAnswer> = {};
|
||||
for (const question of questionSet.questions) {
|
||||
const path = `/response/answers/${question.id}`;
|
||||
const raw = rawAnswers[question.id];
|
||||
if (raw === undefined) {
|
||||
if (question.required)
|
||||
throw new PaperclipQuestionValidationError(path, "is required");
|
||||
continue;
|
||||
}
|
||||
const answer = record(raw);
|
||||
if (answer === null)
|
||||
throw new PaperclipQuestionValidationError(path, "must be an object");
|
||||
rejectUnknownKeys(
|
||||
answer,
|
||||
["selectedOptionIds", "text", "customText"],
|
||||
path,
|
||||
);
|
||||
const selectedOptionIds =
|
||||
answer.selectedOptionIds === undefined
|
||||
? undefined
|
||||
: Array.isArray(answer.selectedOptionIds) &&
|
||||
answer.selectedOptionIds.every((entry) => typeof entry === "string")
|
||||
? [...answer.selectedOptionIds]
|
||||
: null;
|
||||
if (selectedOptionIds === null)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/selectedOptionIds`,
|
||||
"must be an array of strings",
|
||||
);
|
||||
if (
|
||||
selectedOptionIds !== undefined &&
|
||||
new Set(selectedOptionIds).size !== selectedOptionIds.length
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/selectedOptionIds`,
|
||||
"cannot contain duplicates",
|
||||
);
|
||||
}
|
||||
const textValue = optionalText(answer.text, `${path}/text`, 100_000);
|
||||
const customText = optionalText(
|
||||
answer.customText,
|
||||
`${path}/customText`,
|
||||
100_000,
|
||||
);
|
||||
if (question.answerMode === "text") {
|
||||
if (selectedOptionIds?.length || customText !== undefined)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
"text answers only carry text",
|
||||
);
|
||||
} else {
|
||||
if (textValue !== undefined)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
"select answers do not carry text",
|
||||
);
|
||||
const allowed = new Set(
|
||||
(question.options ?? []).map((option) => option.id),
|
||||
);
|
||||
for (const optionId of selectedOptionIds ?? []) {
|
||||
if (!allowed.has(optionId))
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/selectedOptionIds`,
|
||||
`contains unknown option ${optionId}`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
question.answerMode === "single_select" &&
|
||||
(selectedOptionIds?.length ?? 0) > 1
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/selectedOptionIds`,
|
||||
"single-select answers choose at most one option",
|
||||
);
|
||||
}
|
||||
if (customText !== undefined && question.customAnswer?.enabled !== true) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
`${path}/customText`,
|
||||
"custom answers are not enabled for this question",
|
||||
);
|
||||
}
|
||||
if (
|
||||
customText?.trim() &&
|
||||
(selectedOptionIds?.length ?? 0) > 0 &&
|
||||
question.answerMode === "single_select"
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
"single-select answers cannot select an option and a custom answer",
|
||||
);
|
||||
}
|
||||
}
|
||||
const parsed: PaperclipQuestionAnswer = {
|
||||
...(selectedOptionIds !== undefined ? { selectedOptionIds } : {}),
|
||||
...(textValue !== undefined ? { text: textValue } : {}),
|
||||
...(customText !== undefined ? { customText } : {}),
|
||||
};
|
||||
if (question.required && !answerHasValue(parsed))
|
||||
throw new PaperclipQuestionValidationError(path, "is required");
|
||||
const boundedText =
|
||||
question.answerMode === "text" ? parsed.text : parsed.customText;
|
||||
if (boundedText !== undefined) {
|
||||
const validation = question.textValidation;
|
||||
if (
|
||||
validation?.minLength !== undefined &&
|
||||
boundedText.length < validation.minLength
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must contain at least ${validation.minLength} characters`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
validation?.maxLength !== undefined &&
|
||||
boundedText.length > validation.maxLength
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must contain at most ${validation.maxLength} characters`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
validation?.pattern !== undefined &&
|
||||
!new RegExp(validation.pattern).test(boundedText)
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
"does not match the required format",
|
||||
);
|
||||
}
|
||||
if (
|
||||
validation?.inputType === "number" ||
|
||||
validation?.inputType === "integer"
|
||||
) {
|
||||
const numeric = Number(boundedText);
|
||||
if (
|
||||
!Number.isFinite(numeric) ||
|
||||
(validation.inputType === "integer" && !Number.isInteger(numeric))
|
||||
) {
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must be a valid ${validation.inputType}`,
|
||||
);
|
||||
}
|
||||
if (validation.minimum !== undefined && numeric < validation.minimum)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must be at least ${validation.minimum}`,
|
||||
);
|
||||
if (validation.maximum !== undefined && numeric > validation.maximum)
|
||||
throw new PaperclipQuestionValidationError(
|
||||
path,
|
||||
`must be at most ${validation.maximum}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (answerHasValue(parsed)) answers[question.id] = parsed;
|
||||
}
|
||||
return { schema: PAPERCLIP_QUESTION_RESPONSE_SCHEMA, answers };
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
export * from "./contracts/completion-result.js";
|
||||
export * from "./contracts/question-set.js";
|
||||
export * from "./protocol/replay-contract.js";
|
||||
export * from "./protocol/replay-loader.js";
|
||||
export * from "./protocol/result-normalization.js";
|
||||
export * from "./reducer/session-reducer.js";
|
||||
export * from "./tracer/replay.js";
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,172 @@
|
|||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
negotiateProtocolVersion,
|
||||
parsePrpFixtureText,
|
||||
PRP_PROTOCOL_VERSION,
|
||||
} from "./replay-contract.js";
|
||||
|
||||
const fixtureDirectory = new URL(
|
||||
"../../protocol/fixtures/replay/",
|
||||
import.meta.url,
|
||||
);
|
||||
const validFixtures = [
|
||||
"happy-path.json",
|
||||
"failed-run.json",
|
||||
"interrupted-run.json",
|
||||
"duplicate-event.json",
|
||||
"source-gap.json",
|
||||
"unknown-optional-fields.json",
|
||||
];
|
||||
|
||||
async function readFixture(
|
||||
name = "happy-path.json",
|
||||
): Promise<Record<string, unknown>> {
|
||||
return JSON.parse(
|
||||
await readFile(new URL(name, fixtureDirectory), "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
}
|
||||
|
||||
describe("PRP v1 JSON Schema contract", () => {
|
||||
for (const fixtureName of validFixtures) {
|
||||
it(`validates ${fixtureName}`, async () => {
|
||||
const result = parsePrpFixtureText(
|
||||
await readFile(new URL(fixtureName, fixtureDirectory), "utf8"),
|
||||
);
|
||||
expect(result.ok).toBe(true);
|
||||
});
|
||||
}
|
||||
|
||||
it("preserves unknown optional fields for forward compatibility", async () => {
|
||||
const result = parsePrpFixtureText(
|
||||
await readFile(
|
||||
new URL("unknown-optional-fields.json", fixtureDirectory),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
expect(result.ok).toBe(true);
|
||||
if (result.ok) {
|
||||
expect(result.fixture.futureFixtureHint).toEqual({
|
||||
producerVersion: "1.1-preview",
|
||||
});
|
||||
expect(result.fixture.events[0]?.futureEnvelopeField).toBe(42);
|
||||
}
|
||||
});
|
||||
|
||||
it("fails closed on an unsupported required protocol version", async () => {
|
||||
const result = parsePrpFixtureText(
|
||||
await readFile(
|
||||
new URL("unsupported-required-version.json", fixtureDirectory),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
issues: [
|
||||
{
|
||||
code: "unsupported_required_version",
|
||||
path: "/protocolVersion",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("fails closed on unsupported nested required schema versions", async () => {
|
||||
const fixture = await readFixture();
|
||||
const events = fixture.events as Array<Record<string, unknown>>;
|
||||
events[0]!.schemaVersion = 2;
|
||||
expect(parsePrpFixtureText(JSON.stringify(fixture))).toMatchObject({
|
||||
ok: false,
|
||||
issues: [
|
||||
{
|
||||
code: "unsupported_required_version",
|
||||
path: "/events/0/schemaVersion",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects source sequences that cannot be represented exactly", async () => {
|
||||
const fixture = await readFixture();
|
||||
const events = fixture.events as Array<Record<string, unknown>>;
|
||||
events[0]!.sourceSeq = Number.MAX_SAFE_INTEGER + 1;
|
||||
expect(parsePrpFixtureText(JSON.stringify(fixture))).toMatchObject({
|
||||
ok: false,
|
||||
issues: [
|
||||
{
|
||||
code: "schema_validation",
|
||||
path: "/events/0/sourceSeq",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("requires the declared result to match the replayed result event", async () => {
|
||||
const fixture = await readFixture();
|
||||
const result = fixture.result as Record<string, unknown>;
|
||||
result.summary = "A contradictory expected result.";
|
||||
expect(parsePrpFixtureText(JSON.stringify(fixture))).toMatchObject({
|
||||
ok: false,
|
||||
issues: [
|
||||
{
|
||||
code: "binding_mismatch",
|
||||
path: "/result",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects a duplicate event id carrying different content", async () => {
|
||||
const fixture = await readFixture("duplicate-event.json");
|
||||
const events = fixture.events as Array<Record<string, unknown>>;
|
||||
const payload = events[3]!.payload as Record<string, unknown>;
|
||||
payload.text = "A mutated duplicate.";
|
||||
expect(parsePrpFixtureText(JSON.stringify(fixture))).toMatchObject({
|
||||
ok: false,
|
||||
issues: [
|
||||
{
|
||||
code: "binding_mismatch",
|
||||
path: "/events/3/sourceEventId",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("requires exactly one unique terminal event", async () => {
|
||||
const fixture = await readFixture();
|
||||
const events = fixture.events as Array<Record<string, unknown>>;
|
||||
fixture.events = events.filter(
|
||||
(event) => event.eventType !== "run.terminal",
|
||||
);
|
||||
expect(parsePrpFixtureText(JSON.stringify(fixture))).toMatchObject({
|
||||
ok: false,
|
||||
issues: [
|
||||
{
|
||||
code: "binding_mismatch",
|
||||
path: "/events",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("reports invalid JSON without throwing", () => {
|
||||
expect(parsePrpFixtureText("{")).toMatchObject({
|
||||
ok: false,
|
||||
issues: [{ code: "invalid_json", path: "/" }],
|
||||
});
|
||||
});
|
||||
|
||||
it("selects only an overlapping supported protocol version", () => {
|
||||
expect(
|
||||
negotiateProtocolVersion(
|
||||
{ min: 1, max: PRP_PROTOCOL_VERSION },
|
||||
{ min: 1, max: 2 },
|
||||
),
|
||||
).toBe(1);
|
||||
expect(
|
||||
negotiateProtocolVersion({ min: 2, max: 3 }, { min: 1, max: 1 }),
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,457 @@
|
|||
import { Ajv2020 } from "ajv/dist/2020.js";
|
||||
import type { ErrorObject, ValidateFunction } from "ajv/dist/2020.js";
|
||||
import type { FromSchema } from "json-schema-to-ts";
|
||||
|
||||
import {
|
||||
capabilitiesSchema,
|
||||
commandSchema,
|
||||
eventSchema,
|
||||
fixtureSchema,
|
||||
identitySchema,
|
||||
questionSetSchema,
|
||||
requestSchema,
|
||||
resultSchema,
|
||||
semanticToolSchema,
|
||||
stopReasonSchema,
|
||||
terminalSchema,
|
||||
prpSchemaBundle,
|
||||
} from "./generated/schema-bundle.js";
|
||||
import { normalizeLegacyPrpStructuredRunResult } from "./result-normalization.js";
|
||||
|
||||
export const PRP_PROTOCOL_NAME = "paperclip.runner";
|
||||
export const PRP_PROTOCOL_VERSION = 1;
|
||||
export const PRP_FIXTURE_SCHEMA = "paperclip.prp.fixture.v1";
|
||||
|
||||
type TerminalReferences = [typeof stopReasonSchema];
|
||||
type EventReferences = [
|
||||
typeof semanticToolSchema,
|
||||
typeof stopReasonSchema,
|
||||
typeof terminalSchema,
|
||||
typeof resultSchema,
|
||||
];
|
||||
export type PrpIdentity = FromSchema<typeof identitySchema>;
|
||||
export type PrpCapabilities = FromSchema<typeof capabilitiesSchema>;
|
||||
export type PrpCommand = FromSchema<typeof commandSchema>;
|
||||
export type PrpSemanticToolEnvelope = FromSchema<typeof semanticToolSchema>;
|
||||
export type PrpStopReason = FromSchema<typeof stopReasonSchema>;
|
||||
export type PrpTerminalState = FromSchema<
|
||||
typeof terminalSchema,
|
||||
{ references: TerminalReferences }
|
||||
>;
|
||||
type RequestReferences = [typeof questionSetSchema];
|
||||
export type PrpRequest = FromSchema<
|
||||
typeof requestSchema,
|
||||
{ references: RequestReferences }
|
||||
>;
|
||||
export type PrpStructuredRunResult = FromSchema<typeof resultSchema>;
|
||||
export type PrpEvent = FromSchema<
|
||||
typeof eventSchema,
|
||||
{ references: EventReferences }
|
||||
>;
|
||||
/** Runtime-validated composition of the JSON-Schema-derived contract types. */
|
||||
export interface PrpFixture {
|
||||
schema: typeof PRP_FIXTURE_SCHEMA;
|
||||
fixtureVersion: 1;
|
||||
protocolVersion: typeof PRP_PROTOCOL_VERSION;
|
||||
name: string;
|
||||
description: string;
|
||||
identity: PrpIdentity;
|
||||
capabilities: PrpCapabilities;
|
||||
commands: PrpCommand[];
|
||||
events: PrpEvent[];
|
||||
requests?: PrpRequest[];
|
||||
result: PrpStructuredRunResult;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export type ProtocolValidationIssueCode =
|
||||
| "invalid_json"
|
||||
| "schema_validation"
|
||||
| "unsupported_required_version"
|
||||
| "binding_mismatch";
|
||||
|
||||
export interface ProtocolValidationIssue {
|
||||
code: ProtocolValidationIssueCode;
|
||||
path: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export type ProtocolValidationResult =
|
||||
| { ok: true; fixture: PrpFixture; issues: [] }
|
||||
| { ok: false; fixture: null; issues: ProtocolValidationIssue[] };
|
||||
|
||||
export interface ProtocolVersionRange {
|
||||
min: number;
|
||||
max: number;
|
||||
}
|
||||
|
||||
// Runtime validation compiles the same checked-in schemas used to generate
|
||||
// the public TypeScript types. Browser/CSP-specific precompiled validators are
|
||||
// intentionally deferred until the browser SDK package boundary is introduced.
|
||||
const ajv = new Ajv2020({
|
||||
allErrors: true,
|
||||
strict: true,
|
||||
strictRequired: false,
|
||||
formats: {
|
||||
"date-time": /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/,
|
||||
},
|
||||
});
|
||||
for (const schema of Object.values(prpSchemaBundle)) ajv.addSchema(schema);
|
||||
|
||||
function validatorFor<T>(schemaId: string): ValidateFunction<T> {
|
||||
const validator = ajv.getSchema<T>(schemaId);
|
||||
if (validator === undefined) {
|
||||
throw new Error(`Missing generated PRP validator for ${schemaId}`);
|
||||
}
|
||||
return validator;
|
||||
}
|
||||
|
||||
const fixtureValidator = validatorFor<PrpFixture>(fixtureSchema.$id);
|
||||
const eventValidator = validatorFor<PrpEvent>(eventSchema.$id);
|
||||
const resultValidator = validatorFor<PrpStructuredRunResult>(resultSchema.$id);
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> | null {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: null;
|
||||
}
|
||||
|
||||
function canonicalJson(value: unknown): string {
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map(canonicalJson).join(",")}]`;
|
||||
}
|
||||
const object = asRecord(value);
|
||||
if (object !== null) {
|
||||
return `{${Object.keys(object)
|
||||
.sort()
|
||||
.map((key) => `${JSON.stringify(key)}:${canonicalJson(object[key])}`)
|
||||
.join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value) ?? "undefined";
|
||||
}
|
||||
|
||||
function versionIssues(value: unknown): ProtocolValidationIssue[] {
|
||||
const fixture = asRecord(value);
|
||||
if (fixture === null) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const issues: ProtocolValidationIssue[] = [];
|
||||
for (const [field, supported] of [
|
||||
["fixtureVersion", 1],
|
||||
["protocolVersion", PRP_PROTOCOL_VERSION],
|
||||
] as const) {
|
||||
const actual = fixture[field];
|
||||
if (typeof actual === "number" && actual !== supported) {
|
||||
issues.push({
|
||||
code: "unsupported_required_version",
|
||||
path: `/${field}`,
|
||||
message: `${field} ${actual} is unsupported; this implementation requires ${supported}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (Array.isArray(fixture.events)) {
|
||||
fixture.events.forEach((entry, index) => {
|
||||
const event = asRecord(entry);
|
||||
const actual = event?.schemaVersion;
|
||||
if (typeof actual === "number" && actual !== 1) {
|
||||
issues.push({
|
||||
code: "unsupported_required_version",
|
||||
path: `/events/${index}/schemaVersion`,
|
||||
message: `event schemaVersion ${actual} is unsupported; this implementation requires 1`,
|
||||
});
|
||||
}
|
||||
const payload = asRecord(event?.payload);
|
||||
const semanticTool = asRecord(payload?.semantic_tool);
|
||||
const semanticToolVersion = semanticTool?.schemaVersion;
|
||||
if (
|
||||
typeof semanticToolVersion === "number" &&
|
||||
semanticToolVersion !== 1
|
||||
) {
|
||||
issues.push({
|
||||
code: "unsupported_required_version",
|
||||
path: `/events/${index}/payload/semantic_tool/schemaVersion`,
|
||||
message: `semantic_tool schemaVersion ${semanticToolVersion} is unsupported; this implementation requires 1`,
|
||||
});
|
||||
}
|
||||
const stopReason = asRecord(payload?.stopReason);
|
||||
const stopReasonVersion = stopReason?.schemaVersion;
|
||||
if (typeof stopReasonVersion === "number" && stopReasonVersion !== 1) {
|
||||
issues.push({
|
||||
code: "unsupported_required_version",
|
||||
path: `/events/${index}/payload/stopReason/schemaVersion`,
|
||||
message: `stopReason schemaVersion ${stopReasonVersion} is unsupported; this implementation requires 1`,
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
const capabilities = asRecord(fixture.capabilities);
|
||||
const semanticTools = asRecord(capabilities?.semanticTools);
|
||||
const semanticToolsVersion = semanticTools?.schemaVersion;
|
||||
if (typeof semanticToolsVersion === "number" && semanticToolsVersion !== 1) {
|
||||
issues.push({
|
||||
code: "unsupported_required_version",
|
||||
path: "/capabilities/semanticTools/schemaVersion",
|
||||
message: `semanticTools schemaVersion ${semanticToolsVersion} is unsupported; this implementation requires 1`,
|
||||
});
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
|
||||
function ajvIssue(error: ErrorObject): ProtocolValidationIssue {
|
||||
return {
|
||||
code: "schema_validation",
|
||||
path: error.instancePath || "/",
|
||||
message: error.message ?? "does not match the PRP schema",
|
||||
};
|
||||
}
|
||||
|
||||
function bindingIssues(fixture: PrpFixture): ProtocolValidationIssue[] {
|
||||
const issues: ProtocolValidationIssue[] = [];
|
||||
const uniqueEvents = new Map<string, PrpEvent>();
|
||||
const semanticCalls = new Map<
|
||||
string,
|
||||
{
|
||||
input?: { envelope: PrpSemanticToolEnvelope; index: number };
|
||||
result?: { envelope: PrpSemanticToolEnvelope; index: number };
|
||||
}
|
||||
>();
|
||||
fixture.events.forEach((event, index) => {
|
||||
if (event.runId !== fixture.identity.runId) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${index}/runId`,
|
||||
message: "event runId must match identity.runId",
|
||||
});
|
||||
}
|
||||
if (
|
||||
event.normalizedSessionId !== undefined &&
|
||||
event.normalizedSessionId !== fixture.identity.normalizedSessionId
|
||||
) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${index}/normalizedSessionId`,
|
||||
message:
|
||||
"event normalizedSessionId must match identity.normalizedSessionId",
|
||||
});
|
||||
}
|
||||
const existing = uniqueEvents.get(event.sourceEventId);
|
||||
if (existing === undefined) {
|
||||
uniqueEvents.set(event.sourceEventId, event);
|
||||
} else {
|
||||
if (canonicalJson(existing) !== canonicalJson(event)) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${index}/sourceEventId`,
|
||||
message: "duplicate sourceEventId deliveries must be byte-equivalent",
|
||||
});
|
||||
}
|
||||
// At-least-once source delivery is not a second semantic invocation.
|
||||
return;
|
||||
}
|
||||
const payload = asRecord(event.payload);
|
||||
const semanticTool = asRecord(
|
||||
payload?.semantic_tool,
|
||||
) as PrpSemanticToolEnvelope | null;
|
||||
if (semanticTool !== null) {
|
||||
const correlation = asRecord(semanticTool.correlation);
|
||||
for (const [field, actual, expected] of [
|
||||
["runId", correlation?.runId, event.runId],
|
||||
[
|
||||
"normalizedSessionId",
|
||||
correlation?.normalizedSessionId,
|
||||
event.normalizedSessionId,
|
||||
],
|
||||
["turnId", correlation?.turnId, event.turnId],
|
||||
["itemId", correlation?.itemId, event.itemId],
|
||||
] as const) {
|
||||
if (actual !== expected) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${index}/payload/semantic_tool/correlation/${field}`,
|
||||
message: `semantic_tool correlation ${field} must match the containing event`,
|
||||
});
|
||||
}
|
||||
}
|
||||
const call = semanticCalls.get(semanticTool.callId) ?? {};
|
||||
const phase = semanticTool.phase;
|
||||
if (call[phase] !== undefined) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${index}/payload/semantic_tool/callId`,
|
||||
message: `semantic_tool call ${semanticTool.callId} must contain exactly one ${phase} envelope`,
|
||||
});
|
||||
} else {
|
||||
call[phase] = { envelope: semanticTool, index };
|
||||
semanticCalls.set(semanticTool.callId, call);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
for (const [callId, call] of semanticCalls) {
|
||||
if (call.input === undefined || call.result === undefined) {
|
||||
const present = call.input ?? call.result;
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${present?.index ?? 0}/payload/semantic_tool/callId`,
|
||||
message: `semantic_tool call ${callId} must contain one input and one result envelope`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
for (const field of ["operationId", "idempotencyKey"] as const) {
|
||||
if (
|
||||
canonicalJson(call.input.envelope[field]) !==
|
||||
canonicalJson(call.result.envelope[field])
|
||||
) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/events/${call.result.index}/payload/semantic_tool/${field}`,
|
||||
message: `semantic_tool result ${field} must match its input envelope`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fixture.commands.forEach((command, index) => {
|
||||
if (command.controllerSeq !== index + 1) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: `/commands/${index}/controllerSeq`,
|
||||
message: `controllerSeq must be ${index + 1} in a scripted fixture`,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
const distinctEvents = [...uniqueEvents.values()];
|
||||
const proposedResults = distinctEvents.filter(
|
||||
(event) => event.eventType === "run.result.proposed",
|
||||
);
|
||||
if (proposedResults.length !== 1) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: "/events",
|
||||
message:
|
||||
"scripted fixtures must contain exactly one unique run.result.proposed event",
|
||||
});
|
||||
} else if (
|
||||
canonicalJson(proposedResults[0]?.payload) !== canonicalJson(fixture.result)
|
||||
) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: "/result",
|
||||
message:
|
||||
"fixture result must match the run.result.proposed event payload",
|
||||
});
|
||||
}
|
||||
|
||||
const terminalEvents = distinctEvents.filter(
|
||||
(event) => event.eventType === "run.terminal",
|
||||
);
|
||||
if (terminalEvents.length !== 1) {
|
||||
issues.push({
|
||||
code: "binding_mismatch",
|
||||
path: "/events",
|
||||
message:
|
||||
"scripted fixtures must contain exactly one unique run.terminal event",
|
||||
});
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
|
||||
export function validatePrpFixture(value: unknown): ProtocolValidationResult {
|
||||
const unsupported = versionIssues(value);
|
||||
if (unsupported.length > 0) {
|
||||
return { ok: false, fixture: null, issues: unsupported };
|
||||
}
|
||||
if (!fixtureValidator(value)) {
|
||||
return {
|
||||
ok: false,
|
||||
fixture: null,
|
||||
issues: (fixtureValidator.errors ?? []).map(ajvIssue),
|
||||
};
|
||||
}
|
||||
|
||||
const bindings = bindingIssues(value);
|
||||
return bindings.length === 0
|
||||
? { ok: true, fixture: value, issues: [] }
|
||||
: { ok: false, fixture: null, issues: bindings };
|
||||
}
|
||||
|
||||
export function parsePrpFixtureText(text: string): ProtocolValidationResult {
|
||||
try {
|
||||
return validatePrpFixture(JSON.parse(text) as unknown);
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
fixture: null,
|
||||
issues: [
|
||||
{
|
||||
code: "invalid_json",
|
||||
path: "/",
|
||||
message:
|
||||
error instanceof Error
|
||||
? error.message
|
||||
: "fixture is not valid JSON",
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export type EventValidationResult =
|
||||
| { ok: true; event: PrpEvent; issues: [] }
|
||||
| { ok: false; event: null; issues: ProtocolValidationIssue[] };
|
||||
|
||||
export function validatePrpEvent(value: unknown): EventValidationResult {
|
||||
const record = asRecord(value);
|
||||
const schemaVersion = record?.schemaVersion;
|
||||
if (typeof schemaVersion === "number" && schemaVersion !== 1) {
|
||||
return {
|
||||
ok: false,
|
||||
event: null,
|
||||
issues: [
|
||||
{
|
||||
code: "unsupported_required_version",
|
||||
path: "/schemaVersion",
|
||||
message: `event schemaVersion ${schemaVersion} is unsupported; this implementation requires 1`,
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
if (!eventValidator(value)) {
|
||||
return {
|
||||
ok: false,
|
||||
event: null,
|
||||
issues: (eventValidator.errors ?? []).map(ajvIssue),
|
||||
};
|
||||
}
|
||||
return { ok: true, event: value, issues: [] };
|
||||
}
|
||||
|
||||
export type StructuredResultValidationResult =
|
||||
| { ok: true; result: PrpStructuredRunResult; issues: [] }
|
||||
| { ok: false; result: null; issues: ProtocolValidationIssue[] };
|
||||
|
||||
/** Validate a semantic completion independently from a complete replay fixture. */
|
||||
export function validatePrpStructuredRunResult(
|
||||
value: unknown,
|
||||
): StructuredResultValidationResult {
|
||||
const normalized = normalizeLegacyPrpStructuredRunResult(value);
|
||||
if (!resultValidator(normalized)) {
|
||||
return {
|
||||
ok: false,
|
||||
result: null,
|
||||
issues: (resultValidator.errors ?? []).map(ajvIssue),
|
||||
};
|
||||
}
|
||||
return { ok: true, result: normalized, issues: [] };
|
||||
}
|
||||
|
||||
export function negotiateProtocolVersion(
|
||||
runner: ProtocolVersionRange,
|
||||
controller: ProtocolVersionRange,
|
||||
): number | null {
|
||||
const selected = Math.min(runner.max, controller.max);
|
||||
return selected >= Math.max(runner.min, controller.min) ? selected : null;
|
||||
}
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import {
|
||||
parsePrpFixtureText,
|
||||
type ProtocolValidationIssue,
|
||||
type PrpFixture,
|
||||
} from "./replay-contract.js";
|
||||
|
||||
export const replayHappyFixtureUrl = new URL(
|
||||
"../../protocol/fixtures/replay/happy-path.json",
|
||||
import.meta.url,
|
||||
);
|
||||
|
||||
export class PrpFixtureValidationError extends Error {
|
||||
readonly issues: ProtocolValidationIssue[];
|
||||
|
||||
constructor(issues: ProtocolValidationIssue[]) {
|
||||
super(issues.map((issue) => `${issue.path}: ${issue.message}`).join("; "));
|
||||
this.name = "PrpFixtureValidationError";
|
||||
this.issues = issues;
|
||||
}
|
||||
}
|
||||
|
||||
export async function loadPrpFixture(
|
||||
url: URL = replayHappyFixtureUrl,
|
||||
): Promise<PrpFixture> {
|
||||
const result = parsePrpFixtureText(await readFile(url, "utf8"));
|
||||
if (!result.ok) {
|
||||
throw new PrpFixtureValidationError(result.issues);
|
||||
}
|
||||
return result.fixture;
|
||||
}
|
||||
|
|
@ -0,0 +1,274 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
normalizeLegacyPrpStructuredRunResult,
|
||||
normalizePrpResultSignals,
|
||||
} from "./result-normalization.js";
|
||||
import { validatePrpStructuredRunResult } from "./replay-contract.js";
|
||||
|
||||
describe("normalizePrpResultSignals", () => {
|
||||
it("accepts legacy completion aliases as one canonical PRP result", () => {
|
||||
const legacy = {
|
||||
schema: "paperclip.run_result.v1",
|
||||
reportedWorkDisposition: "completed",
|
||||
summary: "Implemented and verified the utility.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{
|
||||
criterionId: "objective",
|
||||
status: "satisfied",
|
||||
evidenceRefs: ["file:a"],
|
||||
},
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [{ ref: "file:a" }],
|
||||
verification: [
|
||||
{
|
||||
command: "node --test",
|
||||
cwd: "fixture",
|
||||
status: "passed",
|
||||
result: "4 passed",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
expect(normalizeLegacyPrpStructuredRunResult(legacy)).toMatchObject({
|
||||
reportedWorkDisposition: "done",
|
||||
attentionRequests: [],
|
||||
artifacts: [],
|
||||
verification: [
|
||||
{ commandOrCheck: "node --test", status: "passed", detail: "4 passed" },
|
||||
],
|
||||
});
|
||||
expect(validatePrpStructuredRunResult(legacy)).toMatchObject({
|
||||
ok: true,
|
||||
result: { reportedWorkDisposition: "done" },
|
||||
});
|
||||
});
|
||||
|
||||
it("restores the canonical schema discriminator omitted by a provider tool caller", () => {
|
||||
const withoutSchema = {
|
||||
reportedWorkDisposition: "done",
|
||||
summary: "Completed the requested work.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{ criterionId: "objective", status: "satisfied", evidenceRefs: [] },
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [],
|
||||
};
|
||||
expect(validatePrpStructuredRunResult(withoutSchema)).toMatchObject({
|
||||
ok: true,
|
||||
result: { schema: "paperclip.run_result.v1" },
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes unambiguous completion aliases emitted by smaller tool callers", () => {
|
||||
const toolShaped = {
|
||||
schema: "paperclip_paperclip_finish",
|
||||
reportedWorkDisposition: "completed",
|
||||
summary: "Answered the question.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{ criterionId: "objective", status: "passed", evidenceRefs: [] },
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [],
|
||||
};
|
||||
expect(validatePrpStructuredRunResult(toolShaped)).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
schema: "paperclip.run_result.v1",
|
||||
reportedWorkDisposition: "done",
|
||||
completionClaim: {
|
||||
criteria: [{ status: "satisfied" }],
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes unambiguous verification aliases from smaller tool callers", () => {
|
||||
expect(
|
||||
validatePrpStructuredRunResult({
|
||||
reportedWorkDisposition: "completed",
|
||||
summary: "Answered the question.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{ criterionId: "objective", status: "pass", evidenceRefs: [] },
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [{ commandOrCheck: "model check", status: "success" }],
|
||||
}),
|
||||
).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
verification: [{ commandOrCheck: "model check", status: "passed" }],
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("turns legacy environment_constraint attention into a verification caveat", () => {
|
||||
const normalized = normalizePrpResultSignals({
|
||||
verification: [
|
||||
{
|
||||
commandOrCheck: "Run npm test",
|
||||
status: "not_run",
|
||||
},
|
||||
],
|
||||
attentionRequests: [
|
||||
{
|
||||
kind: "environment_constraint",
|
||||
summary: "Node and npm are unavailable in this sandbox.",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalized.actionableAttentionRequests).toEqual([]);
|
||||
expect(normalized.verification).toEqual([
|
||||
expect.objectContaining({
|
||||
commandOrCheck: "Run npm test",
|
||||
status: "not_run",
|
||||
reasonCode: "tool_unavailable",
|
||||
detail: "Node and npm are unavailable in this sandbox.",
|
||||
}),
|
||||
]);
|
||||
expect(normalized.ignoredAttentionRequests).toEqual([
|
||||
expect.objectContaining({
|
||||
sourceKind: "environment_constraint",
|
||||
disposition: "verification_caveat",
|
||||
reasonCode: "legacy_environment_constraint_normalized",
|
||||
}),
|
||||
]);
|
||||
|
||||
const canonical = normalizeLegacyPrpStructuredRunResult({
|
||||
schema: "paperclip.run_result.v1",
|
||||
reportedWorkDisposition: "done",
|
||||
summary: "Work is complete; local verification was unavailable.",
|
||||
completionClaim: {
|
||||
contractRevision: "1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [
|
||||
{ criterionId: "objective", status: "satisfied", evidenceRefs: [] },
|
||||
],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [
|
||||
{
|
||||
command: "npm test",
|
||||
status: "blocked",
|
||||
result: "Node is unavailable.",
|
||||
},
|
||||
],
|
||||
attentionRequests: [
|
||||
{
|
||||
kind: "environment_constraint",
|
||||
summary: "Node and npm are unavailable in this sandbox.",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(canonical).toMatchObject({
|
||||
attentionRequests: [],
|
||||
verification: [
|
||||
{
|
||||
commandOrCheck: "npm test",
|
||||
status: "not_run",
|
||||
reasonCode: "tool_unavailable",
|
||||
detail:
|
||||
"Node is unavailable.\nNode and npm are unavailable in this sandbox.",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(validatePrpStructuredRunResult(canonical)).toMatchObject({
|
||||
ok: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes an unavailable command reported as failed into a not-run caveat", () => {
|
||||
const normalized = normalizePrpResultSignals({
|
||||
verification: [
|
||||
{
|
||||
commandOrCheck: "tsc --noEmit",
|
||||
status: "failed",
|
||||
reasonCode: "dependency_missing",
|
||||
detail:
|
||||
"Repository dependencies are not installed, so the compiler could not start.",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalized.verification).toEqual([
|
||||
expect.objectContaining({
|
||||
commandOrCheck: "tsc --noEmit",
|
||||
status: "not_run",
|
||||
reasonCode: "dependency_missing",
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("preserves typed actionable requests and legacy review requests", () => {
|
||||
const normalized = normalizePrpResultSignals({
|
||||
attentionRequests: [
|
||||
{
|
||||
kind: "approval",
|
||||
summary: "Approve publication",
|
||||
ownerClass: "human",
|
||||
},
|
||||
{ kind: "review", summary: "Review the result" },
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalized.actionableAttentionRequests).toEqual([
|
||||
expect.objectContaining({
|
||||
kind: "approval",
|
||||
ownerClass: "human",
|
||||
legacy: false,
|
||||
}),
|
||||
expect.objectContaining({
|
||||
kind: "review",
|
||||
ownerClass: "human",
|
||||
legacy: true,
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("records unknown and malformed attention as diagnostics instead of throwing", () => {
|
||||
const normalized = normalizePrpResultSignals({
|
||||
attentionRequests: [
|
||||
{ kind: "future_provider_value", summary: "Unknown request" },
|
||||
{ kind: "approval" },
|
||||
null,
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalized.actionableAttentionRequests).toEqual([]);
|
||||
expect(normalized.ignoredAttentionRequests).toEqual([
|
||||
expect.objectContaining({
|
||||
disposition: "ignored",
|
||||
reasonCode: "attention_request_kind_unsupported",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
disposition: "rejected",
|
||||
reasonCode: "attention_request_malformed",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
disposition: "rejected",
|
||||
reasonCode: "attention_request_malformed",
|
||||
}),
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,410 @@
|
|||
import {
|
||||
PRP_ACTIONABLE_ATTENTION_KINDS,
|
||||
PRP_ATTENTION_OWNER_CLASSES,
|
||||
PRP_VERIFICATION_REASON_CODES,
|
||||
} from "../contracts/completion-result.js";
|
||||
|
||||
export type PrpVerificationReasonCode =
|
||||
(typeof PRP_VERIFICATION_REASON_CODES)[number];
|
||||
export type PrpActionableAttentionKind =
|
||||
(typeof PRP_ACTIONABLE_ATTENTION_KINDS)[number];
|
||||
export type PrpAttentionOwnerClass =
|
||||
(typeof PRP_ATTENTION_OWNER_CLASSES)[number];
|
||||
|
||||
export interface PrpNormalizedVerification {
|
||||
commandOrCheck: string;
|
||||
status: "passed" | "failed" | "not_run";
|
||||
reasonCode: PrpVerificationReasonCode | null;
|
||||
detail: string | null;
|
||||
artifactRef: string | null;
|
||||
sourceIndex: number;
|
||||
}
|
||||
|
||||
export interface PrpNormalizedAttentionRequest {
|
||||
kind: PrpActionableAttentionKind;
|
||||
summary: string;
|
||||
ownerClass: PrpAttentionOwnerClass;
|
||||
targetAgentId: string | null;
|
||||
sourceIndex: number;
|
||||
sourceKind: string;
|
||||
legacy: boolean;
|
||||
}
|
||||
|
||||
export interface PrpIgnoredAttentionRequest {
|
||||
sourceIndex: number;
|
||||
sourceKind: string | null;
|
||||
summary: string | null;
|
||||
disposition: "verification_caveat" | "ignored" | "rejected";
|
||||
reasonCode: string;
|
||||
}
|
||||
|
||||
export interface PrpResultSignals {
|
||||
verification: PrpNormalizedVerification[];
|
||||
actionableAttentionRequests: PrpNormalizedAttentionRequest[];
|
||||
ignoredAttentionRequests: PrpIgnoredAttentionRequest[];
|
||||
}
|
||||
|
||||
const verificationReasonCodes = new Set<string>(PRP_VERIFICATION_REASON_CODES);
|
||||
const unavailableVerificationReasonCodes = new Set<string>(
|
||||
PRP_VERIFICATION_REASON_CODES.filter((reasonCode) => reasonCode !== "other"),
|
||||
);
|
||||
const attentionKinds = new Set<string>(PRP_ACTIONABLE_ATTENTION_KINDS);
|
||||
const ownerClasses = new Set<string>(PRP_ATTENTION_OWNER_CLASSES);
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: {};
|
||||
}
|
||||
|
||||
function text(value: unknown): string | null {
|
||||
return typeof value === "string" && value.trim().length > 0
|
||||
? value.trim()
|
||||
: null;
|
||||
}
|
||||
|
||||
function inferVerificationReason(value: string): PrpVerificationReasonCode {
|
||||
const lower = value.toLowerCase();
|
||||
if (/budget|quota|token limit/.test(lower)) return "budget_exhausted";
|
||||
if (
|
||||
/credential|api key|secret|login|auth(?:entication)? required/.test(lower)
|
||||
)
|
||||
return "credential_missing";
|
||||
if (/permission|denied|forbidden|not allowed/.test(lower))
|
||||
return "permission_denied";
|
||||
if (/policy|governance|approval policy/.test(lower))
|
||||
return "policy_restricted";
|
||||
if (/external|service|upstream|network|offline|dns/.test(lower))
|
||||
return "external_service_unavailable";
|
||||
if (/dependency|package|module|library/.test(lower))
|
||||
return "dependency_missing";
|
||||
if (/tool|command|binary|node|npm|sandbox/.test(lower))
|
||||
return "tool_unavailable";
|
||||
if (/environment|runtime|platform/.test(lower))
|
||||
return "environment_unavailable";
|
||||
return "other";
|
||||
}
|
||||
|
||||
function normalizeVerification(
|
||||
result: Record<string, unknown>,
|
||||
): PrpNormalizedVerification[] {
|
||||
const entries = Array.isArray(result.verification) ? result.verification : [];
|
||||
return entries.map((value, sourceIndex) => {
|
||||
const entry = record(value);
|
||||
const rawStatus = String(entry.status);
|
||||
const reportedStatus: PrpNormalizedVerification["status"] = [
|
||||
"pass",
|
||||
"success",
|
||||
"succeeded",
|
||||
].includes(rawStatus)
|
||||
? "passed"
|
||||
: rawStatus === "fail"
|
||||
? "failed"
|
||||
: ["passed", "failed", "not_run"].includes(rawStatus)
|
||||
? (rawStatus as PrpNormalizedVerification["status"])
|
||||
: "not_run";
|
||||
const detail = text(entry.detail) ?? text(entry.result);
|
||||
const commandOrCheck =
|
||||
text(entry.commandOrCheck) ??
|
||||
text(entry.command) ??
|
||||
`verification-${sourceIndex + 1}`;
|
||||
const reportedReason = text(entry.reasonCode);
|
||||
const normalizedReportedReason =
|
||||
reportedReason && verificationReasonCodes.has(reportedReason)
|
||||
? (reportedReason as PrpVerificationReasonCode)
|
||||
: null;
|
||||
// Provider models sometimes report an unavailable command as `failed`
|
||||
// because the shell returned non-zero, while also supplying a reason code
|
||||
// that explicitly says no meaningful verification verdict was possible.
|
||||
// The reason code is the stronger semantic signal: reserve `failed` for a
|
||||
// check that actually ran and found the work incorrect.
|
||||
const status =
|
||||
reportedStatus === "failed" &&
|
||||
normalizedReportedReason &&
|
||||
unavailableVerificationReasonCodes.has(normalizedReportedReason)
|
||||
? "not_run"
|
||||
: reportedStatus;
|
||||
const reasonCode =
|
||||
normalizedReportedReason ??
|
||||
(status === "not_run"
|
||||
? inferVerificationReason(`${commandOrCheck} ${detail ?? ""}`)
|
||||
: null);
|
||||
return {
|
||||
commandOrCheck,
|
||||
status,
|
||||
reasonCode,
|
||||
detail,
|
||||
artifactRef: text(entry.artifactRef),
|
||||
sourceIndex,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Normalize accepted PRP v1 aliases before strict canonical validation. */
|
||||
export function normalizeLegacyPrpStructuredRunResult(value: unknown): unknown {
|
||||
const source = record(value);
|
||||
if (Object.keys(source).length === 0) return value;
|
||||
const normalized: Record<string, unknown> = { ...source };
|
||||
// Provider tool-callers do not consistently echo constant discriminator
|
||||
// fields even when they are present in the advertised JSON schema. The
|
||||
// transport already selected the semantic-result tool, so adding its
|
||||
// canonical discriminator here is deterministic rather than inferential.
|
||||
const reportedSchema = text(normalized.schema);
|
||||
if (
|
||||
reportedSchema === null ||
|
||||
[
|
||||
"paperclip_finish",
|
||||
"paperclip_paperclip_finish",
|
||||
"paperclip_finish.v1",
|
||||
].includes(reportedSchema)
|
||||
) {
|
||||
normalized.schema = "paperclip.run_result.v1";
|
||||
}
|
||||
const signals = normalizePrpResultSignals(source);
|
||||
if (
|
||||
["complete", "completed"].includes(
|
||||
String(normalized.reportedWorkDisposition),
|
||||
)
|
||||
) {
|
||||
normalized.reportedWorkDisposition = "done";
|
||||
}
|
||||
const completionClaim = record(normalized.completionClaim);
|
||||
if (Array.isArray(completionClaim.criteria)) {
|
||||
normalized.completionClaim = {
|
||||
...completionClaim,
|
||||
criteria: completionClaim.criteria.map((value) => {
|
||||
const criterion = record(value);
|
||||
const reportedStatus = text(criterion.status);
|
||||
const status =
|
||||
reportedStatus &&
|
||||
["pass", "passed", "complete", "completed"].includes(reportedStatus)
|
||||
? "satisfied"
|
||||
: reportedStatus &&
|
||||
["fail", "failed", "incomplete"].includes(reportedStatus)
|
||||
? "not_satisfied"
|
||||
: criterion.status;
|
||||
return { ...criterion, status };
|
||||
}),
|
||||
};
|
||||
}
|
||||
normalized.attentionRequests = signals.actionableAttentionRequests.map(
|
||||
(entry) => ({
|
||||
kind: entry.kind,
|
||||
summary: entry.summary,
|
||||
ownerClass: entry.ownerClass,
|
||||
...(entry.targetAgentId ? { targetAgentId: entry.targetAgentId } : {}),
|
||||
}),
|
||||
);
|
||||
normalized.artifacts = Array.isArray(source.artifacts)
|
||||
? source.artifacts
|
||||
: [];
|
||||
normalized.verification = signals.verification.map((entry) => ({
|
||||
commandOrCheck: entry.commandOrCheck,
|
||||
status: entry.status,
|
||||
...(entry.reasonCode ? { reasonCode: entry.reasonCode } : {}),
|
||||
...(entry.detail ? { detail: entry.detail } : {}),
|
||||
...(entry.artifactRef ? { artifactRef: entry.artifactRef } : {}),
|
||||
}));
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function legacyAttentionRequest(
|
||||
entry: Record<string, unknown>,
|
||||
sourceIndex: number,
|
||||
sourceKind: string,
|
||||
summary: string,
|
||||
): PrpNormalizedAttentionRequest | null {
|
||||
const target = record(entry.target);
|
||||
const targetAgentId = text(entry.targetAgentId) ?? text(target.agentId);
|
||||
const ownerClass = text(entry.ownerClass) ?? text(target.ownerClass);
|
||||
const requiredAuthority = text(entry.requiredAuthority);
|
||||
switch (sourceKind) {
|
||||
case "credential_grant":
|
||||
return {
|
||||
kind: "credential",
|
||||
summary,
|
||||
ownerClass: "human",
|
||||
targetAgentId: null,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: true,
|
||||
};
|
||||
case "governed_approval":
|
||||
return {
|
||||
kind: "approval",
|
||||
summary,
|
||||
ownerClass: "human",
|
||||
targetAgentId: null,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: true,
|
||||
};
|
||||
case "subjective_decision":
|
||||
return {
|
||||
kind: "human_input",
|
||||
summary,
|
||||
ownerClass: "human",
|
||||
targetAgentId: null,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: true,
|
||||
};
|
||||
case "domain_expertise":
|
||||
return targetAgentId
|
||||
? {
|
||||
kind: "agent_handoff",
|
||||
summary,
|
||||
ownerClass: "agent",
|
||||
targetAgentId,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: true,
|
||||
}
|
||||
: null;
|
||||
case "external_action":
|
||||
return {
|
||||
kind: "external_action",
|
||||
summary,
|
||||
ownerClass:
|
||||
ownerClass === "external_system" || requiredAuthority === "external"
|
||||
? "external_system"
|
||||
: "human",
|
||||
targetAgentId: null,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: true,
|
||||
};
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize model-authored result signals without rewriting the immutable PRP result.
|
||||
* Unknown legacy attention is diagnostic data, never a finalization exception.
|
||||
*/
|
||||
export function normalizePrpResultSignals(value: unknown): PrpResultSignals {
|
||||
const result = record(value);
|
||||
const verification = normalizeVerification(result);
|
||||
const actionableAttentionRequests: PrpNormalizedAttentionRequest[] = [];
|
||||
const ignoredAttentionRequests: PrpIgnoredAttentionRequest[] = [];
|
||||
const requests = Array.isArray(result.attentionRequests)
|
||||
? result.attentionRequests
|
||||
: [];
|
||||
|
||||
requests.forEach((value, sourceIndex) => {
|
||||
const entry = record(value);
|
||||
const sourceKind = text(entry.kind) ?? text(entry.requestedCapability);
|
||||
const summary = text(entry.summary);
|
||||
if (!sourceKind || !summary) {
|
||||
ignoredAttentionRequests.push({
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
summary,
|
||||
disposition: "rejected",
|
||||
reasonCode: "attention_request_malformed",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (sourceKind === "environment_constraint") {
|
||||
const target = verification.find(
|
||||
(candidate) => candidate.status === "not_run",
|
||||
);
|
||||
if (target) {
|
||||
target.detail = target.detail
|
||||
? target.detail.includes(summary)
|
||||
? target.detail
|
||||
: `${target.detail}\n${summary}`
|
||||
: summary;
|
||||
target.reasonCode = inferVerificationReason(
|
||||
`${target.commandOrCheck} ${target.detail}`,
|
||||
);
|
||||
}
|
||||
ignoredAttentionRequests.push({
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
summary,
|
||||
disposition: "verification_caveat",
|
||||
reasonCode: target
|
||||
? "legacy_environment_constraint_normalized"
|
||||
: "legacy_environment_constraint_unbound",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (attentionKinds.has(sourceKind) && text(entry.ownerClass)) {
|
||||
const ownerClass = text(entry.ownerClass);
|
||||
const targetAgentId = text(entry.targetAgentId);
|
||||
if (
|
||||
!ownerClass ||
|
||||
!ownerClasses.has(ownerClass) ||
|
||||
(ownerClass === "agent" && !targetAgentId)
|
||||
) {
|
||||
ignoredAttentionRequests.push({
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
summary,
|
||||
disposition: "rejected",
|
||||
reasonCode: "attention_request_owner_invalid",
|
||||
});
|
||||
return;
|
||||
}
|
||||
actionableAttentionRequests.push({
|
||||
kind: sourceKind as PrpActionableAttentionKind,
|
||||
summary,
|
||||
ownerClass: ownerClass as PrpAttentionOwnerClass,
|
||||
targetAgentId,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: false,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (sourceKind === "review") {
|
||||
actionableAttentionRequests.push({
|
||||
kind: "review",
|
||||
summary,
|
||||
ownerClass: "human",
|
||||
targetAgentId: null,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
legacy: true,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const legacy = legacyAttentionRequest(
|
||||
entry,
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
summary,
|
||||
);
|
||||
if (legacy) {
|
||||
actionableAttentionRequests.push(legacy);
|
||||
return;
|
||||
}
|
||||
|
||||
ignoredAttentionRequests.push({
|
||||
sourceIndex,
|
||||
sourceKind,
|
||||
summary,
|
||||
disposition: "ignored",
|
||||
reasonCode: [
|
||||
"context_lookup",
|
||||
"retry",
|
||||
"duplicate",
|
||||
"alternate_track",
|
||||
].includes(sourceKind)
|
||||
? "attention_request_internal_capability_not_provider_actionable"
|
||||
: "attention_request_kind_unsupported",
|
||||
});
|
||||
});
|
||||
|
||||
return {
|
||||
verification,
|
||||
actionableAttentionRequests,
|
||||
ignoredAttentionRequests,
|
||||
};
|
||||
}
|
||||
|
|
@ -0,0 +1,147 @@
|
|||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
parsePrpFixtureText,
|
||||
type PrpEvent,
|
||||
type PrpFixture,
|
||||
} from "../protocol/replay-contract.js";
|
||||
import {
|
||||
createSessionSnapshot,
|
||||
MAX_RECORDED_MISSING_SEQUENCES,
|
||||
reducePrpFixture,
|
||||
reduceSessionEvents,
|
||||
type SessionSnapshot,
|
||||
} from "./session-reducer.js";
|
||||
|
||||
const fixtureDirectory = new URL(
|
||||
"../../protocol/fixtures/replay/",
|
||||
import.meta.url,
|
||||
);
|
||||
const fixtureNames = [
|
||||
"happy-path",
|
||||
"failed-run",
|
||||
"interrupted-run",
|
||||
"duplicate-event",
|
||||
"source-gap",
|
||||
"unknown-optional-fields",
|
||||
];
|
||||
|
||||
async function loadFixture(name: string): Promise<PrpFixture> {
|
||||
const result = parsePrpFixtureText(
|
||||
await readFile(new URL(`${name}.json`, fixtureDirectory), "utf8"),
|
||||
);
|
||||
if (!result.ok) {
|
||||
throw new Error(result.issues.map((issue) => issue.message).join("; "));
|
||||
}
|
||||
return result.fixture;
|
||||
}
|
||||
|
||||
async function loadGolden(name: string): Promise<SessionSnapshot> {
|
||||
return JSON.parse(
|
||||
await readFile(
|
||||
new URL(`golden/${name}.snapshot.json`, fixtureDirectory),
|
||||
"utf8",
|
||||
),
|
||||
) as SessionSnapshot;
|
||||
}
|
||||
|
||||
describe("deterministic PRP session reducer", () => {
|
||||
for (const fixtureName of fixtureNames) {
|
||||
it(`matches the ${fixtureName} golden snapshot`, async () => {
|
||||
expect(reducePrpFixture(await loadFixture(fixtureName))).toEqual(
|
||||
await loadGolden(fixtureName),
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
it("is deterministic and idempotent when the same batch is replayed", async () => {
|
||||
const fixture = await loadFixture("happy-path");
|
||||
const first = reducePrpFixture(fixture);
|
||||
expect(reducePrpFixture(fixture)).toEqual(first);
|
||||
expect(reduceSessionEvents(first, fixture.events)).toEqual(first);
|
||||
});
|
||||
|
||||
it("deduplicates at-least-once delivery before projection effects", async () => {
|
||||
const snapshot = reducePrpFixture(await loadFixture("duplicate-event"));
|
||||
expect(snapshot.duplicateEventIds).toEqual(["event_duplicate_03"]);
|
||||
expect(snapshot.items).toHaveLength(1);
|
||||
expect(snapshot.timeline).toHaveLength(6);
|
||||
});
|
||||
|
||||
it("records a source cursor gap without inventing the missing event", async () => {
|
||||
const snapshot = reducePrpFixture(await loadFixture("source-gap"));
|
||||
expect(snapshot.integrity).toBe("gap_detected");
|
||||
expect(snapshot.gaps).toEqual([
|
||||
{
|
||||
sourceKey: "runner:runner_replay",
|
||||
expected: 3,
|
||||
received: 4,
|
||||
missingCount: 1,
|
||||
missing: [3],
|
||||
truncated: false,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("bounds sequence-gap details for an untrusted large cursor jump", async () => {
|
||||
const fixture = await loadFixture("source-gap");
|
||||
const event: PrpEvent = {
|
||||
...fixture.events[0]!,
|
||||
sourceEventId: "event_large_gap",
|
||||
sourceSeq: Number.MAX_SAFE_INTEGER,
|
||||
};
|
||||
|
||||
const snapshot = reduceSessionEvents(createSessionSnapshot(fixture), [
|
||||
event,
|
||||
]);
|
||||
|
||||
expect(snapshot.gaps).toHaveLength(1);
|
||||
expect(snapshot.gaps[0]).toMatchObject({
|
||||
expected: 1,
|
||||
received: Number.MAX_SAFE_INTEGER,
|
||||
missingCount: Number.MAX_SAFE_INTEGER - 1,
|
||||
truncated: true,
|
||||
});
|
||||
expect(snapshot.gaps[0]?.missing).toHaveLength(
|
||||
MAX_RECORDED_MISSING_SEQUENCES,
|
||||
);
|
||||
expect(snapshot.gaps[0]?.missing.at(-1)).toBe(
|
||||
MAX_RECORDED_MISSING_SEQUENCES,
|
||||
);
|
||||
});
|
||||
|
||||
it("summarizes runtime request creation and resolution with type and prompt", async () => {
|
||||
const fixture = await loadFixture("interrupted-run");
|
||||
const fixtureRequest = fixture.events.find(
|
||||
(event) => event.eventType === "runtime_request.created",
|
||||
);
|
||||
if (fixtureRequest === undefined) {
|
||||
throw new Error("interrupted-run fixture must create a runtime request");
|
||||
}
|
||||
const created: PrpEvent = {
|
||||
...fixtureRequest,
|
||||
sourceEventId: "event_request_summary_01",
|
||||
sourceSeq: 1,
|
||||
};
|
||||
const resolved: PrpEvent = {
|
||||
...fixtureRequest,
|
||||
sourceEventId: "event_request_summary_02",
|
||||
sourceSeq: 2,
|
||||
eventType: "runtime_request.resolved",
|
||||
payload: { requestId: "request_interrupted_permission" },
|
||||
};
|
||||
|
||||
const snapshot = reduceSessionEvents(createSessionSnapshot(fixture), [
|
||||
created,
|
||||
resolved,
|
||||
]);
|
||||
|
||||
expect(snapshot.requests[0]).toMatchObject({ type: "permission" });
|
||||
expect(snapshot.timeline.map((entry) => entry.summary)).toEqual([
|
||||
"permission: Allow the fake command?",
|
||||
"Resolved permission: Allow the fake command?",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,440 @@
|
|||
import type {
|
||||
PrpCapabilities,
|
||||
PrpEvent,
|
||||
PrpFixture,
|
||||
PrpIdentity,
|
||||
PrpStructuredRunResult,
|
||||
PrpTerminalState,
|
||||
} from "../protocol/replay-contract.js";
|
||||
|
||||
export interface SessionTimelineEntry {
|
||||
position: number;
|
||||
sourceEventId: string;
|
||||
sourceSeq: number;
|
||||
eventType: PrpEvent["eventType"];
|
||||
emittedAt: string;
|
||||
itemId?: string;
|
||||
summary: string;
|
||||
}
|
||||
|
||||
export interface SessionItemSnapshot {
|
||||
itemId: string;
|
||||
kind: string;
|
||||
status: "running" | "completed" | "failed";
|
||||
text: string;
|
||||
}
|
||||
|
||||
export interface SessionRequestSnapshot {
|
||||
requestId: string;
|
||||
requestKind: string;
|
||||
type: string;
|
||||
status: string;
|
||||
prompt: string;
|
||||
}
|
||||
|
||||
export interface SequenceGap {
|
||||
sourceKey: string;
|
||||
expected: number;
|
||||
received: number;
|
||||
missingCount: number;
|
||||
missing: number[];
|
||||
truncated: boolean;
|
||||
}
|
||||
|
||||
export const MAX_RECORDED_MISSING_SEQUENCES = 256;
|
||||
|
||||
export interface SessionSnapshot {
|
||||
schema: "paperclip.prp.session-snapshot.v1";
|
||||
fixtureName: string;
|
||||
identity: PrpIdentity;
|
||||
capabilities: PrpCapabilities;
|
||||
runPhase: string;
|
||||
sessionState: "not_started" | "running" | "closed" | "failed";
|
||||
turnState:
|
||||
| "not_started"
|
||||
| "running"
|
||||
| "completed"
|
||||
| "failed"
|
||||
| "interrupted"
|
||||
| "cancelled";
|
||||
activeTurnId: string | null;
|
||||
items: SessionItemSnapshot[];
|
||||
requests: SessionRequestSnapshot[];
|
||||
proposedResult: PrpStructuredRunResult | null;
|
||||
terminal: PrpTerminalState | null;
|
||||
timeline: SessionTimelineEntry[];
|
||||
sourceCursors: Record<string, number>;
|
||||
processedEventIds: string[];
|
||||
duplicateEventIds: string[];
|
||||
outOfOrderEventIds: string[];
|
||||
gaps: SequenceGap[];
|
||||
integrity: "complete" | "gap_detected";
|
||||
}
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: {};
|
||||
}
|
||||
|
||||
function stringValue(value: unknown, fallback = ""): string {
|
||||
return typeof value === "string" ? value : fallback;
|
||||
}
|
||||
|
||||
function sourceKey(event: PrpEvent): string {
|
||||
return `${event.sourceKind}:${event.sourceInstanceId}`;
|
||||
}
|
||||
|
||||
function humanizeProtocolValue(value: string): string {
|
||||
return value.replaceAll("_", " ");
|
||||
}
|
||||
|
||||
function sentenceCase(value: string): string {
|
||||
return value.length === 0
|
||||
? value
|
||||
: `${value[0]?.toUpperCase()}${value.slice(1)}`;
|
||||
}
|
||||
|
||||
function requestSummary(type: string, prompt: string, action?: string): string {
|
||||
const detail = prompt.length > 0 ? `${type}: ${prompt}` : `${type} request`;
|
||||
return action === undefined ? detail : `${action} ${detail}`;
|
||||
}
|
||||
|
||||
function timelineSummary(snapshot: SessionSnapshot, event: PrpEvent): string {
|
||||
const payload = record(event.payload);
|
||||
if (event.eventType === "runtime.phase.changed") {
|
||||
return `Phase changed to ${humanizeProtocolValue(stringValue(payload.phase, "unknown"))}`;
|
||||
}
|
||||
if (event.eventType === "item.started") {
|
||||
return `Started ${stringValue(payload.kind, "item")}`;
|
||||
}
|
||||
if (event.eventType === "item.delta") {
|
||||
return stringValue(payload.text, "Item updated");
|
||||
}
|
||||
if (event.eventType === "item.completed") {
|
||||
return `Completed ${stringValue(payload.kind, "item")}`;
|
||||
}
|
||||
if (event.eventType === "item.failed") {
|
||||
return stringValue(payload.message, "Item failed");
|
||||
}
|
||||
if (event.eventType === "run.result.proposed") {
|
||||
return stringValue(payload.summary, "Structured result proposed");
|
||||
}
|
||||
if (event.eventType === "run.terminal") {
|
||||
return `Run ${stringValue(payload.runTerminalState, "terminal")}`;
|
||||
}
|
||||
if (
|
||||
event.eventType === "runner.diagnostic" ||
|
||||
event.eventType === "harness.diagnostic"
|
||||
) {
|
||||
const message = stringValue(payload.message);
|
||||
if (message.length > 0) {
|
||||
return message;
|
||||
}
|
||||
const code = stringValue(payload.code);
|
||||
if (code.length > 0) {
|
||||
return sentenceCase(humanizeProtocolValue(code));
|
||||
}
|
||||
}
|
||||
if (event.eventType === "runtime_request.created") {
|
||||
const request = record(payload.request ?? payload);
|
||||
return requestSummary(
|
||||
stringValue(request.type, "runtime"),
|
||||
stringValue(request.prompt),
|
||||
);
|
||||
}
|
||||
if (
|
||||
event.eventType === "runtime_request.resolved" ||
|
||||
event.eventType === "runtime_request.expired" ||
|
||||
event.eventType === "runtime_request.cancelled"
|
||||
) {
|
||||
const requestId = stringValue(payload.requestId);
|
||||
const request = snapshot.requests.find(
|
||||
(candidate) => candidate.requestId === requestId,
|
||||
);
|
||||
const action = sentenceCase(
|
||||
humanizeProtocolValue(event.eventType.split(".").at(-1) ?? "updated"),
|
||||
);
|
||||
return request === undefined
|
||||
? `${action} runtime request ${requestId}`.trim()
|
||||
: requestSummary(request.type, request.prompt, action);
|
||||
}
|
||||
return event.eventType.replaceAll(".", " ");
|
||||
}
|
||||
|
||||
function upsertItem(
|
||||
items: SessionItemSnapshot[],
|
||||
event: PrpEvent,
|
||||
status: SessionItemSnapshot["status"],
|
||||
): void {
|
||||
if (event.itemId === undefined) {
|
||||
return;
|
||||
}
|
||||
const payload = record(event.payload);
|
||||
const existing = items.find((item) => item.itemId === event.itemId);
|
||||
const text = stringValue(payload.text);
|
||||
if (existing === undefined) {
|
||||
items.push({
|
||||
itemId: event.itemId,
|
||||
kind: stringValue(payload.kind, "diagnostic"),
|
||||
status,
|
||||
text,
|
||||
});
|
||||
return;
|
||||
}
|
||||
existing.kind = stringValue(payload.kind, existing.kind);
|
||||
existing.status = status;
|
||||
if (event.eventType === "item.delta") {
|
||||
existing.text += text;
|
||||
} else if (text.length > 0) {
|
||||
existing.text = text;
|
||||
}
|
||||
}
|
||||
|
||||
function applyProjection(snapshot: SessionSnapshot, event: PrpEvent): void {
|
||||
const payload = record(event.payload);
|
||||
switch (event.eventType) {
|
||||
case "runtime.phase.changed":
|
||||
snapshot.runPhase = stringValue(payload.phase, snapshot.runPhase);
|
||||
break;
|
||||
case "session.started":
|
||||
case "session.resumed":
|
||||
snapshot.sessionState = "running";
|
||||
break;
|
||||
case "session.closed":
|
||||
snapshot.sessionState = "closed";
|
||||
break;
|
||||
case "session.failed":
|
||||
snapshot.sessionState = "failed";
|
||||
break;
|
||||
case "turn.started":
|
||||
snapshot.turnState = "running";
|
||||
snapshot.activeTurnId = event.turnId ?? null;
|
||||
break;
|
||||
case "turn.completed":
|
||||
snapshot.turnState = "completed";
|
||||
snapshot.activeTurnId = null;
|
||||
break;
|
||||
case "turn.failed":
|
||||
snapshot.turnState = "failed";
|
||||
snapshot.activeTurnId = null;
|
||||
break;
|
||||
case "turn.interrupted":
|
||||
snapshot.turnState = "interrupted";
|
||||
snapshot.activeTurnId = null;
|
||||
break;
|
||||
case "turn.cancelled":
|
||||
snapshot.turnState = "cancelled";
|
||||
snapshot.activeTurnId = null;
|
||||
break;
|
||||
case "item.started":
|
||||
case "item.delta":
|
||||
upsertItem(snapshot.items, event, "running");
|
||||
break;
|
||||
case "item.completed":
|
||||
upsertItem(snapshot.items, event, "completed");
|
||||
break;
|
||||
case "item.failed":
|
||||
upsertItem(snapshot.items, event, "failed");
|
||||
break;
|
||||
case "runtime_request.created": {
|
||||
const request = record(payload.request ?? payload);
|
||||
const requestId = stringValue(request.requestId);
|
||||
if (
|
||||
requestId.length > 0 &&
|
||||
!snapshot.requests.some(
|
||||
(candidate) => candidate.requestId === requestId,
|
||||
)
|
||||
) {
|
||||
snapshot.requests.push({
|
||||
requestId,
|
||||
requestKind: stringValue(request.requestKind, "runtime"),
|
||||
type: stringValue(request.type, "runtime"),
|
||||
status: stringValue(request.status, "pending"),
|
||||
prompt: stringValue(request.prompt),
|
||||
});
|
||||
}
|
||||
break;
|
||||
}
|
||||
case "runtime_request.resolved":
|
||||
case "runtime_request.expired":
|
||||
case "runtime_request.cancelled": {
|
||||
const requestId = stringValue(payload.requestId);
|
||||
const request = snapshot.requests.find(
|
||||
(candidate) => candidate.requestId === requestId,
|
||||
);
|
||||
if (request !== undefined) {
|
||||
request.status = event.eventType.split(".").at(-1) ?? request.status;
|
||||
}
|
||||
break;
|
||||
}
|
||||
case "run.result.proposed":
|
||||
snapshot.proposedResult = structuredClone(
|
||||
event.payload as PrpStructuredRunResult,
|
||||
);
|
||||
break;
|
||||
case "run.terminal":
|
||||
snapshot.terminal = structuredClone(event.payload as PrpTerminalState);
|
||||
snapshot.runPhase = "terminal";
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
function addGap(
|
||||
snapshot: SessionSnapshot,
|
||||
event: PrpEvent,
|
||||
expected: number,
|
||||
): void {
|
||||
const key = sourceKey(event);
|
||||
if (
|
||||
snapshot.gaps.some(
|
||||
(gap) =>
|
||||
gap.sourceKey === key &&
|
||||
gap.expected === expected &&
|
||||
gap.received === event.sourceSeq,
|
||||
)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const missingCount = event.sourceSeq - expected;
|
||||
const recordedCount = Math.min(missingCount, MAX_RECORDED_MISSING_SEQUENCES);
|
||||
snapshot.gaps.push({
|
||||
sourceKey: key,
|
||||
expected,
|
||||
received: event.sourceSeq,
|
||||
missingCount,
|
||||
missing: Array.from(
|
||||
{ length: recordedCount },
|
||||
(_, index) => expected + index,
|
||||
),
|
||||
truncated: recordedCount < missingCount,
|
||||
});
|
||||
}
|
||||
|
||||
export function createSessionSnapshot(fixture: PrpFixture): SessionSnapshot {
|
||||
return createSessionSnapshotFromMetadata({
|
||||
fixtureName: fixture.name,
|
||||
identity: fixture.identity,
|
||||
capabilities: fixture.capabilities,
|
||||
});
|
||||
}
|
||||
|
||||
export function createSessionSnapshotFromMetadata(input: {
|
||||
fixtureName: string;
|
||||
identity: PrpIdentity;
|
||||
capabilities: PrpCapabilities;
|
||||
}): SessionSnapshot {
|
||||
return {
|
||||
schema: "paperclip.prp.session-snapshot.v1",
|
||||
fixtureName: input.fixtureName,
|
||||
identity: structuredClone(input.identity),
|
||||
capabilities: structuredClone(input.capabilities),
|
||||
runPhase: "queued",
|
||||
sessionState: "not_started",
|
||||
turnState: "not_started",
|
||||
activeTurnId: null,
|
||||
items: [],
|
||||
requests: [],
|
||||
proposedResult: null,
|
||||
terminal: null,
|
||||
timeline: [],
|
||||
sourceCursors: {},
|
||||
processedEventIds: [],
|
||||
duplicateEventIds: [],
|
||||
outOfOrderEventIds: [],
|
||||
gaps: [],
|
||||
integrity: "complete",
|
||||
};
|
||||
}
|
||||
|
||||
export function applyPrpEvent(
|
||||
current: SessionSnapshot,
|
||||
event: PrpEvent,
|
||||
): SessionSnapshot {
|
||||
if (current.processedEventIds.includes(event.sourceEventId)) {
|
||||
return current;
|
||||
}
|
||||
|
||||
const snapshot = structuredClone(current);
|
||||
const key = sourceKey(event);
|
||||
const cursor = snapshot.sourceCursors[key] ?? 0;
|
||||
const expected = cursor + 1;
|
||||
if (event.sourceSeq <= cursor) {
|
||||
if (!snapshot.outOfOrderEventIds.includes(event.sourceEventId)) {
|
||||
snapshot.outOfOrderEventIds.push(event.sourceEventId);
|
||||
}
|
||||
snapshot.integrity = "gap_detected";
|
||||
return snapshot;
|
||||
}
|
||||
if (event.sourceSeq > expected) {
|
||||
addGap(snapshot, event, expected);
|
||||
}
|
||||
|
||||
snapshot.sourceCursors[key] = event.sourceSeq;
|
||||
snapshot.processedEventIds.push(event.sourceEventId);
|
||||
snapshot.timeline.push({
|
||||
position: snapshot.timeline.length + 1,
|
||||
sourceEventId: event.sourceEventId,
|
||||
sourceSeq: event.sourceSeq,
|
||||
eventType: event.eventType,
|
||||
emittedAt: event.emittedAt,
|
||||
...(event.itemId === undefined ? {} : { itemId: event.itemId }),
|
||||
summary: timelineSummary(snapshot, event),
|
||||
});
|
||||
applyProjection(snapshot, event);
|
||||
snapshot.integrity =
|
||||
snapshot.gaps.length > 0 || snapshot.outOfOrderEventIds.length > 0
|
||||
? "gap_detected"
|
||||
: "complete";
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
export function reduceSessionEvents(
|
||||
current: SessionSnapshot,
|
||||
events: readonly PrpEvent[],
|
||||
): SessionSnapshot {
|
||||
const counts = new Map<string, number>();
|
||||
for (const event of events) {
|
||||
counts.set(event.sourceEventId, (counts.get(event.sourceEventId) ?? 0) + 1);
|
||||
}
|
||||
|
||||
let snapshot = structuredClone(current);
|
||||
for (const [eventId, count] of counts) {
|
||||
if (count > 1 && !snapshot.duplicateEventIds.includes(eventId)) {
|
||||
snapshot.duplicateEventIds.push(eventId);
|
||||
}
|
||||
}
|
||||
snapshot.duplicateEventIds.sort();
|
||||
for (const event of events) {
|
||||
snapshot = applyPrpEvent(snapshot, event);
|
||||
}
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
export function reducePrpFixture(fixture: PrpFixture): SessionSnapshot {
|
||||
return reduceSessionEvents(createSessionSnapshot(fixture), fixture.events);
|
||||
}
|
||||
|
||||
export interface ReplayParitySummary {
|
||||
runId: string;
|
||||
integrity: SessionSnapshot["integrity"];
|
||||
timelineCount: number;
|
||||
duplicateEventIds: string[];
|
||||
gaps: SequenceGap[];
|
||||
turnTerminalState: string | null;
|
||||
runTerminalState: string | null;
|
||||
}
|
||||
|
||||
export function replayParitySummary(
|
||||
snapshot: SessionSnapshot,
|
||||
): ReplayParitySummary {
|
||||
return {
|
||||
runId: snapshot.identity.runId,
|
||||
integrity: snapshot.integrity,
|
||||
timelineCount: snapshot.timeline.length,
|
||||
duplicateEventIds: [...snapshot.duplicateEventIds],
|
||||
gaps: structuredClone(snapshot.gaps),
|
||||
turnTerminalState: snapshot.terminal?.turnTerminalState ?? null,
|
||||
runTerminalState: snapshot.terminal?.runTerminalState ?? null,
|
||||
};
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { formatReplayResult, replayFixtureText } from "./replay.js";
|
||||
|
||||
describe("Replay tracer", () => {
|
||||
it("formats the validated reducer snapshot for CLI consumers", async () => {
|
||||
const source = await readFile(
|
||||
new URL(
|
||||
"../../protocol/fixtures/replay/happy-path.json",
|
||||
import.meta.url,
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
const result = replayFixtureText(source);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(JSON.parse(formatReplayResult(result))).toMatchObject({
|
||||
ok: true,
|
||||
snapshot: {
|
||||
integrity: "complete",
|
||||
terminal: { runTerminalState: "succeeded" },
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
import {
|
||||
parsePrpFixtureText,
|
||||
type ProtocolValidationIssue,
|
||||
} from "../protocol/replay-contract.js";
|
||||
import {
|
||||
reducePrpFixture,
|
||||
type SessionSnapshot,
|
||||
} from "../reducer/session-reducer.js";
|
||||
|
||||
export type ReplayResult =
|
||||
| { ok: true; snapshot: SessionSnapshot; issues: [] }
|
||||
| { ok: false; snapshot: null; issues: ProtocolValidationIssue[] };
|
||||
|
||||
export function replayFixtureText(text: string): ReplayResult {
|
||||
const validation = parsePrpFixtureText(text);
|
||||
if (!validation.ok) {
|
||||
return { ok: false, snapshot: null, issues: validation.issues };
|
||||
}
|
||||
return {
|
||||
ok: true,
|
||||
snapshot: reducePrpFixture(validation.fixture),
|
||||
issues: [],
|
||||
};
|
||||
}
|
||||
|
||||
export function formatReplayResult(result: ReplayResult): string {
|
||||
return JSON.stringify(result, null, 2);
|
||||
}
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"lib": ["ES2023"],
|
||||
"outDir": "dist",
|
||||
"rootDir": "src"
|
||||
},
|
||||
"include": ["src/**/*.ts"],
|
||||
"exclude": ["src/**/*.test.ts"]
|
||||
}
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
environment: "node",
|
||||
include: ["src/**/*.test.ts"],
|
||||
},
|
||||
});
|
||||
Loading…
Reference in New Issue