History

Dotta 5458940a6e feat(runner): add offline evaluation tooling (#12653 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner needs repeatable evaluation contracts. > - Evaluation code must stay separate from provider launch and production orchestration. > - Offline fixtures need stable compatibility, scoring, traceability, and report rules. > - Published Runner consumers need only the supported evaluation contract surface. > - This pull request adds offline evaluation tooling and a workspace-private matrix kernel. > - The benefit is deterministic evaluation without credentials or paid provider calls. ## Linked Issues or Issue Description Refs #11297 This pull request extracts the offline evaluation unit from the earlier aggregate Runner work. ## What Changed - Add a workspace-private, provider-neutral evaluation matrix kernel. - Add the public `@paperclipai/paperclip-runner/evals` compatibility and native execution contracts. - Add fail-closed runnerd artifact and protocol compatibility checks. - Add deterministic workflow catalogs, scoring, traceability, and report generation. - Add sanitized Codex, OpenCode, and ACPX fixtures. - Add package-boundary and clean-consumer checks. - Add the eval package manifest to the Docker dependency stage. - Add the generated protocol fixture digest without changing the lockfile. ## Verification GitHub Actions must run: - Runner TypeScript and Rust type checks. - Runner unit and protocol tests. - Evaluation kernel tests. - Workflow traceability checks. - Clean-consumer and package-boundary checks. - Repository test, type-check, build, policy, and security gates. No local test command was run. The repository owner requested GitHub-only verification. ## Risks This is a large greenfield review surface with 51 files. The code does not launch a live provider or load credentials. Package and protocol drift fail closed. The workspace lockfile remains under the existing CI-owned process. ## Model Used OpenAI Codex with the GPT-5 agent model. The work used high reasoning, repository inspection, tool use, and parallel code review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge		2026-09-01 05:19:47 -05:00
..
fixtures	feat(runner): add offline evaluation tooling (#12653 )	2026-09-01 05:19:47 -05:00
provider-schemas	feat(runner): project durable ACPX events (#12424 )	2026-08-31 03:33:38 -05:00
schemas	feat(runner): add SDK and developer tooling (#12608 )	2026-08-31 21:33:11 -05:00
README.md	…
manifest.json	feat(runner): add offline evaluation tooling (#12653 )	2026-09-01 05:19:47 -05:00

README.md

PRP v1 Contract

The JSON Schema files in schemas/ are the language-neutral source of truth for Paperclip Runner Protocol version 1. The fixtures in fixtures/ define accepted and rejected compatibility cases.

Compatibility

protocolVersion, fixtureVersion, and event.schemaVersion are required.
A consumer fails closed when a required version or schema discriminator is not supported.
A v1 envelope can contain an unknown optional property when its schema marks that object as extensible.
A consumer ignores an unknown optional property until a later contract gives it meaning.
A required field, enum value, or typed structured-input field is not optional.
Question and answer identifiers are stable across the provider boundary.

The unknown-optional-fields.json fixture must be accepted. The unsupported-required-version.json fixture must be rejected.

Scope

The first provider descriptor and adapter fixture cover Codex only. The schemas for provider-neutral events and semantic receipts do not enable those actions. Discovery and authorization are separate contracts.

The conformance manifest records every source file and its SHA-256 digest. Run pnpm generate:protocol-manifest from this package after a source change. CI runs the same generator with --check to reject drift. This check also compiles the JSON Schemas and validates every replay, question, and cross-language conformance fixture against its declared schema.

The files in fixtures/replay/golden/ are deterministic reducer oracles. Each accepted replay fixture has a complete session snapshot and a compact parity summary. pnpm generate:replay-goldens updates them after an intentional reducer change; package build and CI fail when they drift.

The files in fixtures/local-runner/scripts/ drive the package-local fake harness. They cover successful, failed, interrupted, interactive, duplicate terminal, process-cleanup, and oversized-frame behavior without starting a production adapter.