paperclip/packages/paperclip-runner/docs/adr/0001-runner-testing-eval-package-boundaries.md

ADR 0001: Runner, Testing, and Eval Package Boundaries

• Status: Accepted
• Date: 2026-08-11
• Decision owners: Paperclip App

Context

The runner package previously exported production contracts, deterministic mocks, conformance fixtures, scenario/eval helpers, and demo servers from one root. A workspace checkout hid two packaging defects: consumers could use undeclared deep surfaces, and the public semantic dispatcher loaded ajv even though ajv was only a development dependency.

External conformance consumers must consume App contracts and test helpers without making the App runtime depend on scenario corpora, provider experiments, or a separate eval package.

Decision

Ownership is:

Owner	Stable responsibility
Paperclip App	PRP schemas and fixtures, canonical semantic catalog and dispatcher, runnerd/client interfaces, ControlPlanePort, the production binding, deterministic mock, and mock/real parity fixtures
Eval consumers	Scenario corpus, provider configuration, experiment reports, and provider-backed orchestration

The production binding remains App code at server/src/services/native-runtime/paperclip-control-plane-port.ts. It implements the App-owned ControlPlanePort; the runner package never imports the server, database, UI, or CLI.

Public runner exports are:

Export	Stability and purpose
@paperclipai/paperclip-runner	Runtime contracts, runner clients/backends, PRP validation/replay, canonical catalog/dispatcher, and compatibility preflight
@paperclipai/paperclip-runner/evals	Versioned native-attempt/build metadata, compatibility negotiation, and explicit runnerd artifact resolution
@paperclipai/paperclip-runner/testing	Deterministic mocks, PRP port conformance, and provider-neutral semantic conformance kit
./browser, ./react, ./standalone, ./styles.css	Existing explicitly named UI/standalone consumers

Mock adapters and conformance constants are no longer package-root exports. Tests and external conformance consumers must use ./testing. Scenario content, provider-backed matrices, reports, and provider configuration are not public App package exports.

The testkit remains an App ./testing subpath rather than a separate package. It shares the runner protocol/catalog release cadence, and no independent consumer or version cadence currently justifies another package. Split it only after an independent release requirement exists; a directory preference is not sufficient.

Generic credential-free matrix orchestration lives in the separately versioned, workspace-private @paperclipai/paperclip-eval-kernel package. It contains no runner imports, provider configuration, scenario corpus, scorer, or report renderer. The runner may use it only as a development dependency; runtime, optional, and peer dependency sets remain free of eval packages. Paid provider campaigns remain external.

The dependency graph is acyclic:

Paperclip App production binding
            |
            v
@paperclipai/paperclip-runner (runtime contracts)
            ^
            |
Eval consumers ----> @paperclipai/paperclip-runner/evals
       |             @paperclipai/paperclip-runner/testing
       +-----------> @paperclipai/paperclip-eval-kernel

No arrow points from App runtime to an external eval repository.

Compatibility and versioning

Package semver describes distribution compatibility. Independently versioned contracts are published in PAPERCLIP_RUNNER_COMPATIBILITY:

Component	Current contract	Compatibility rule
Canonical catalog	1	Operation removals, renames, placement changes, or incompatible schemas require a new contract version
PRP	1	Highest overlapping required protocol version; no overlap fails closed
Runner client	1	Breaking runnerd/client interface changes require a new version
runnerd artifact	2	Binary metadata/package disagreement or digest mismatch fails before launch
Harness driver	1	Breaking descriptor/config/session/conformance behavior requires a new version
Native execution	1	Breaking App attempt-bundle semantics require a new schema version and converter
Evals integration	1	Breaking package/binary/catalog/driver join behavior requires a new version
Control-plane adapter	1	Breaking ControlPlanePort or production-binding expectations require a new version
Testkit	1	Breaking mock seed, vector, observation, or conformance behavior requires a new version
Eval corpus	1	Runner declares a supported inclusive corpus-version range; out-of-range bundles fail before execution

assertPaperclipRunnerCompatibility performs a fail-closed preflight. It emits paperclip_runner_incompatible with stable issue codes for component mismatch, unsupported corpus versions, unknown catalog operations, missing provider capability declarations, and provider-operation gaps. Provider-specific runtime errors must not stand in for this preflight.

Clean-consumer proof

Run:

pnpm --filter @paperclipai/paperclip-runner check:package-boundaries
pnpm --filter @paperclipai/paperclip-runner check:clean-consumers

The second command builds and packs the runner, installs its tarball into a clean consumer, imports only the root, ./evals, and ./testing exports, executes deterministic PRP, harness-driver, and semantic conformance, and verifies the separately staged runnerd artifact digest. The consumer uses no workspace protocol, source-relative import, or deep package path. This is the packaging gate; workspace tests alone are not proof.

Consequences

• Existing tests importing mock/conformance values from the package root must migrate to @paperclipai/paperclip-runner/testing.
• ajv is a runtime dependency because the public dispatcher imports it.
• The semantic conformance kit defines normalized comparison; real App service adapters and risk-weighted vectors may evolve behind its testkit version.
• Scenario corpus changes do not force an App runtime release unless their declared compatibility requirement changes.