125 lines
6.0 KiB
Markdown
125 lines
6.0 KiB
Markdown
# ADR 0001: Runner, Testing, and Eval Package Boundaries
|
|
|
|
- Status: Accepted
|
|
- Date: 2026-08-11
|
|
- Decision owners: Paperclip App
|
|
|
|
## Context
|
|
|
|
The runner package previously exported production contracts, deterministic
|
|
mocks, conformance fixtures, scenario/eval helpers, and demo servers from one
|
|
root. A workspace checkout hid two packaging defects: consumers could use
|
|
undeclared deep surfaces, and the public semantic dispatcher loaded `ajv` even
|
|
though `ajv` was only a development dependency.
|
|
|
|
External conformance consumers must consume App contracts and test helpers
|
|
without making the App runtime depend on scenario corpora, provider
|
|
experiments, or a separate eval package.
|
|
|
|
## Decision
|
|
|
|
Ownership is:
|
|
|
|
| Owner | Stable responsibility |
|
|
|---|---|
|
|
| Paperclip App | PRP schemas and fixtures, canonical semantic catalog and dispatcher, runnerd/client interfaces, `ControlPlanePort`, the production binding, deterministic mock, and mock/real parity fixtures |
|
|
| Eval consumers | Scenario corpus, provider configuration, experiment reports, and provider-backed orchestration |
|
|
|
|
The production binding remains App code at
|
|
`server/src/services/native-runtime/paperclip-control-plane-port.ts`. It
|
|
implements the App-owned `ControlPlanePort`; the runner package never imports
|
|
the server, database, UI, or CLI.
|
|
|
|
Public runner exports are:
|
|
|
|
| Export | Stability and purpose |
|
|
|---|---|
|
|
| `@paperclipai/paperclip-runner` | Runtime contracts, runner clients/backends, PRP validation/replay, canonical catalog/dispatcher, and compatibility preflight |
|
|
| `@paperclipai/paperclip-runner/evals` | Versioned native-attempt/build metadata, compatibility negotiation, and explicit runnerd artifact resolution |
|
|
| `@paperclipai/paperclip-runner/testing` | Deterministic mocks, PRP port conformance, and provider-neutral semantic conformance kit |
|
|
| `./browser`, `./react`, `./standalone`, `./styles.css` | Existing explicitly named UI/standalone consumers |
|
|
|
|
Mock adapters and conformance constants are no longer package-root exports.
|
|
Tests and external conformance consumers must use `./testing`. Scenario
|
|
content, provider-backed matrices, reports, and provider configuration are not
|
|
public App package exports.
|
|
|
|
The testkit remains an App `./testing` subpath rather than a separate package.
|
|
It shares the runner protocol/catalog release cadence, and no independent
|
|
consumer or version cadence currently justifies another package. Split it only
|
|
after an independent release requirement exists; a directory preference is not
|
|
sufficient.
|
|
|
|
Generic credential-free matrix orchestration lives in the separately versioned,
|
|
workspace-private `@paperclipai/paperclip-eval-kernel` package. It contains no
|
|
runner imports, provider configuration, scenario corpus, scorer, or report
|
|
renderer. The runner may use it only as a development dependency; runtime,
|
|
optional, and peer dependency sets remain free of eval packages. Paid provider
|
|
campaigns remain external.
|
|
|
|
The dependency graph is acyclic:
|
|
|
|
```text
|
|
Paperclip App production binding
|
|
|
|
|
v
|
|
@paperclipai/paperclip-runner (runtime contracts)
|
|
^
|
|
|
|
|
Eval consumers ----> @paperclipai/paperclip-runner/evals
|
|
| @paperclipai/paperclip-runner/testing
|
|
+-----------> @paperclipai/paperclip-eval-kernel
|
|
```
|
|
|
|
No arrow points from App runtime to an external eval repository.
|
|
|
|
## Compatibility and versioning
|
|
|
|
Package semver describes distribution compatibility. Independently versioned
|
|
contracts are published in `PAPERCLIP_RUNNER_COMPATIBILITY`:
|
|
|
|
| Component | Current contract | Compatibility rule |
|
|
|---|---:|---|
|
|
| Canonical catalog | 1 | Operation removals, renames, placement changes, or incompatible schemas require a new contract version |
|
|
| PRP | 1 | Highest overlapping required protocol version; no overlap fails closed |
|
|
| Runner client | 1 | Breaking runnerd/client interface changes require a new version |
|
|
| runnerd artifact | 2 | Binary metadata/package disagreement or digest mismatch fails before launch |
|
|
| Harness driver | 1 | Breaking descriptor/config/session/conformance behavior requires a new version |
|
|
| Native execution | 1 | Breaking App attempt-bundle semantics require a new schema version and converter |
|
|
| Evals integration | 1 | Breaking package/binary/catalog/driver join behavior requires a new version |
|
|
| Control-plane adapter | 1 | Breaking `ControlPlanePort` or production-binding expectations require a new version |
|
|
| Testkit | 1 | Breaking mock seed, vector, observation, or conformance behavior requires a new version |
|
|
| Eval corpus | 1 | Runner declares a supported inclusive corpus-version range; out-of-range bundles fail before execution |
|
|
|
|
`assertPaperclipRunnerCompatibility` performs a fail-closed preflight. It emits
|
|
`paperclip_runner_incompatible` with stable issue codes for component mismatch,
|
|
unsupported corpus versions, unknown catalog operations, missing provider
|
|
capability declarations, and provider-operation gaps. Provider-specific runtime
|
|
errors must not stand in for this preflight.
|
|
|
|
## Clean-consumer proof
|
|
|
|
Run:
|
|
|
|
```sh
|
|
pnpm --filter @paperclipai/paperclip-runner check:package-boundaries
|
|
pnpm --filter @paperclipai/paperclip-runner check:clean-consumers
|
|
```
|
|
|
|
The second command builds and packs the runner, installs its tarball into a
|
|
clean consumer, imports only the root, `./evals`, and `./testing` exports, executes
|
|
deterministic PRP, harness-driver, and semantic conformance, and verifies the
|
|
separately staged runnerd artifact digest. The consumer uses no workspace
|
|
protocol, source-relative import, or deep package path. This is the packaging
|
|
gate; workspace tests alone are not proof.
|
|
|
|
## Consequences
|
|
|
|
- Existing tests importing mock/conformance values from the package root must
|
|
migrate to `@paperclipai/paperclip-runner/testing`.
|
|
- `ajv` is a runtime dependency because the public dispatcher imports it.
|
|
- The semantic conformance kit defines normalized comparison; real App service
|
|
adapters and risk-weighted vectors may evolve behind its testkit version.
|
|
- Scenario corpus changes do not force an App runtime release unless their
|
|
declared compatibility requirement changes.
|