7.9 KiB

Raw Permalink Blame History

Capability Clean-Start Tutorial: Capability Contract, Mock Control Plane, and Scenario Explorer

Time to first success: about 5 minutes. Two commands take you from a clean checkout to 106 passing conformance cases and a browser you can click through. The whole tutorial runs from the repository root. It starts no Paperclip service, contacts no Paperclip control plane, clones no external eval repository, and holds no provider credential. Everything it needs is checked in under packages/paperclip-runner/.

Capability does not integrate the runner into Paperclip. It builds a package-local model of what a native Paperclip run would do — a deterministic mock control plane, a transport-neutral semantic tool catalog, an authorization engine, a 106-case conformance suite derived from the Paperclip Evals corpus, and a read-only browser explorer over all of it. Real integration is future upload integration and requires separate approval. See the future binding boundary reference.

What you need

Node.js 20 or newer and pnpm 9 or newer. This tutorial was verified with Node 22.22.2 and pnpm 9.15.4.
No Rust toolchain. Every command in this tutorial is TypeScript/Node only. (A full verify still builds Rust, but nothing here does.)
No network access after pnpm install.

Install the package workspace from the repository root:

pnpm install --filter @paperclipai/paperclip-runner --lockfile=false --offline --ignore-scripts --dev

1. Prove the 106-case conformance suite (about 1 minute)

pnpm --filter @paperclipai/paperclip-runner test:capability-evals

This runs the eval-derived conformance suite entirely in-process against the mock control plane. Expected final line:

 Test Files  1 passed (1)
      Tests  1 passed (1)

The single test file drives all 106 cases across the 16 eval groups. To see the per-group counts, assertion classes, the fake-agent operation matrix, and the bounded Codex binding sample, generate the parity report:

pnpm --filter @paperclipai/paperclip-runner report:capability-evals

Expected final line:

Capability eval conformance passed: 106 cases across 16 groups.

The report is written to .paperclip-local/evidence/capability/eval-parity-report.{json,md}. It is a generated-on-demand artifact, not a committed file; a clean checkout does not contain it. Delete it before you run pnpm --filter @paperclipai/paperclip-runner docs:validate (the report carries no OKF frontmatter and would otherwise fail the knowledge bundle check — see Known gaps).

The 16 groups and case counts are fixed by the capability contract:

Group	Cases	Group	Cases	Group	Cases	Group	Cases
hb	5	co	6	st	8	cm	6
se	4	su	4	bl	5	dp	3
ix	9	ap	6	ar	4	er	9
rf	22	mh	4	rs	3	wk	8

2. Open the scenario explorer (about 2 minutes)

pnpm --filter @paperclipai/paperclip-runner demo:scenarios

Open http://127.0.0.1:4183/scenario-explorer/. The explorer is read-only. It runs each scenario in the browser against the same mock control plane and renders the run artifact; it never re-judges parity and never leaves its own origin. Walk this path:

Home. Confirm 16 group facets whose counts sum to 106.
Pick a scenario. Filter to ap and open ap-mcp-gate-01. The authorization view shows a deny row and a "1 deny" tab chip; the typed denial card carries no task data.
Read a heartbeat. Open hb-context-01. The control-plane checkout entry is labelled "no agent tool exists for this" — it is control-plane-owned, not a semantic tool.
Inspect a manager scenario. Open rf-api-mgr-heartbeat-01. Optional tools are listed with the grant that unlocked them, alongside the control-plane "no tool" list and a traceability panel.
Confirm the credential boundary. Codex mode is a disabled option with a stated reason. localStorage holds no run artifact or grant, and the page makes no request beyond its own assets.

The explorer imports the frozen 0.1.2 SDK through the package-local @paperclip-runner-local/capability alias — deliberately not the published package name. See the browser explorer reference.

3. Read what each surface guarantees

The explorer renders four surfaces produced by the runtime. Each has a package-local reference page:

Capability disposition — how every skill/reference behavior and eval case is classified as control-plane-owned, always-agent-tool, or optional-agent-tool, and how the generated contract is produced and checked.
Mock ControlPlanePort — the deterministic in-memory adapter, its entity domains, and its boundary.
Semantic tool catalog — the transport-neutral tools an agent may call.
Authorization and exposure — grants, typed denials, and secret redaction.
Eval conformance — how the 106 cases are derived from the checked-in traceability derivative with no external repo.

4. Run the full focused verification set

Every command below is offline, deterministic, and package-local. The verification commands reference lists each one with its purpose and expected result.

# Capability contract: completeness, uniqueness, one-to-one MCP folds, no drift.
pnpm --filter @paperclipai/paperclip-runner check:capability-inventory
pnpm --filter @paperclipai/paperclip-runner test:capability-inventory

# Mock control plane adapter and shared ControlPlanePort conformance.
pnpm --filter @paperclipai/paperclip-runner exec vitest run \
  src/conformance/control-plane-port.test.ts \
  src/mock-core/capability-mock-control-plane-adapter.test.ts

# Semantic tool catalog and authorization/redaction engine.
pnpm --filter @paperclipai/paperclip-runner exec vitest run \
  src/tools/capability-semantic-tools.test.ts

# 106-case conformance and the parity/fake-agent/bounded-Codex report.
pnpm --filter @paperclipai/paperclip-runner test:capability-evals
pnpm --filter @paperclipai/paperclip-runner report:capability-evals

# Scenario runtime, explorer components, and route determinism (49 tests).
pnpm --filter @paperclipai/paperclip-runner test:scenarios

# Browser information architecture, accessibility, determinism, and boundary.
pnpm --filter @paperclipai/paperclip-runner test:browser:scenarios

# Deterministic 24-image screenshot acceptance set.
# Recorded evidence generation is deferred from this release.

# Documentation links.
pnpm --filter @paperclipai/paperclip-runner docs:validate

Known gaps and boundaries

Reports are local. report:capability-evals writes ignored output under .paperclip-local/evidence/; docs:validate does not scan it.
Codex mode is disabled. The explorer offers a bounded real-Codex binding sample only through report:capability-evals; the browser holds no credential and the in-page Codex mode is inert with a stated reason.
No Paperclip integration. Nothing here touches a real control plane, database, or provider. future upload integration (ACPX) binds the real adapter behind the same seam and requires separate CTO approval.

Where this sits

Plan: Capability plan, issue TASK-16897 (see its #document-plan document).
Child issues: 7A capability inventory (TASK-16898), 7B UX map (TASK-16899), 7C mock adapter (TASK-16900), 7D semantic tools and authorization (TASK-16901, security gate TASK-16902, remediation TASK-16909), 7E conformance suite (TASK-16903), 7F browser explorer (TASK-16904, UX gate TASK-16905), 7G docs and evidence (TASK-16906), 7H clean-room QA (TASK-16907), Capability checkpoint (TASK-16908).

7.9 KiB Raw Permalink Blame History