# Runner E2E fixture authoring The fixture catalog is executable production-contract data. Keep it small, typed, deterministic, and free of raw credentials. ## Suites and matrices A `RunnerSuiteFixture` declares one durable testing purpose: stable ID, label, description, profiles, environments, cases, expected size, and definition or ranking metadata. Its execution IDs are globally prefixed as `...`. Add a new suite when the testing purpose or desired cross-product differs; do not inflate an existing suite with unrelated dimensions. The suite definition fingerprint is historical comparison metadata. Any profile, model qualification, environment, task, or ranking-snapshot change must change that fingerprint automatically so the dashboard can annotate the boundary instead of silently joining unlike totals. ## Agent profiles Add `RunnerProfileFixture` entries in `catalog.ts`. A profile declares: - a stable ID and searchable groups; - legacy or native generation; - adapter/provider and required credential; - a model imported from its adapter constant or qualified runner profile; - supported environment IDs; - expected runtime metadata; and - an agent payload factory. Do not duplicate model IDs, qualification decisions, CLI versions, or runner artifact rules. Codex profiles import `DEFAULT_CODEX_LOCAL_MODEL`, OpenCode profiles import `QUALIFIED_OPENCODE_MODEL`, and ACPX profiles import `QUALIFIED_ACPX_PROFILES`. Add or qualify models at their owning production source first. OpenRouter breadth profiles are generated from `openrouter-models.json`, not written by hand. That reviewed snapshot must contain exactly five unique, available, tool-capable models with rank, canonical ID, display name, supported parameters, source URL, capture time, and verified content hash. Refresh it manually with `pnpm test:e2e:runner:models:update`; nightly campaigns never change fixture definitions. Agent `adapterConfig.env` values must be `{type:"secret_ref", secretId, version:"latest"}` objects supplied to the factory. A fixture source containing a raw secret-looking value is rejected by catalog validation. ## Environments An `EnvironmentFixture` declares driver/provider, credential requirements, attempt deadline, lifecycle behavior, expected execution target, and a payload factory validated by the shared environment schema. The local environment is instance-managed: company creation ensures it exists, and the public API intentionally rejects a second local environment. The setup registry therefore discovers that row through the public environments API. This still provides full isolation because every cell starts a new Paperclip instance and database. Daytona creates sandbox environments through the public API. The core fixture keeps `reuseLease:false` and `runnerLifecycleMode:"per_turn"`. The dedicated warm-continuity fixture uses `reuseLease:true` and `runnerLifecycleMode:"warm"`; its distinct `configurationKey` is part of the suite fingerprint even though both fixtures report `environmentId:"daytona"`. Keep short provider cleanup backstops, a Daytona secret reference, and an immutable image digest. Teardown must delete the environment with reusable-lease destruction and must fail the cell if cleanup cannot be confirmed. Keep CPU, memory, and disk explicit: lease metadata and the per-test public-list-price runtime estimate depend on that pinned billable resource shape. Changing it requires updating billing tests and reviewing the versioned Daytona rates in `billing.ts`. ## Usage and billing data Do not add fixture-authored token or dollar expectations. The live harness reads usage from selected public heartbeat-run records and records coverage per run. Provider-reported dollars remain distinct from runtime estimates. A zero or missing native usage payload is `unavailable` unless a real token-bearing receipt or provider cost proves otherwise. New execution environments must provide lease/resource metadata for a runtime estimate or explicitly remain `unavailable`; never infer that missing billing data means free execution. Future providers (SSH, E2B, Modal, Cloudflare, Kubernetes, Novita, exe.dev) should implement the same setup/probe/cleanup contract before being added to a matrix. Unsupported profile/environment combinations belong in `supportedEnvironments`, not in ad hoc test conditionals. ## Task cases and matchers A `RunnerTaskFixture` owns a work mode, a typed flow, expected run count, nonce-based title/prompt/marker factories, per-environment attempt deadlines, deterministic matchers, and expected terminal state. Single-turn prompts should make one bounded request with observable output and no nondeterministic judging. The `plan_revision_acceptance` flow must also provide revision-request and Plan marker factories. `question_resume_completion` must define the deterministic browser answer and prove exactly two successful runs with no pending interaction. `plan_approval_completion` must target the exact two-step canonical Plan revision, capture its pending UI, approve in the browser, and prove exactly two successful runs. `warm_three_turn` provides exactly two browser follow-up messages, preserves one project/execution-workspace scope, verifies host file contents after every turn, and finishes within three ten-minute turn deadlines. Every selected case runs in its own isolated Paperclip process, and independent cases may run concurrently. Follow-up turns inside one case retain their shared task state. Each case creates and tears down its own company, secrets, environment selection, agent, and browser-created task. The current plan case proves three runs on the same issue: publish a two-step Plan, request a three-step revision through the UI, and accept the exact new revision through the UI before verifying implementation and Done. The matcher union supports message exact/contains/regex/ordered checks, issue and run state, runtime/environment metadata, files, artifacts, JSON paths, and JSON Schema. The initial cases use normalized `message_contains` plus state, runtime, and environment assertions; the plan flow additionally verifies canonical document revision IDs, bodies, step counts, interaction targets, and visible previews. Add matcher behavior and credential-free tests together. Adding a task expands its suite's matrix. Update the suite's intentional size, the complete-catalog size, and credential-free unit tests in the same change. Paid tests never silently skip a missing credential or unsupported artifact. ## New Paperclip object fixtures Register new objects in `live-fixtures.ts` with explicit dependencies in `FixtureRegistry`. Setup must use a public API. Teardown runs in reverse order and is invoked after partial setup failures. Direct database writes and private test-only runner endpoints are prohibited. The expected dependency shape is: ```text company └── encrypted secrets └── environment └── agent └── browser-created task ``` Projects, goals, apps, and configuration fixtures can be inserted into that graph without changing the launcher. Keep returned fixture state to IDs and sanitized metadata; never retain raw secret values. ## Required checks Run before a fixture change is reviewed: ```bash pnpm test:e2e:runner:unit pnpm test:e2e:runner:typecheck pnpm test:e2e:runner -- --list ``` Then run the narrowest paid cell that exercises the fixture. A full matrix is a manual or scheduled campaign, not a PR requirement.