## Thinking Path
> - Paperclip manages AI agents and their work.
> - The new runner gives agents dedicated tools for common tasks.
> - Some API operations and parameters have no dedicated tool.
> - Agents need a controlled way to find and use those operations.
> - This pull request adds API search and calls through the real server
routes.
> - Existing tools remain the preferred path. The new tools are disabled
by default.
> - Paired tests measure correctness, tool choice, cost and time.
## Linked Issues or Issue Description
**Subsystem affected**
Paperclip Runner contracts, production tool authority and the server API
catalog.
**Problem or motivation**
The runner cannot use much of the API described by the old Paperclip
skill. A generic HTTP client would also let agents bypass runner control
rules.
**Proposed solution**
Add `search_api` and `call_api`. Resolve calls from the mounted API
catalog. Use server-held, run-bound credentials. Preserve route checks
and runner lifecycle rules. Keep the tools disabled until an operator
enables selected companies.
**Alternatives considered**
A dedicated tool for every endpoint would add a large initial prompt. An
unrestricted HTTP tool would weaken authorization and replay controls.
**Roadmap alignment**
This extends the native runner tooling. The repository owner requested
this design and implementation. The roadmap and related open PRs were
checked. No duplicate API escape-hatch PR was found.
## What Changed
- Register two compact fallback tools in canonical contracts and
provider projections.
- Build deterministic API discovery from OpenAPI, mounted experimental
routes and the old skill reference.
- Execute bounded JSON, text, file and download requests through
authenticated HTTP routes.
- Recheck active runs, company access and work modes. Block runner
lifecycle, scheduling, credential and approval bypasses. Keep routine
annotation collaboration available.
- Retain mutation receipts. Report uncertain outcomes without blindly
repeating writes.
- Add a company rollout gate and a durable eval worker with complete
cost accounting checks.
- Record child-task creation in the activity log with the agent and run.
- Add contract, authorization, file, replay and real runnerd/PRP/HTTP
tests.
- Document rollout gates and paid coverage limits. The companion eval
repository retains immutable attempts and reports.
## Verification
- Final app commit `da58370524c3626a744eec20164397c5fb6ba9ef`: all 32
checks passed; the unrelated Storybook visual check was skipped.
Greptile 5/5; no unresolved review threads.
- Full Linux build and recursive typecheck passed. Repository tests were
run by project and serialized shard; all 143 serialized server suites
passed.
- Runner TypeScript: 1,599 passed, two skipped. Rust release: 451
passing test reports. Conformance and replay parity passed. The required
API check passed 837 tests, including runnerd → PRP → authority → real
HTTP.
- Bindings cannot enable API tools without the explicit deployment flag.
Unit and real-authority tests prove the default-off boundary.
- The standalone API check builds and stages its own binary. It passed
after existing staged and debug binaries were removed from the test
container.
- UI and CLI tests passed. Initial environment failures (missing jq,
Docker overlay file identity, and parallel linker memory pressure) and
focused passing reruns are retained. The macOS full runner suite has
platform-specific failures; Linux is the qualified full-check platform.
- Eval harness: 27 tests passed; existing CI discovery ran 86 tests with
two unrelated skips. Credential export rejection is tested against the
actual report command.
- Luna and OpenRouter Sonnet each passed 60 common-workflow runs: ten
workflows, three repetitions per arm, zero unnecessary API fallback.
- Sonnet passed 11 selected capability/contract cases after fixes.
Gemini passed three smoke cases. DeepSeek exceeded the 120-second limit
and remains unqualified.
- Luna's two cost flags received focused follow-up. The original flags
and a later n=1 latency flag remain visible. Sonnet had no cost or
latency increase above 20%.
- The catalog contains 785 entries; 58 were exercised across all stages.
Most operation probes remain unrun and some need additional fixtures.
Authored probes do not establish successful coverage.
- Total conservative accounted cost: $9.875960. Active paid-campaign
time: 88.16/90 minutes. No missing accounting. Later security and
harness fixes have provider-free verification; no paid validation is
claimed for those revisions.
- Inspect the [qualification
report](https://github.com/paperclipai/paperclip-evals/blob/codex/seach-call-api-tools/evals/runner-api-tools/reports/2026-09-07-production/READINESS.md)
and [verification
record](https://github.com/paperclipai/paperclip-evals/blob/codex/seach-call-api-tools/evals/runner-api-tools/reports/2026-09-07-production/verification.json).
## Risks
- This is a broad authenticated API surface. Keep the default-off gate
until an operator selects initial rollout companies.
- Paid coverage is incomplete. Small regression samples do not prove all
workflows are unchanged.
- A timeout or server failure can follow a committed mutation. The
result reports an unknown outcome and requires state inspection.
- The new definitions add prompt tokens. The report retains cost flags
and cache variation.
- No database migration is required.
- Repository rules require code-owner approval before merge. Technical
CI and automated review are complete.
## Model Used
OpenAI Codex based on GPT-6 assisted with code, tests and review. The
exact serving model ID and context window are not exposed in this
session. It used reasoning, tool calls and code execution.
Eval models: `gpt-5.6-luna` with low reasoning,
`openrouter/anthropic/claude-sonnet-5`,
`openrouter/google/gemini-3.8-flash`, and
`openrouter/deepseek/deepseek-v4-flash-0731`. Attempts retain runtime
versions, model identity, usage and source provenance.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Direct Runner evals retain evidence across model configurations.
> - Evalbook already has a grid and a read-only Runner Lab chat viewer.
> - Public projection stripped the view and selected a second plain
result page.
> - This change uses the existing viewer for public and private results.
> - The data access differs, but the presentation does not.
## Linked Issues or Issue Description
Refs #12931, #12945. Related open runtime-contract PR #11634 does not
contain this report-only change.
**What happened?**
The public direct-eval campaign opened plain result pages. The
access-controlled artifact used the chat viewer. Users could not follow
the same recorded interaction from the published grid.
**Expected behavior**
Every newly generated Runner Evalbook opens the existing chat viewer.
The grid and durable run history remain. Public evidence has explicit
redactions.
**Steps to reproduce**
Open campaign gha-34062394019-1 from the direct-eval history. Click a
result, then compare its plain page with the corresponding Actions
artifact.
**Paperclip version or commit**
Reproduced at 83987210d6.
**Deployment mode**
Static GitHub Actions artifacts and S3/CloudFront publication.
Companion site-theme renderer:
https://github.com/paperclipai/paperclip-evals/pull/19. This removes the
Python light theme and links the same built stylesheet.
## What Changed
- Add a closed public chat projection. Require mock isolation evidence
before publishing recorded text. Scrub private references and withhold
tool payloads, reasoning and provider state.
- Validate public HTML against the exact trusted viewer shell and asset
bytes. Validate the public DTO and local links. Keep CSP restrictions on
outbound requests and forms.
- Make the workflow render both data projections with the canonical
viewer. Pass a viewer-only artifact to the trusted publisher. Reject an
old renderer pin before paid execution.
- Fix report-only start position, missing-state inspector, read-only
controls and redaction labels. Tool evidence links select and highlight
the Evidence tab even when reopened. Runner execution and the full-stack
E2E workflow are unchanged.
- Add a no-model report refresh command. Preserve original campaign
identity, measurements and immutable history; label report revisions.
- Document the single presentation and public/private evidence boundary.
- Use one Runner Lab stylesheet and local fonts for the grid, Latest,
test design, inventory, server gate and S3 history index. Keep static
styles scoped away from live chat.
- Emit exact published report/history URLs to the Actions summary and
job outputs only after successful upload. Set the deployment link from
that output.
- Switch public Eval and Evidence panes without rendering both at once.
- Add all-run history with like-for-like pass-rate and cost timelines,
regression/recovery lists, exact commit links, source refs and Actions
links.
- Record all-attempt costs including retries. Keep provider list cost
separate from estimates. Label missing coverage and historical
final-only costs.
- Retain all run records beyond 200, backfill a separate derived
analytics projection, and exclude report refreshes from measurements.
## Verification
- Focused report/publishing/projection/workflow/adapter tests: 29
passed.
- Workflow Evalbook adapter tests: three passed.
- Viewer unit tests: eight passed; Vite viewer build passed.
- Companion renderer tests: 59 total, 57 passed, two inventory tests
skipped because the expected sibling checkout is absent.
- Re-rendered all 375 retained attempts from the completed campaign and
validated the public bundle. Zero new model calls; 356/358 selected
cells still pass.
- Browser walkthrough: grid to failed chat; prompt, named tool calls,
correct blocked status, visible assertions, no loading spinner or
composer. Public payload redactions are explicit.
- Full typecheck and build passed locally. test:run ended with 17 failed
files and 19 failed tests in unrelated server/worktree areas (3772 tests
passed). Latest-head CI is the final merge gate. The browser sources
also typecheck with a temporary TypeScript-7-compatible path
configuration; the checked-in browser config still uses removed baseUrl
options and is unchanged here.
- Real Chromium verification passed for passing, failing and
missing-recording attempts in both the full and public bundles. It
checks tool expansion, navigation, reload, read-only controls, narrow
viewport visibility and the public no-network boundary. Future
publications run it automatically.
- Fresh live proof: three gpt-5.4-mini native Codex cases passed on the
first attempt (get-task-context, create-child-task,
workflow-context-document-progress); estimated total $0.00632625.
Generated the full canonical report and verified all three file://
pages, all seven DevTools tabs, and evidence cross-links in Chromium.
Private screenshots remain local.
- Follow-up 99697c2c5: viewer build, eight unit tests, browser token
gate and browser-source typecheck (existing TS7 configuration
workaround) passed. Added repeated tool-to-evidence navigation to the
publication browser gate.
- Site-theme follow-up: 26 publishing/security tests, 59 Python tests
(57 passed, two existing skips), eight viewer unit tests and the viewer
build passed. Chromium verified shared colors and
grid/test-design/Latest navigation.
- Re-rendered all 375 retained attempts with the shared site theme,
without new provider calls.
- Successful hosted publication: [refreshed
Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34062394019-1-report-site-theme-v3/index.html)
and [themed
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html).
Real browser verified grid → test design → chat → All results and
history → run.
- Full maintained live suite completed on AWS: [Actions run
34074939112](https://github.com/paperclipai/paperclip/actions/runs/34074939112),
Paperclip 856813ba3a, evals
34e1846c06a39e641182dadce5de7ea739f657f1. All 358 cells across 11
configurations ran; 355 passed (99.2%), two behavior failures and one
infrastructure failure. Nine configurations are entirely green. All 360
retained attempts were rendered using the new design and published as an
immutable, zero-provider-call report refresh: [full
Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34074939112-1-report-history-v1/index.html).
The trusted master workflow ran the models; this PR's viewer rendered
their results afterward.
- Remaining live failures: native Codex mini did not emit the expected
discovery event for lazy-unauthorized-undisclosed (no mutation
occurred); GLM 5.3 scheduled a wake before approval in
workflow-governed-wait; GLM 5.3 timed out on create-task-document on
both attempts. No scoring rules were relaxed.
- Retry-inclusive recorded estimated model cost is at least $7.067692,
with usage for 357/360 attempts. Provider-reported list cost is a
separate alternative (at least $14.002806), not an additive cost.
Missing usage is unknown, not zero; AWS compute is excluded.
- [Production
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
now contains nine model runs and two separately labeled report
refreshes. Matching full-suite comparison reports three newly failing
cells and two recoveries versus the previous run. This is observed
run-to-run variation, not proof of a deterministic code regression. Live
HTTP checks verify the newest report link, shared theme, exact source
SHAs and cost analytics.
- The new full report passed Chromium checks for passing, failing and
missing-recording chat pages, tool-to-Evidence links, grid/design/Latest
navigation, reload, read-only controls and narrow layouts. The Mac was
locked during final hosted verification; the new hosted history was
checked by HTTP and generated-page browser checks, not a fresh
interactive desktop walkthrough.
- History follow-up: 31 publishing/security/metrics tests pass. Viewer
build passes. Chromium checks desktop and narrow history pages with no
document-level horizontal overflow.
### Visual verification
Generated from the scrubbed completed campaign; no private provider
identities or raw tool payloads are included. Full private pages were
also browser-tested, but their private metadata is not published as
screenshots.
Latest shared-theme proof:
## Risks
- Public chat text is newly visible, but only for the isolated mock
boundary. The producer excludes raw payloads and the publisher fails
closed on unknown fields, secrets, shell changes and asset
substitutions.
- Requires the companion canonical renderer revision and an updated
RUNNER_PROTOCOL_EVALS_SHA after merge. Old pins fail before paid
execution.
- Existing published campaigns remain immutable. A report refresh is a
separate history entry, not a new model qualification.
- Successfully published the immutable site-theme refresh with the
configured report-bucket SSO profile. Original run records and
qualification pointers are preserved.
## Model Used
OpenAI Codex, GPT-5-based coding agent with reasoning, shell and browser
tools. Exact deployment model ID and context-window size are not exposed
in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
cad99dbf04
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on cad99dbf04
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner package already provides the production protocol and
execution spine.
> - Contributors still need stable SDK surfaces, deterministic test
tools, and local inspection tools.
> - Those surfaces share generated contracts and must change as one
package boundary.
> - This pull request adds the package-local SDK, labs, examples, and
drift checks.
> - The benefit is a reviewable developer platform that does not change
application execution selection.
## Linked Issues or Issue Description
**Subsystem affected**
`packages/paperclip-runner` — runner SDK, conformance tools, and
developer tooling.
**Problem or motivation**
The production runner spine is present, but package consumers cannot
build deterministic integrations, inspect sessions, or verify
provider-neutral behavior through supported surfaces.
**Proposed solution**
Add browser, React, standalone, live-session, scenario, conformance, and
evaluation surfaces. Add generated contract inventories and
package-local verification scripts. Keep production application routing
unchanged.
**Alternatives considered**
We considered splitting each generated catalog, SDK surface, and demo
into separate pull requests. Those changes share exports, fixtures, and
drift gates. Splitting them would create intermediate package states
that do not build.
**Roadmap alignment**
No overlapping item appears in `ROADMAP.md`. This work extends the
runner package that is already on `master`.
## What Changed
- Add browser, React, standalone, live-session, and issue-thread SDK
surfaces.
- Add deterministic mock control-plane, scenario, conformance, replay,
and evaluation tools.
- Add bounded Codex, OpenCode, and ACPX development transports and
fixtures.
- Keep deferred managed-provider execution fail-closed. Persisted
compatibility data remains readable.
- Add generated capability inventories with their source files and drift
checks.
- Add examples, package documentation, browser checks, and
clean-consumer checks.
- Preserve the reviewed protocol bounds, replay compatibility aliases,
process environment isolation, and semantic redaction limits.
- Update the ACPX package patch that the existing workspace patch
registry already tracks.
- Do not change `pnpm-lock.yaml`, repository workflows, server runtime
selection, or the application UI.
## Verification
GitHub Actions is the verification authority for this pull request. The
repository CI, package TypeScript and Rust checks, package tests,
generated-output drift checks, browser checks, security scans, and
Greptile review must pass on the exact head.
Local test suites were not run because this series uses parallel GitHub
Actions for verification.
## Risks
This is a large greenfield package change. The main risks are public
export drift, generated-output drift, and optional React consumer
compatibility. Package boundary checks, clean-consumer checks, and
browser tests cover those risks. Production adapter selection and server
execution are outside this pull request.
## Stack
1. **This PR:** runner SDK and developer tooling.
2. [Codex production server
integration](https://github.com/paperclipai/paperclip/pull/12616).
3. [Provider-neutral task-thread
UI](https://github.com/paperclipai/paperclip/pull/12617).
## Model Used
OpenAI Codex, GPT-5, high-reasoning mode, with tool use and code
execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the feature request
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Dotta
