## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner includes an issue-thread viewer for direct
Evalbook reports.
> - The runner package uses Vite 6.4.3.
> - Dependabot changed the React plugin from version 4.7.0 to version
6.1.1.
> - React plugin 6.1.1 requires Vite 7 or Vite 8 package internals.
> - The live evaluation workflow found this mismatch when it built the
viewer on a clean Linux worker.
> - This pull request restores the compatible plugin and adds the viewer
build to pull request CI.
> - The benefit is that CI detects this class of report-viewer build
failure before a paid evaluation campaign starts.
## Linked Issues or Issue Description
**What happened?**
The direct live evaluation workflow failed before model execution. The
`build:issue-thread` command could not load `@vitejs/plugin-react@6.1.1`
with Vite 6.4.3. The plugin imported the unavailable `vite/internal`
package path.
**Expected behavior**
The Evalbook issue-thread viewer must build from a clean frozen-lockfile
installation before the live evaluation matrix starts.
**Steps to reproduce**
1. Check out commit `165ca56a22adb60e5fda56045442d9c8498116a8`.
2. Run `pnpm install --frozen-lockfile --ignore-scripts`.
3. Run `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`.
4. Observe the `ERR_PACKAGE_PATH_NOT_EXPORTED` error for
`vite/internal`.
**Paperclip version or commit**
`165ca56a22adb60e5fda56045442d9c8498116a8`
**Deployment mode**
Built from source in GitHub Actions on Ubuntu.
## What Changed
- Restore `@vitejs/plugin-react` 4.7.0 in the Vite 6 runner package.
- Follow the repository policy: trusted PR CI regenerates and verifies
the lockfile artifact, and the master refresh workflow commits the
lock-only update after merge.
- Build the Runner Evalbook viewer in pull request CI.
## Verification
- `ci / policy`: regenerated the dependency lock artifact successfully
- `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`
- `actionlint .github/workflows/pr-trusted.yml
.github/workflows/runner-protocol-live-evals.yml`
- `node --test
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs`
- `git diff --check`
## Risks
Low risk. This change restores the previous React plugin major version
for one package. The selected version declares support for Vite 6. Pull
request CI now builds the affected viewer directly.
> This bug fix does not add or change a roadmap feature.
## Model Used
- OpenAI Codex with GPT-5. Tool use and code execution were enabled. The
Codex app managed the context window.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - The merged direct live eval workflow must read the private
`paperclip-evals` repository at an exact commit.
> - Its first hosted dispatch failed before provider execution because
the GitHub App token was minted from the `paperclip` repository
installation.
> - GitHub returned 404 while resolving the private eval commit, proving
that token did not have the required repository scope.
> - Minting each short-lived token from the exact private eval
repository installation supplies only the cross-repository read boundary
the workflow needs.
> - A workflow regression now verifies every eval-token block keeps that
exact scope.
## Linked Issues or Issue Description
The first default-branch run of Runner Direct Live Protocol Evals failed
in its immutable eval-commit verification step with HTTP 404. No
provider jobs ran and no provider spend occurred.
**What existing behavior does this improve?**
It allows the protected direct live eval workflow to verify and check
out the private `paperclipai/paperclip-evals` repository.
**Current behavior**
All four eval-token blocks set `GH_REPO` to `paperclipai/paperclip`,
selecting a token installation that cannot read the private eval
repository.
**Proposed behavior**
Set `GH_REPO` to the exact `paperclipai/paperclip-evals` repository in
authorization, catalog, matrix, and report jobs.
**Reason and benefit**
The app mints a short-lived token from the correct repository
installation while the main repository continues to use its ordinary
read-only workflow token.
**Breaking changes**
None.
## What Changed
- Scoped all four private-eval installation tokens to
`paperclipai/paperclip-evals`.
- Added a regression requiring that exact scope in every token block.
## Verification
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-protocol-eval-publish` — 15 passed.
- `node --test .github/scripts/tests/get-bot-token.test.mjs` — 3 passed.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` —
passed.
- `git diff --check` — passed.
## Risks
- The workflow reads a private repository. The token is still
short-lived, repository-specific, masked immediately, and used only by
the protected default-branch workflow.
- This changes no provider execution, Runner behavior, report content,
S3 publishing, or browser E2E behavior.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, repository inspection, code
editing, GitHub Actions diagnostics, and test execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation where applicable
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner executes agents through native and managed provider
drivers.
> - The direct live eval layer had drifted from the current Runner
contracts.
> - The old local workflow did not provide a complete parallel campaign
or durable report history.
> - The Runner also needed current native OpenCode and OpenRouter
qualification.
> - This pull request restores the direct campaign, corrects the runtime
gaps that the campaign found, and adds safe hosted Evalbook history.
> - The benefit is repeatable model comparison against an immutable
Runner and eval source revision.
## Linked Issues or Issue Description
Refs #11297
Refs #11634
**What existing behavior does this improve?**
This improves the direct live `paperclip-runner` eval workflow, provider
execution contract, and static Evalbook reporting path.
**Current behavior**
The direct evals do not have one maintained full campaign on current
`master`. OpenCode has no qualified multi-model OpenRouter roster.
Parallel provider bursts can compact committed events before the
transport observes them. Local reports do not have a separate safe S3
history index.
**Proposed behavior**
Run one immutable roster-plus-case matrix. Use the shared paid AWS
runner fleet. Keep raw artifacts access-controlled. Publish a sanitized
canonical Evalbook report under the separate `runner-protocol-evals` S3
prefix. Keep immutable campaign directories plus root history, latest,
and latest-green pointers.
**Reason and benefit**
Maintainers can compare native Codex, native OpenCode, ACPX, Claude
Managed, and AWS AgentCore behavior over time. They can inspect failures
without mixing this direct protocol layer with browser full-stack E2E.
**Breaking changes**
None. The new workflow and S3 prefix are additive. The existing Runner
full-stack E2E workflow and report remain separate.
## What Changed
- Added a trusted two-shard direct live workflow for up to 393
roster-plus-case cells.
- Reused the numeric actor allowlist, protected paid environment, and
RunsOn fleet controls from Runner full-stack E2E.
- Added immutable Runner and eval revision resolution, exact credential
boundaries, bounded retries, and cost ceilings.
- Added a public report projection that removes sessions, transcripts,
tool payloads, state, traces, raw failures, remote profile identities,
and credential-shaped values.
- Added additive S3 history under `runner-protocol-evals`, with
immutable campaigns and mutable root index pointers.
- Added native OpenCode model injection and current OpenRouter pricing
contracts.
- Fixed direct eval completion, workflow execution, semantic discovery,
warm-attach state reset, executable binding, and event-burst handling.
- Kept Runner browser full-stack E2E behavior and publication separate.
- Documented local and hosted direct eval operation.
## Verification
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-protocol-eval-publish` — 15 passed.
- `pnpm --filter @paperclipai/paperclip-runner build:typescript` —
passed.
- `actionlint .github/workflows/runner-protocol-live-evals.yml
.github/workflows/runner-full-stack-e2e.yml` — passed.
- Local current matrix at the revision in
[paperclip-evals#17](https://github.com/paperclipai/paperclip-evals/pull/17)
— 323 cells across 10 enabled configurations completed.
- Final local current matrix — 269 passed, 11 behavior failures, and 43
expected macOS-only ACPX platform failures.
- Targeted Runner checks — 13/13 eval-session tests, 15/15
publisher/security tests, and package typecheck passed; complete PR CI
is green, including all browser E2E shards.
## Risks
- Paid live campaigns can consume provider budget. Actor authorization,
exact per-cell ceilings, protected environments, and explicit schedule
enablement bound this risk.
- Public reports can leak provider data. The workflow publishes only a
separately projected report and validates every file before upload.
- The new workflow cannot publish until it is present on the default
branch. This pull request does not change the existing
`runner-full-stack-e2e` publication path.
- The campaign is large. It uses two GitHub matrices and caps combined
concurrency at the shared fleet limit.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, code editing, browser
inspection, repository tools, and live provider execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Dotta
Dotta