RomanNum3ral
/
paperclip
mirror of https://github.com/paperclipai/paperclip.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fix(evals): make the chat viewer the default published Evalbook (#12952) 

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Direct Runner evals retain evidence across model configurations.
> - Evalbook already has a grid and a read-only Runner Lab chat viewer.
> - Public projection stripped the view and selected a second plain
result page.
> - This change uses the existing viewer for public and private results.
> - The data access differs, but the presentation does not.

## Linked Issues or Issue Description

Refs #12931, #12945. Related open runtime-contract PR #11634 does not
contain this report-only change.

**What happened?**

The public direct-eval campaign opened plain result pages. The
access-controlled artifact used the chat viewer. Users could not follow
the same recorded interaction from the published grid.

**Expected behavior**

Every newly generated Runner Evalbook opens the existing chat viewer.
The grid and durable run history remain. Public evidence has explicit
redactions.

**Steps to reproduce**

Open campaign gha-34062394019-1 from the direct-eval history. Click a
result, then compare its plain page with the corresponding Actions
artifact.

**Paperclip version or commit**

Reproduced at 83987210d6.

**Deployment mode**

Static GitHub Actions artifacts and S3/CloudFront publication.

Companion site-theme renderer:
https://github.com/paperclipai/paperclip-evals/pull/19. This removes the
Python light theme and links the same built stylesheet.

## What Changed

- Add a closed public chat projection. Require mock isolation evidence
before publishing recorded text. Scrub private references and withhold
tool payloads, reasoning and provider state.
- Validate public HTML against the exact trusted viewer shell and asset
bytes. Validate the public DTO and local links. Keep CSP restrictions on
outbound requests and forms.
- Make the workflow render both data projections with the canonical
viewer. Pass a viewer-only artifact to the trusted publisher. Reject an
old renderer pin before paid execution.
- Fix report-only start position, missing-state inspector, read-only
controls and redaction labels. Tool evidence links select and highlight
the Evidence tab even when reopened. Runner execution and the full-stack
E2E workflow are unchanged.
- Add a no-model report refresh command. Preserve original campaign
identity, measurements and immutable history; label report revisions.
- Document the single presentation and public/private evidence boundary.
- Use one Runner Lab stylesheet and local fonts for the grid, Latest,
test design, inventory, server gate and S3 history index. Keep static
styles scoped away from live chat.
- Emit exact published report/history URLs to the Actions summary and
job outputs only after successful upload. Set the deployment link from
that output.
- Switch public Eval and Evidence panes without rendering both at once.

- Add all-run history with like-for-like pass-rate and cost timelines,
regression/recovery lists, exact commit links, source refs and Actions
links.
- Record all-attempt costs including retries. Keep provider list cost
separate from estimates. Label missing coverage and historical
final-only costs.
- Retain all run records beyond 200, backfill a separate derived
analytics projection, and exclude report refreshes from measurements.

## Verification

- Focused report/publishing/projection/workflow/adapter tests: 29
passed.
- Workflow Evalbook adapter tests: three passed.
- Viewer unit tests: eight passed; Vite viewer build passed.
- Companion renderer tests: 59 total, 57 passed, two inventory tests
skipped because the expected sibling checkout is absent.
- Re-rendered all 375 retained attempts from the completed campaign and
validated the public bundle. Zero new model calls; 356/358 selected
cells still pass.
- Browser walkthrough: grid to failed chat; prompt, named tool calls,
correct blocked status, visible assertions, no loading spinner or
composer. Public payload redactions are explicit.
- Full typecheck and build passed locally. test:run ended with 17 failed
files and 19 failed tests in unrelated server/worktree areas (3772 tests
passed). Latest-head CI is the final merge gate. The browser sources
also typecheck with a temporary TypeScript-7-compatible path
configuration; the checked-in browser config still uses removed baseUrl
options and is unchanged here.
- Real Chromium verification passed for passing, failing and
missing-recording attempts in both the full and public bundles. It
checks tool expansion, navigation, reload, read-only controls, narrow
viewport visibility and the public no-network boundary. Future
publications run it automatically.

- Fresh live proof: three gpt-5.4-mini native Codex cases passed on the
first attempt (get-task-context, create-child-task,
workflow-context-document-progress); estimated total $0.00632625.
Generated the full canonical report and verified all three file://
pages, all seven DevTools tabs, and evidence cross-links in Chromium.
Private screenshots remain local.
- Follow-up 99697c2c5: viewer build, eight unit tests, browser token
gate and browser-source typecheck (existing TS7 configuration
workaround) passed. Added repeated tool-to-evidence navigation to the
publication browser gate.

- Site-theme follow-up: 26 publishing/security tests, 59 Python tests
(57 passed, two existing skips), eight viewer unit tests and the viewer
build passed. Chromium verified shared colors and
grid/test-design/Latest navigation.
- Re-rendered all 375 retained attempts with the shared site theme,
without new provider calls.

- Successful hosted publication: [refreshed
Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34062394019-1-report-site-theme-v3/index.html)
and [themed
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html).
Real browser verified grid → test design → chat → All results and
history → run.

- Full maintained live suite completed on AWS: [Actions run
34074939112](https://github.com/paperclipai/paperclip/actions/runs/34074939112),
Paperclip 856813ba3a, evals
34e1846c06a39e641182dadce5de7ea739f657f1. All 358 cells across 11
configurations ran; 355 passed (99.2%), two behavior failures and one
infrastructure failure. Nine configurations are entirely green. All 360
retained attempts were rendered using the new design and published as an
immutable, zero-provider-call report refresh: [full
Evalbook](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/campaigns/gha-34074939112-1-report-history-v1/index.html).
The trusted master workflow ran the models; this PR's viewer rendered
their results afterward.
- Remaining live failures: native Codex mini did not emit the expected
discovery event for lazy-unauthorized-undisclosed (no mutation
occurred); GLM 5.3 scheduled a wake before approval in
workflow-governed-wait; GLM 5.3 timed out on create-task-document on
both attempts. No scoring rules were relaxed.
- Retry-inclusive recorded estimated model cost is at least $7.067692,
with usage for 357/360 attempts. Provider-reported list cost is a
separate alternative (at least $14.002806), not an additive cost.
Missing usage is unknown, not zero; AWS compute is excluded.
- [Production
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
now contains nine model runs and two separately labeled report
refreshes. Matching full-suite comparison reports three newly failing
cells and two recoveries versus the previous run. This is observed
run-to-run variation, not proof of a deterministic code regression. Live
HTTP checks verify the newest report link, shared theme, exact source
SHAs and cost analytics.
- The new full report passed Chromium checks for passing, failing and
missing-recording chat pages, tool-to-Evidence links, grid/design/Latest
navigation, reload, read-only controls and narrow layouts. The Mac was
locked during final hosted verification; the new hosted history was
checked by HTTP and generated-page browser checks, not a fresh
interactive desktop walkthrough.
- History follow-up: 31 publishing/security/metrics tests pass. Viewer
build passes. Chromium checks desktop and narrow history pages with no
document-level horizontal overflow.

### Visual verification

Generated from the scrubbed completed campaign; no private provider
identities or raw tool payloads are included. Full private pages were
also browser-tested, but their private metadata is not published as
screenshots.

![Passing
replay](https://raw.githubusercontent.com/paperclipai/paperclip/c660fec08bd0492e844b1198ddcf0e6f387dd597/packages/paperclip-runner/docs/images/evalbook-chat/passed.png)
![Failed
replay](https://raw.githubusercontent.com/paperclipai/paperclip/c660fec08bd0492e844b1198ddcf0e6f387dd597/packages/paperclip-runner/docs/images/evalbook-chat/failed.png)
![Missing
recording](https://raw.githubusercontent.com/paperclipai/paperclip/c660fec08bd0492e844b1198ddcf0e6f387dd597/packages/paperclip-runner/docs/images/evalbook-chat/missing-recording.png)

Latest shared-theme proof:

![Pass-rate and cost
history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/proofs/cost-history-sept6/history.png)

![Overview
grid](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/proofs/site-theme-sept6/index.png)
![Test
design](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/proofs/site-theme-sept6/test-design.png)
![Recorded
chat](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/proofs/site-theme-sept6/passed.png)

## Risks

- Public chat text is newly visible, but only for the isolated mock
boundary. The producer excludes raw payloads and the publisher fails
closed on unknown fields, secrets, shell changes and asset
substitutions.
- Requires the companion canonical renderer revision and an updated
RUNNER_PROTOCOL_EVALS_SHA after merge. Old pins fail before paid
execution.
- Existing published campaigns remain immutable. A report refresh is a
separate history entry, not a new model qualification.
- Successfully published the immutable site-theme refresh with the
configured report-bucket SSO profile. Original run records and
qualification pointers are preserved.

## Model Used

OpenAI Codex, GPT-5-based coding agent with reasoning, shell and browser
tools. Exact deployment model ID and context-window size are not exposed
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
cad99dbf04
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on cad99dbf04
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta 2026-09-07 07:47:17 -05:00 committed by GitHub
parent ae03465ad4
commit 54a99d8840
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
25 changed files with 2085 additions and 246 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
.github/workflows/runner-protocol-live-evals.yml vendored
View File

@ -213,6 +213,11 @@ jobs:
--max-parallel "$MAX_PARALLEL" \
--output runner-protocol-eval-catalog.json
- name: Require the chat-report renderer before paid execution
run: |
set -euo pipefail
python3 .paperclip-evals/evals/paperclip-runner/tools/eval_program.py report --help | grep -q -- --public-viewer
- name: Upload immutable campaign catalog
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
@ -259,6 +264,9 @@ jobs:
pnpm --filter @paperclipai/paperclip-runner build:typescript
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
pnpm --filter @paperclipai/paperclip-runner build:issue-thread
# Older target refs must fail before paid cells, not publish an empty viewer.
grep -q 'paperclip-eval-report' packages/paperclip-runner/dist-issue-thread/assets/*.js
grep -q 'evalbook-site' packages/paperclip-runner/dist-issue-thread/assets/*.css
- name: Package a portable provider runtime
run: |
@ -290,6 +298,14 @@ jobs:
compression-level: 0
if-no-files-found: error
- name: Upload canonical viewer for publisher byte verification
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-protocol-viewer-${{ github.run_id }}-${{ github.run_attempt }}
path: packages/paperclip-runner/dist-issue-thread/
retention-days: 30
if-no-files-found: error
eval_shard_0:
name: Direct eval ${{ matrix.rosterId }} / ${{ matrix.caseId }}
needs: [authorize, catalog, build_runner]
@ -595,14 +611,28 @@ jobs:
python3 .paperclip-evals/evals/paperclip-runner/tools/eval_program.py report \
--runs-root runner-protocol-merged/public-runs \
--output runner-protocol-merged/public-report \
--viewer-root runner-protocol-build/extracted/dist-issue-thread \
--public-viewer \
--inventory .paperclip-evals/evals/paperclip-runner/inventory.json \
--coverage-matrix .paperclip-evals/evals/paperclip-runner/coverage-matrix.json
cp runner-protocol-merged/campaign.json runner-protocol-merged/public-report/campaign.json
- name: Set up report browser verification
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
- name: Verify the actual chat viewer before publication
run: |
pnpm install --frozen-lockfile --ignore-scripts
pnpm --filter @paperclipai/paperclip-runner exec playwright install --with-deps chromium
node packages/paperclip-runner/scripts/verify-runner-evalbook-viewer.mjs --report-root runner-protocol-merged/public-report --screenshots runner-protocol-merged/viewer-proof
node packages/paperclip-runner/scripts/verify-runner-evalbook-viewer.mjs --report-root runner-protocol-merged/report
- name: Enforce the static public allowlist
id: public_report
run: |
node --input-type=module -e 'import { validatePublicProtocolEvalReport } from "./packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs"; await validatePublicProtocolEvalReport("runner-protocol-merged/public-report");'
node --input-type=module -e 'import { validatePublicProtocolEvalReport } from "./packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs"; await validatePublicProtocolEvalReport("runner-protocol-merged/public-report", { viewerRoot: "runner-protocol-build/extracted/dist-issue-thread" });'
echo "ready=true" >> "$GITHUB_OUTPUT"
- name: Add campaign result to the workflow summary
@ -649,6 +679,7 @@ jobs:
id-token: write
environment:
name: runner-e2e-history
url: ${{ steps.publish.outputs.report_url }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
@ -666,6 +697,12 @@ jobs:
name: runner-protocol-eval-public-${{ github.run_id }}-${{ github.run_attempt }}
path: runner-protocol-public-report
- name: Download the same-run canonical viewer for byte verification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: runner-protocol-viewer-${{ github.run_id }}-${{ github.run_attempt }}
path: runner-protocol-trusted-viewer
- name: Exchange GitHub OIDC identity for scoped AWS credentials
uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6
with:
@ -673,8 +710,10 @@ jobs:
aws-region: ${{ vars.RUNNER_PROTOCOL_EVAL_HISTORY_AWS_REGION || vars.RUNNER_E2E_HISTORY_AWS_REGION }}
- name: Publish versioned report and refresh the root index
id: publish
env:
PAPERCLIP_RUNNER_PROTOCOL_EVAL_PUBLIC_REPORT_DIR: ${{ github.workspace }}/runner-protocol-public-report
PAPERCLIP_RUNNER_PROTOCOL_EVAL_VIEWER_DIR: ${{ github.workspace }}/runner-protocol-trusted-viewer
RUNNER_PROTOCOL_EVAL_HISTORY_S3_BUCKET: ${{ vars.RUNNER_PROTOCOL_EVAL_HISTORY_S3_BUCKET || vars.RUNNER_E2E_HISTORY_S3_BUCKET }}
RUNNER_PROTOCOL_EVAL_HISTORY_PREFIX: ${{ vars.RUNNER_PROTOCOL_EVAL_HISTORY_PREFIX || 'runner-protocol-evals' }}
RUNNER_PROTOCOL_EVAL_HISTORY_PUBLIC_BASE_URL: ${{ vars.RUNNER_PROTOCOL_EVAL_HISTORY_PUBLIC_BASE_URL || vars.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL }}

24
packages/paperclip-runner/devtools/issue-thread/src/App.tsx
View File

@ -12,6 +12,7 @@ import type { CapabilityDevtoolsSnapshot } from "../../../src/devtools";
import { capabilityDenialCount } from "../../../src/issue-thread/types";
import { Composer } from "./Composer";
import { EvidencePanel } from "./EvidencePanel";
import type { CapabilityDevtoolsTab } from "./DevtoolsInspector";
import { Icon } from "./Icons";
import { IssueHeader } from "./IssueHeader";
import { applyFakeInteractionResponse } from "./fake-store";
@ -78,6 +79,7 @@ interface EmbeddedEvalCheck {
}
interface EmbeddedEvalReport {
publication?: { schema: string; notice: string };
attemptId: string;
caseId: string;
disposition: string;
@ -101,7 +103,7 @@ interface EmbeddedEvalReport {
runnerBuild: string;
startedAt: string;
finishedAt: string;
durationMs: number;
durationMs: number | null;
initialRevision: number;
finalRevision: number;
usage: {
@ -117,7 +119,7 @@ interface EmbeddedEvalReport {
} | null;
};
view: CapabilityIssueThreadSnapshot;
devtools: CapabilityDevtoolsSnapshot;
devtools: CapabilityDevtoolsSnapshot | null;
navigation: { suiteHref: string; previous: { label: string; href: string } | null; next: { label: string; href: string } | null };
}
@ -332,6 +334,7 @@ export function App() {
const [panelOpen, setPanelOpen] = useState(() => embeddedEval !== null ||
readStoredFlag(route.surface === "chat" ? CHAT_PANEL_OPEN_KEY : PANEL_OPEN_KEY, false));
const [panelWidth, setPanelWidth] = useState(() => readStoredNumber(PANEL_WIDTH_KEY, 384));
const [devtoolsTab, setDevtoolsTab] = useState<CapabilityDevtoolsTab>(embeddedEval !== null ? "eval" : "evidence");
const [segment, setSegment] = useState<"thread" | "evidence">(route.segment);
const [openSections, setOpenSections] = useState<CapabilityEvidenceSectionId[]>(["tools"]);
const [selectedTurnId, setSelectedTurnId] = useState<string | "all">("all");
@ -491,7 +494,7 @@ export function App() {
await document.fonts.ready;
}
if (cancelled) return;
if (scroller !== null) scroller.scrollTop = scroller.scrollHeight;
if (scroller !== null) scroller.scrollTop = embeddedEval === null ? scroller.scrollHeight : 0;
await new Promise<void>((resolve) => requestAnimationFrame(() => resolve()));
if (!cancelled) setSettled(true);
})();
@ -501,14 +504,14 @@ export function App() {
}, [snapshot]);
useEffect(() => {
if (!panelOpen || snapshot === null || route.mode !== "live") return;
if (embeddedEval !== null || !panelOpen || snapshot === null || route.mode !== "live") return;
if (historicSessionId !== null) return;
let cancelled = false;
void capabilityLiveClient.devtools(snapshot.sessionId)
.then((next) => { if (!cancelled) setDevtools(next); })
.catch((cause) => { if (!cancelled) setActionError(describe(cause)); });
return () => { cancelled = true; };
}, [historicSessionId, panelOpen, route.mode, snapshot?.renderedAt, snapshot?.sessionId]);
}, [embeddedEval, historicSessionId, panelOpen, route.mode, snapshot?.renderedAt, snapshot?.sessionId]);
useEffect(() => {
if (!chat || snapshot === null || historicSessionId !== null) return;
@ -571,6 +574,7 @@ export function App() {
const openEvidence = useCallback(
(section: CapabilityEvidenceSectionId, recordId: string) => {
setPanelOpen(true);
setDevtoolsTab("evidence");
setSegment("evidence");
setSelectedTurnId("all");
setOpenSections((current) => (current.includes(section) ? current : [...current, section]));
@ -1363,7 +1367,7 @@ export function App() {
>
<div className="pit-thread">
{embeddedEval !== null ? (
<div className="pit-eval-boundary" data-phase="execution"><strong>Eval execution</strong></div>
<div className="pit-eval-boundary" data-phase="execution"><strong>Eval execution</strong>{embeddedEval.publication ? <span>{embeddedEval.publication.notice}</span> : null}</div>
) : null}
{snapshot.turns.length === 0 ? (
<section className="pit-empty-thread" data-testid="clean-room-empty">
@ -1404,7 +1408,7 @@ export function App() {
{embeddedEval !== null ? (
<div className="pit-eval-boundary" data-phase="post-run">
<strong>Post-run state</strong>
<span>Final mock control-plane revision {embeddedEval.run.finalRevision}</span>
<span>{embeddedEval.publication ? "Company-state details withheld from public replay" : `Final mock control-plane revision ${embeddedEval.run.finalRevision}`}</span>
<EvalAssertions assertions={embeddedEval.checks.filter((check) => check.anchor.kind === "run")} />
</div>
) : null}
@ -1425,7 +1429,7 @@ export function App() {
) : null}
</div>
<Composer
{embeddedEval === null ? <Composer
model={snapshot.composer}
sessionId={snapshot.sessionId}
onSend={send}
@ -1439,7 +1443,7 @@ export function App() {
.getElementById(`interaction-${interactionId}`)
?.scrollIntoView({ block: "center" });
}}
/>
/> : null}
</main>
{showPanel && layout === "side" ? (
@ -1496,6 +1500,8 @@ export function App() {
{showPanel ? (
<EvidencePanel
snapshot={snapshot}
devtoolsTab={devtoolsTab}
onDevtoolsTabChange={setDevtoolsTab}
evalReport={embeddedEval}
{...(embeddedEval !== null || (route.mode === "live" && historicSessionId === null) ? { devtools } : {})}
onForkRevision={(revision) => {

269
packages/paperclip-runner/devtools/issue-thread/src/DevtoolsInspector.tsx
View File

@ -23,6 +23,7 @@ export interface EvalInspectorReport {
disposition: string;
passed: boolean;
checks: EvalAssertion[];
publication?: { schema: string; notice: string };
run: {
model: string;
provider: string;
@ -41,7 +42,7 @@ export interface EvalInspectorReport {
runnerBuild: string;
startedAt: string;
finishedAt: string;
durationMs: number;
durationMs: number | null;
initialRevision: number;
finalRevision: number;
usage: {
@ -53,7 +54,7 @@ export interface EvalInspectorReport {
reasoningTokens: number;
providerReportedCostNanodollars?: number;
estimatedCostNanodollars: number;
pricingVersion: string;
pricingVersion?: string;
} | null;
};
}
@ -197,6 +198,149 @@ function documentsOf(state: Json): Array<{
});
}
export function EvalReportInspector({
evalReport,
}: {
evalReport: EvalInspectorReport;
}) {
const isPublic = Boolean(evalReport.publication);
return (
<div className="pit-devtools-pane pit-eval-inspector">
<div className="pit-eval-summary-head">
<a href="../../index.html">← Eval suite</a>
<strong>
{evalReport.passed
? "PASS"
: evalReport.disposition.replaceAll("_", " ").toUpperCase()}
</strong>
<code>{evalReport.attemptId}</code>
</div>
<dl className="pit-eval-run-facts">
<div>
<dt>Model</dt>
<dd>
{evalReport.run.model.startsWith(`${evalReport.run.provider}/`)
? evalReport.run.model
: `${evalReport.run.provider}/${evalReport.run.model}`}
</dd>
</div>
<div>
<dt>Configuration</dt>
<dd>{evalReport.run.configuration}</dd>
</div>
<div>
<dt>Session</dt>
<dd>
{isPublic
? "Withheld from public replay"
: evalReport.run.sessionId}
</dd>
</div>
<div>
<dt>Provider session</dt>
<dd>
{isPublic
? "Withheld from public replay"
: (evalReport.run.providerSessionId ?? "unavailable")}
</dd>
</div>
<div>
<dt>Driver</dt>
<dd>
{evalReport.run.driver}
{evalReport.run.providerVersion
? ` · ${evalReport.run.providerVersion}`
: ""}
</dd>
</div>
{evalReport.run.agentVersion ? (
<div>
<dt>Agent version</dt>
<dd>{evalReport.run.agentVersion}</dd>
</div>
) : null}
<div>
<dt>Retained session</dt>
<dd>
{isPublic
? "Withheld from public replay"
: evalReport.run.retainedSession === true
? (evalReport.run.retainedSessionStatus ?? "retained")
: "not applicable"}
</dd>
</div>
<div>
<dt>Duration</dt>
<dd>
{evalReport.run.durationMs == null
? "unavailable"
: `${evalReport.run.durationMs} ms`}
</dd>
</div>
<div>
<dt>Fixture</dt>
<dd>{evalReport.run.fixtureDigest}</dd>
</div>
<div>
<dt>State</dt>
<dd>
{isPublic
? "Withheld from public replay"
: `r${evalReport.run.initialRevision} → r${evalReport.run.finalRevision}`}
</dd>
</div>
<div>
<dt>Tokens</dt>
<dd>
{evalReport.run.usage === null
? "unknown"
: `${evalReport.run.usage.inputTokens} in · ${evalReport.run.usage.outputTokens} out · ${evalReport.run.usage.cachedInputTokens} cached`}
</dd>
</div>
<div>
<dt>Agent turns</dt>
<dd>{evalReport.run.usage?.agentTurns ?? "unknown"}</dd>
</div>
<div>
<dt>Provider requests</dt>
<dd>{evalReport.run.usage?.providerRequests ?? "unavailable"}</dd>
</div>
<div>
<dt>Estimated cost</dt>
<dd>
{evalReport.run.usage === null
? "unknown"
: `$${(evalReport.run.usage.estimatedCostNanodollars / 1_000_000_000).toFixed(6)}${evalReport.run.usage.pricingVersion ? ` · ${evalReport.run.usage.pricingVersion}` : ""}`}
</dd>
</div>
<div>
<dt>Provider list cost</dt>
<dd>
{typeof evalReport.run.usage?.providerReportedCostNanodollars !==
"number"
? "unknown"
: `$${(evalReport.run.usage.providerReportedCostNanodollars / 1_000_000_000).toFixed(6)}`}
</dd>
</div>
<div>
<dt>Runner</dt>
<dd>{evalReport.run.runnerPackageDigest}</dd>
</div>
<div>
<dt>Runner build</dt>
<dd>{evalReport.run.runnerBuild}</dd>
</div>
<div>
<dt>runnerd</dt>
<dd>{evalReport.run.runnerdDigest}</dd>
</div>
</dl>
<h3>Assertions</h3>
<EvalAssertions assertions={evalReport.checks} />
</div>
);
}
export function DevtoolsInspector({
snapshot,
onFork,
@ -291,6 +435,8 @@ export function DevtoolsInspector({
className="pit-button"
type="button"
onClick={() => onFork(revision)}
disabled={evalReport != null}
title={evalReport ? "Eval reports are read-only" : undefined}
>
<Icon name="branch" /> Fork r{revision}
</button>
@ -328,124 +474,7 @@ export function DevtoolsInspector({
))}
</div>
{tab === "eval" && evalReport ? (
<div className="pit-devtools-pane pit-eval-inspector">
<div className="pit-eval-summary-head">
<a href="../../index.html">← Eval suite</a>
<strong>
{evalReport.passed
? "PASS"
: evalReport.disposition.replaceAll("_", " ").toUpperCase()}
</strong>
<code>{evalReport.attemptId}</code>
</div>
<dl className="pit-eval-run-facts">
<div>
<dt>Model</dt>
<dd>
{evalReport.run.model.startsWith(`${evalReport.run.provider}/`)
? evalReport.run.model
: `${evalReport.run.provider}/${evalReport.run.model}`}
</dd>
</div>
<div>
<dt>Configuration</dt>
<dd>{evalReport.run.configuration}</dd>
</div>
<div>
<dt>Session</dt>
<dd>{evalReport.run.sessionId}</dd>
</div>
<div>
<dt>Provider session</dt>
<dd>{evalReport.run.providerSessionId ?? "unavailable"}</dd>
</div>
<div>
<dt>Driver</dt>
<dd>
{evalReport.run.driver}
{evalReport.run.providerVersion
? ` · ${evalReport.run.providerVersion}`
: ""}
</dd>
</div>
{evalReport.run.agentVersion ? (
<div>
<dt>Agent version</dt>
<dd>{evalReport.run.agentVersion}</dd>
</div>
) : null}
<div>
<dt>Retained session</dt>
<dd>
{evalReport.run.retainedSession === true
? (evalReport.run.retainedSessionStatus ?? "retained")
: "not applicable"}
</dd>
</div>
<div>
<dt>Duration</dt>
<dd>{evalReport.run.durationMs} ms</dd>
</div>
<div>
<dt>Fixture</dt>
<dd>{evalReport.run.fixtureDigest}</dd>
</div>
<div>
<dt>State</dt>
<dd>
r{evalReport.run.initialRevision} → r
{evalReport.run.finalRevision}
</dd>
</div>
<div>
<dt>Tokens</dt>
<dd>
{evalReport.run.usage === null
? "unknown"
: `${evalReport.run.usage.inputTokens} in · ${evalReport.run.usage.outputTokens} out · ${evalReport.run.usage.cachedInputTokens} cached`}
</dd>
</div>
<div>
<dt>Agent turns</dt>
<dd>{evalReport.run.usage?.agentTurns ?? "unknown"}</dd>
</div>
<div>
<dt>Provider requests</dt>
<dd>{evalReport.run.usage?.providerRequests ?? "unavailable"}</dd>
</div>
<div>
<dt>Estimated cost</dt>
<dd>
{evalReport.run.usage === null
? "unknown"
: `$${(evalReport.run.usage.estimatedCostNanodollars / 1_000_000_000).toFixed(6)} · ${evalReport.run.usage.pricingVersion}`}
</dd>
</div>
<div>
<dt>Provider list cost</dt>
<dd>
{typeof evalReport.run.usage
?.providerReportedCostNanodollars !== "number"
? "unknown"
: `$${(evalReport.run.usage.providerReportedCostNanodollars / 1_000_000_000).toFixed(6)}`}
</dd>
</div>
<div>
<dt>Runner</dt>
<dd>{evalReport.run.runnerPackageDigest}</dd>
</div>
<div>
<dt>Runner build</dt>
<dd>{evalReport.run.runnerBuild}</dd>
</div>
<div>
<dt>runnerd</dt>
<dd>{evalReport.run.runnerdDigest}</dd>
</div>
</dl>
<h3>Assertions</h3>
<EvalAssertions assertions={evalReport.checks} />
</div>
<EvalReportInspector evalReport={evalReport} />
) : null}
{tab === "timeline" ? (
<div className="pit-devtools-list">

23
packages/paperclip-runner/devtools/issue-thread/src/EvidencePanel.tsx
View File

@ -10,7 +10,7 @@ import type {
CapabilityToolDisposition,
} from "../../../src/issue-thread/types";
import type { CapabilityDevtoolsSnapshot } from "../../../src/devtools";
import { DevtoolsInspector, type CapabilityDevtoolsTab, type EvalInspectorReport } from "./DevtoolsInspector";
import { DevtoolsInspector, EvalReportInspector, type CapabilityDevtoolsTab, type EvalInspectorReport } from "./DevtoolsInspector";
import { Icon } from "./Icons";
import { capabilitySemanticToolDescriptor } from "../../../src/semantic-tools/catalog";
import {
@ -114,6 +114,8 @@ export interface EvidencePanelProps {
snapshot: CapabilityIssueThreadSnapshot;
devtools?: CapabilityDevtoolsSnapshot | null;
evalReport?: EvalInspectorReport | null;
devtoolsTab: CapabilityDevtoolsTab;
onDevtoolsTabChange: (tab: CapabilityDevtoolsTab) => void;
onForkRevision?: (revision: number) => void;
layout: "side" | "overlay" | "segment";
width: number;
@ -331,11 +333,12 @@ function Section({
}
export function EvidencePanel(props: EvidencePanelProps) {
const [devtoolsTab, setDevtoolsTab] = useState<CapabilityDevtoolsTab>(props.evalReport ? "eval" : "evidence");
const {
snapshot,
devtools,
evalReport,
devtoolsTab,
onDevtoolsTabChange,
onForkRevision = () => undefined,
layout,
width,
@ -392,9 +395,21 @@ export function EvidencePanel(props: EvidencePanelProps) {
{devtools !== undefined ? (
<>
{devtools === null ? (
<p className="pit-muted pit-devtools-loading">Loading company state…</p>
evalReport ? <section className="pit-devtools">
<div className="pit-devtools-tabs" role="tablist" aria-label="Developer tools">
{(["eval", "evidence"] as const).map((tab) => <button
key={tab}
type="button"
role="tab"
aria-selected={devtoolsTab === tab}
className="pit-tab"
onClick={() => onDevtoolsTabChange(tab)}
><span className="pit-tab-glyph"><Icon name="evidence" /></span><span>{tab === "eval" ? "Eval" : "Evidence"}</span></button>)}
</div>
{devtoolsTab === "eval" ? <EvalReportInspector evalReport={evalReport} /> : null}
</section> : <p className="pit-muted pit-devtools-loading">Loading company state…</p>
) : (
<DevtoolsInspector snapshot={devtools} onFork={onForkRevision} tab={devtoolsTab} onTabChange={setDevtoolsTab} evalReport={evalReport} />
<DevtoolsInspector snapshot={devtools} onFork={onForkRevision} tab={devtoolsTab} onTabChange={onDevtoolsTabChange} evalReport={evalReport} />
)}
</>
) : null}

87
packages/paperclip-runner/devtools/issue-thread/src/evalbook-site.css Normal file
View File

@ -0,0 +1,87 @@
/* Static Evalbook pages share the Runner Lab token/font layer and built asset.
Keep every selector scoped so report navigation cannot restyle live chat. */
.evalbook-site { color-scheme: dark; }
.evalbook-site main { width: 100%; max-width: 112rem; margin: auto; padding: var(--pit-space-6); }
.evalbook-site a { color: var(--pit-live); text-underline-offset: var(--pit-space-1); }
.evalbook-site a:hover { color: var(--pit-foreground); }
.evalbook-site :is(a, summary, [tabindex]):focus-visible { outline: 2px solid var(--pit-ring); outline-offset: 3px; }
.evalbook-site .top { display: flex; justify-content: space-between; align-items: center; gap: var(--pit-space-4); padding-bottom: var(--pit-space-4); border-bottom: 1px solid var(--pit-border); }
.evalbook-site .top > a { color: var(--pit-foreground); font-weight: 700; text-decoration: none; }
.evalbook-site h1 { margin: var(--pit-space-6) 0 var(--pit-space-3); font-size: var(--pit-text-xl); line-height: var(--pit-leading-tight); overflow-wrap: anywhere; }
.evalbook-site h2 { margin: var(--pit-space-6) 0 var(--pit-space-3); font-size: var(--pit-text-base); line-height: var(--pit-leading-tight); }
.evalbook-site p { margin: var(--pit-space-3) 0; }
.evalbook-site .badge { padding: var(--pit-space-1) var(--pit-space-2); color: var(--pit-live); background: var(--pit-live-surface); border: 1px solid var(--pit-live); border-radius: var(--pit-radius-sm); font-size: var(--pit-text-xs); white-space: nowrap; }
.evalbook-site :is(.tabs, .pointers, .breadcrumbs) { display: flex; align-items: center; gap: var(--pit-space-4); margin: var(--pit-space-4) 0; overflow-x: auto; }
.evalbook-site .tabs { border-bottom: 1px solid var(--pit-border); gap: 0; }
.evalbook-site .tabs a { flex: 0 0 auto; padding: var(--pit-space-3) var(--pit-space-4); color: var(--pit-muted-foreground); text-decoration: none; border-bottom: 2px solid transparent; }
.evalbook-site .tabs a:hover { background: var(--pit-surface); color: var(--pit-foreground); }
.evalbook-site .tabs a.active { background: var(--pit-accent-surface); border-color: var(--pit-accent); color: var(--pit-foreground); }
.evalbook-site .breadcrumbs span { color: var(--pit-muted-foreground); }
.evalbook-site .spend-summary { display: flex; flex-wrap: wrap; gap: var(--pit-space-6); margin: var(--pit-space-5) 0; }
.evalbook-site .spend-summary > div { display: flex; flex-direction: column; gap: var(--pit-space-1); }
.evalbook-site .spend-summary span { color: var(--pit-muted-foreground); font-size: var(--pit-text-xs); }
.evalbook-site .spend-summary strong { font-size: var(--pit-text-base); font-variant-numeric: tabular-nums; }
.evalbook-site .lede { color: var(--pit-muted-foreground); margin-bottom: var(--pit-space-5); }
.evalbook-site :is(.card, .message, .tool, .check) { margin: var(--pit-space-3) 0; padding: var(--pit-space-4); background: var(--pit-surface); border: 1px solid var(--pit-border); border-radius: var(--pit-radius-sm); overflow-wrap: anywhere; }
.evalbook-site .pass { border-left: 3px solid var(--pit-success); }
.evalbook-site .fail { border-left: 3px solid var(--pit-danger); }
.evalbook-site .infra { border-left: 3px solid var(--pit-warning); }
.evalbook-site .grid { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.3fr); gap: var(--pit-space-6); }
.evalbook-site .grid > section { min-width: 0; }
.evalbook-site :is(.table-shell, .table) { width: 100%; overflow: auto; border: 1px solid var(--pit-border); border-radius: var(--pit-radius-sm); background: var(--pit-surface-sunken); overscroll-behavior-inline: contain; scrollbar-color: var(--pit-border-strong) transparent; }
.evalbook-site table { width: 100%; border-collapse: separate; border-spacing: 0; font-size: var(--pit-text-xs); }
.evalbook-site :is(th, td) { padding: var(--pit-space-3); border-bottom: 1px solid var(--pit-border); text-align: left; vertical-align: top; }
.evalbook-site thead th { background: var(--pit-surface); color: var(--pit-muted-foreground); font-weight: 600; }
.evalbook-site tbody tr:last-child > :is(th, td) { border-bottom: 0; }
.evalbook-site tbody tr:hover > :is(th, td) { background: var(--pit-surface-raised); }
.evalbook-site .matrix { min-width: max(100%, var(--matrix-min-width, 1500px)); }
.evalbook-site .matrix :is(th, td):not(:first-child) { width: 10rem; text-align: center; }
.evalbook-site .matrix thead th { position: sticky; top: 0; z-index: 2; }
.evalbook-site .matrix thead th > div { color: var(--pit-foreground); font-size: var(--pit-text-sm); overflow-wrap: anywhere; }
.evalbook-site .matrix thead small { display: block; max-width: 11rem; margin: var(--pit-space-1) auto 0; font-size: var(--pit-text-xs); font-weight: 400; overflow-wrap: anywhere; }
.evalbook-site .matrix th:first-child { position: sticky; left: 0; z-index: 1; width: 18rem; min-width: 14rem; background: var(--pit-surface-sunken); }
.evalbook-site .matrix thead th:first-child { z-index: 3; background: var(--pit-surface); }
.evalbook-site .matrix tbody th:first-child a { display: block; color: var(--pit-foreground); font-weight: 500; overflow-wrap: anywhere; text-decoration: none; }
.evalbook-site .matrix tbody th:first-child a:hover { color: var(--pit-live); text-decoration: underline; }
.evalbook-site .result { display: flex; align-items: center; justify-content: center; width: 100%; max-width: 10rem; min-height: 2rem; margin-inline: auto; padding: var(--pit-space-1) var(--pit-space-2); border: 1px solid currentColor; border-radius: var(--pit-radius-sm); font-size: var(--pit-text-xs); font-weight: 700; text-decoration: none; }
.evalbook-site :is(.result-pass, .passed) { background: var(--pit-success-surface); color: var(--pit-success); }
.evalbook-site :is(.result-fail, .failed) { background: var(--pit-danger-surface); color: var(--pit-danger); }
.evalbook-site .result-infra { background: var(--pit-warning-surface); color: var(--pit-warning); }
.evalbook-site :is(.muted, .empty, small, footer) { color: var(--pit-muted-foreground); }
.evalbook-site .empty { text-align: center; }
.evalbook-site code { font-family: var(--pit-font-mono); font-size: var(--pit-text-xs); overflow-wrap: anywhere; }
.evalbook-site pre { max-height: 32rem; padding: var(--pit-space-3); overflow: auto; white-space: pre-wrap; overflow-wrap: anywhere; background: var(--pit-surface-sunken); color: var(--pit-foreground); border: 1px solid var(--pit-border); border-radius: var(--pit-radius-sm); font-family: var(--pit-font-mono); font-size: var(--pit-text-xs); }
.evalbook-site details { margin: var(--pit-space-4) 0; }
.evalbook-site summary { cursor: pointer; color: var(--pit-muted-foreground); }
.evalbook-site .status { display: inline-block; padding: var(--pit-space-1) var(--pit-space-2); border-radius: var(--pit-radius-sm); font-weight: 600; text-transform: uppercase; }
.evalbook-site small { display: block; margin-top: var(--pit-space-1); }
.evalbook-site footer { margin-top: var(--pit-space-5); font-size: var(--pit-text-xs); }
.evalbook-site .trend-group { padding: var(--pit-space-4); border: 1px solid var(--pit-border); border-radius: var(--pit-radius-sm); background: var(--pit-surface-sunken); }
.evalbook-site .trend-group > summary { color: var(--pit-foreground); font-weight: 600; }
.evalbook-site .trend-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(--pit-space-5); }
.evalbook-site figure { min-width: 0; margin: var(--pit-space-4) 0; }
.evalbook-site figcaption { color: var(--pit-muted-foreground); font-size: var(--pit-text-xs); }
.evalbook-site .trend-chart { width: 100%; overflow: visible; }
.evalbook-site .trend-chart text { fill: var(--pit-muted-foreground); font: var(--pit-text-xs) var(--pit-font-mono); }
.evalbook-site .trend-axis { fill: none; stroke: var(--pit-border); }
.evalbook-site .trend-line { fill: none; stroke-width: 2; }
.evalbook-site :is(.trend-line, .trend-point).pass { stroke: var(--pit-success); }
.evalbook-site :is(.trend-line, .trend-point).all_attempts { stroke: var(--pit-live); }
.evalbook-site :is(.trend-line, .trend-point).final_attempts { stroke: var(--pit-accent); stroke-dasharray: 4 3; }
.evalbook-site .trend-point { fill: var(--pit-surface); stroke-width: 2; }
.evalbook-site .trend-point:hover { fill: var(--pit-foreground); }
.evalbook-site .pass-rate { display: block; margin-top: var(--pit-space-2); white-space: nowrap; }
.evalbook-site .history-table table { min-width: 68rem; }
.evalbook-site .history-table td:first-child { max-width: 22rem; }
.evalbook-site .history-table details { margin: var(--pit-space-2) 0; }
.evalbook-site .history-table li { margin: var(--pit-space-2) 0; }
.evalbook-site .incomplete { background: var(--pit-warning-surface); color: var(--pit-warning); }
@media (max-width: 760px) {
.evalbook-site main { padding: var(--pit-space-4); }
.evalbook-site .grid { grid-template-columns: 1fr; }
.evalbook-site .trend-grid { grid-template-columns: 1fr; }
.evalbook-site .trend-grid figure { overflow-x: auto; }
.evalbook-site .trend-chart { min-width: 30rem; }
.evalbook-site .matrix th:first-child { width: 12rem; min-width: 10rem; }
.evalbook-site .badge { white-space: normal; text-align: center; }
}

15
packages/paperclip-runner/devtools/issue-thread/src/main.tsx
View File

@ -3,12 +3,25 @@ import { createRoot } from "react-dom/client";
import { App } from "./App";
import "./issue-thread.css";
import "./evalbook-site.css";
// Hosted Evalbooks use inert JSON plus the same trusted viewer bundle. No
// inline executable script or network fetch is needed to load an attempt.
const reportData = document.getElementById("paperclip-eval-report");
if (reportData !== null) {
window.__PAPERCLIP_EVAL_REPORT__ = JSON.parse(
reportData.textContent ?? "null",
);
}
// `?capture=1` freezes animation, caret, and smooth scrolling so the
// screenshot matrix is byte-stable across runs (contract §10.1).
const params = new URLSearchParams(window.location.search);
const hashQuery = window.location.hash.split("?")[1] ?? "";
if (params.get("capture") === "1" || new URLSearchParams(hashQuery).get("capture") === "1") {
if (
params.get("capture") === "1" ||
new URLSearchParams(hashQuery).get("capture") === "1"
) {
document.documentElement.dataset.capture = "true";
}

BIN
packages/paperclip-runner/docs/images/evalbook-chat/failed.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 173 KiB

BIN
packages/paperclip-runner/docs/images/evalbook-chat/missing-recording.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 135 KiB

BIN
packages/paperclip-runner/docs/images/evalbook-chat/passed.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 200 KiB

109
packages/paperclip-runner/docs/runner-protocol-live-evals.md
View File

@ -1,5 +1,84 @@
# Direct live Runner protocol evals
## One Evalbook presentation
Every new report uses the canonical Evalbook grid and the existing Runner Lab
chat viewer for attempt drill-downs. There is no plain-HTML attempt fallback.
The grid, Latest, test-design, inventory and server-gate pages load the same
built stylesheet and fonts as the chat viewer. The long-term S3 history index
references that stylesheet inside an immutable campaign, so every page keeps
the same dark theme. Static site styles live in
`devtools/issue-thread/src/evalbook-site.css`, scoped to `.evalbook-site`;
colors and typography come from the Runner Lab token layer.
Missing recordings show a notice in the same viewer; missing viewer builds
fail generation. Build with
`pnpm --filter @paperclipai/paperclip-runner build:issue-thread` and provide
`--viewer-root` or `PAPERCLIP_EVAL_VIEWER_ROOT` to the canonical Python renderer.
The Actions artifact contains full evidence. S3 uses the same viewer with a
closed public DTO: only isolated mock-run conversation text, scrubbed private
references, named tool outcomes, and checks. Tool arguments/results, reasoning,
provider identities, and company snapshots stay private. The public notice
explains these redactions. An unverified isolation boundary yields no public
conversation, not a guessed reconstruction.
Public attempts use inert JSON and one shared viewer asset directory. The
publisher verifies each shell and asset against the exact same-run viewer build,
checks the public payload contract and local links, and rejects other scripts.
The CSP prohibits network calls, forms and external resources. Supply
`PAPERCLIP_RUNNER_PROTOCOL_EVAL_VIEWER_DIR` to the publisher. The workflow sends
a viewer-only artifact to that job; raw attempts and provider secrets stay out.
After publication succeeds, the publishing job writes **Open this run's
Evalbook** and **All eval runs** links to the Actions summary. Its deployment
URL also points to the exact immutable report, not to the downloadable ZIP.
Failed publication does not advertise a successful deployment.
Before uploading, the report job runs the actual built application in Chromium
against representative passing, failing and missing-recording pages in both
reports. It checks initial rendering, tool expansion, read-only controls,
navigation, reload, a narrow viewport and the public no-API-request boundary.
In full-evidence reports, a tool's **View in Evidence** link selects the Evidence
tab and highlights its record, including when reopening that same record after
switching tabs. The browser check exercises this cross-link too.
Run the same check locally with
`node packages/paperclip-runner/scripts/verify-runner-evalbook-viewer.mjs --report-root /path/to/report`.
Add `--screenshots /path/to/proof` for visual evidence. Public screenshots are
retained in the aggregate Actions artifact, not added to the public report's
closed file allowlist.
These screenshots are generated acceptance-test evidence from the scrubbed
completed campaign, not a separate report design:
![Passing chat replay with expanded tool evidence](images/evalbook-chat/passed.png)
![Failed chat replay with visible assertions](images/evalbook-chat/failed.png)
![Missing recording shown in the same viewer](images/evalbook-chat/missing-recording.png)
### Refresh a completed report without calling models
Download the aggregate Actions artifact, then:
```sh
node packages/paperclip-runner/scripts/refresh-runner-protocol-eval-report.mjs \
--source /path/to/downloaded-aggregate \
--evals-root /path/to/paperclip-evals \
--viewer-root packages/paperclip-runner/dist-issue-thread \
--output /path/to/new-refresh-directory \
--revision chat-v1
```
Publish the returned `reportRoot` with the normal history publisher. The new ID
is `gha-RUN-ATTEMPT-report-chat-v1`. Original reports remain immutable; history
adds a labeled refresh and retains the source campaign, original measurement
timestamp, renderer digest, and `providerCalls: 0`. Scores and evaluated source
revisions do not change. This is not a new model qualification run. Future live
runs create chat reports automatically.
Refreshes are ordered by their render time in the history list, but keep the
original measurement timestamp and never replace the latest or latest-green
qualification identities. The HTML history links each measurement to its newest
presentation, while retaining every original bundle and listing refreshes separately.
Refreshes also recover retry-inclusive estimated and provider-list costs from the
retained attempt records. They never add another model-cost measurement.
This is the provider-backed, one-turn protocol qualification layer in
`paperclipai/paperclip-evals/evals/paperclip-runner`. It is intentionally
separate from both the browser full-stack model E2E and the stress-derived
@ -149,11 +228,11 @@ read-only Runner issue-thread attempt pages, and raw immutable run records.
Public publishing uses a separate projection and a separate trusted OIDC job.
The projection retains model/config identity, status, usage totals, and check
outcomes but removes provider session identifiers, transcripts, semantic-tool
outcomes and scrubbed mock conversation, but removes provider session identifiers, semantic-tool
payloads, state revisions, traces, remote profile identities, and raw failure
text. The same Evalbook `report` command renders that projection, so the public
grid and test pages have the standard Evalbook layout. The publisher rejects
scripts, remote resources, symlinks, unknown paths, broken links, raw session
grid, test pages and chat viewer have the standard Evalbook layout. The publisher rejects
untrusted scripts, remote resources, symlinks, unknown paths, broken links, raw session
fields, and credential-shaped values.
S3 publication is additive:
@ -178,9 +257,27 @@ runner-protocol-evals/
Campaign files use immutable cache headers and a digest manifest. Reusing a
campaign ID with different bytes fails closed. Only the root history and
pointer files are mutable, and the publisher never deletes objects. The root
history retains at most 200 records, reserving one record for the latest green
campaign when it would otherwise fall outside that window so its pointer stays
valid.
history retains **all** run records; it no longer drops entries after 200 campaigns.
The history index includes:
- Pass-rate and cost timelines, grouped by identical cell/model/driver membership
and eval-suite SHA. Different suites and subsets cannot silently share a baseline.
- Regression and recovery lists against the previous matching run, linking to the
affected tests. Infrastructure failures stay distinct from behavior failures.
- Estimated cost and provider-reported list cost, shown separately, never added.
New campaigns include all retained attempts (including retries). Backfilled old
campaigns with only final-cell usage are labeled **historical final attempts only**.
Missing usage is unknown, not zero; partial totals use `≥` and display coverage.
- Exact Paperclip and eval-suite commit links (full SHA on hover), the source ref,
and the GitHub Actions run. These identify the code **evaluated**, not merely
the commit used to render an old report.
- A separate report-refresh list, excluded from trend points and model-spend totals.
`history.json` stores a versioned, derived `analytics` projection separately from
immutable campaign records. The publisher backfills missing analytics from each
original `campaign.json`; a refresh may enrich costs using retained raw attempts
only when its source metadata and scores still match the original record.
The publishing job uses dedicated `RUNNER_PROTOCOL_EVAL_HISTORY_*` variables
when present and falls back to the existing Runner E2E history role, region,

2
packages/paperclip-runner/package.json
View File

@ -115,7 +115,7 @@
"test:capability-evals": "vitest run src/conformance/capability-eval-suite.test.ts",
"test:eval-slice": "pnpm run ensure:eval-build-deps && vitest run src/eval",
"test:runner-workflow-evals": "pnpm run ensure:eval-build-deps && node --test scripts/render-runner-workflow-evalbook.test.mjs && vitest run src/eval/workflow-evals.test.ts src/eval/live-workflow-executor.test.ts",
"test:runner-protocol-eval-publish": "node --test scripts/runner-protocol-eval-campaign.test.mjs scripts/publish-runner-protocol-eval-history.test.mjs scripts/runner-protocol-eval-workflow-security.test.mjs",
"test:runner-protocol-eval-publish": "node --test scripts/runner-protocol-eval-campaign.test.mjs scripts/publish-runner-protocol-eval-history.test.mjs scripts/runner-protocol-eval-metrics.test.mjs scripts/runner-protocol-eval-workflow-security.test.mjs scripts/public-eval-chat.test.mjs",
"check:runner-workflow-traceability": "pnpm run build:typescript && node scripts/check-runner-workflow-traceability.mjs",
"report:capability-evals": "pnpm run build:typescript && node scripts/run-capability-eval-suite.mjs",
"report:capability-live-evals": "pnpm run build:typescript && cargo build --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/run-capability-live-eval-matrix.mjs",

217
packages/paperclip-runner/scripts/public-eval-chat.mjs Normal file
View File

@ -0,0 +1,217 @@
// Public replay is a new DTO, never a recursive copy of a provider artifact.
export const PUBLIC_CHAT_SCHEMA =
"paperclip.runner-protocol-eval.public-chat/v1";
export const PUBLIC_CHAT_NOTICE =
"Public replay of an isolated mock eval. Conversation text is scrubbed; provider identities, tool payloads, traces, and company-state snapshots are withheld. Full evidence remains in the access-controlled Actions artifact.";
export const SECRET_TEXT = [
/\b(?:AKIA|ASIA)[0-9A-Z]{16}\b/gu,
/\bsk-[A-Za-z0-9_-]{16,}\b/gu,
/\b(?:gh[pousr]_|github_pat_)[A-Za-z0-9_]{16,}\b/gu,
/\bBearer\s+[A-Za-z0-9._~+\/-]{8,}=*/giu,
/-----BEGIN [^-]*PRIVATE KEY-----[\s\S]*?-----END [^-]*PRIVATE KEY-----/gu,
/\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\b/gu,
/\b(?:https?|file|s3):\/\/[^\s<>"')]+/giu,
/\barn:aws[^\s<>"')]+/gu,
/(?:\/(?:Users|home|tmp|private|var)\/|[A-Z]:\\)[^\s<>"')]+/gu,
/\b(?:api[_-]?key|access[_-]?token|secret|password|authorization|cookie)\s*[=:]\s*[^\s,;]+/giu,
];
export function publicText(value, privateValues = []) {
let text = typeof value === "string" ? value : "";
for (const secret of privateValues) {
if (typeof secret === "string" && secret.length >= 8)
text = text.replaceAll(secret, "[redacted]");
}
for (const pattern of SECRET_TEXT) text = text.replace(pattern, "[redacted]");
return text.length > 40_000 ? `${text.slice(0, 40_000)}\n[truncated]` : text;
}
function privateIdentities(value, found = new Set()) {
if (value && typeof value === "object") {
for (const [key, child] of Object.entries(value)) {
if (
/(?:session|profile|account|runtime|endpoint|memory|agentversion).*id$|arn$|token$|secret$|password$/i.test(
key,
) &&
typeof child === "string"
)
found.add(child);
else if (child && typeof child === "object")
privateIdentities(child, found);
}
}
return [...found];
}
function timestamp(value) {
return typeof value === "string" && /^\d{4}-\d\d-\d\dT[\d:.]+Z$/.test(value)
? value
: "1970-01-01T00:00:00.000Z";
}
function operation(value) {
return typeof value === "string" && /^[a-z][a-z_]{0,79}$/.test(value)
? value
: "unknown_operation";
}
export function publicChatView(artifact, evalCase) {
const privateValues = privateIdentities(artifact);
const scrub = (value) => publicText(value, privateValues);
const network = artifact.snapshot?.networkEvidence;
// Only the dedicated mock eval boundary can publish recorded conversation.
// Early infrastructure failures still receive a viewer with an honest notice.
const isolated =
network?.realPaperclipRequests === 0 &&
Array.isArray(network?.childPaperclipEnvironmentKeys) &&
network.childPaperclipEnvironmentKeys.length === 0;
const source =
isolated &&
artifact.issueThread?.schema === "paperclip.capability.issue-thread-view.v1"
? artifact.issueThread
: null;
const evidence = Object.fromEntries(
[
"tools",
"calls",
"authorization",
"control_plane",
"runner",
"state",
"traceability",
"parity",
].map((key) => [key, []]),
);
const turns = (source?.turns ?? []).map((turn, turnIndex) => {
const turnId = `public-turn-${turnIndex + 1}`;
const items = [];
for (const item of turn.items ?? []) {
const base = {
id: `public-item-${turnIndex + 1}-${items.length + 1}`,
at: timestamp(item.at),
};
if (
["user_message", "agent_message", "durable_comment"].includes(item.kind)
) {
items.push({
...base,
kind: item.kind === "user_message" ? "user_message" : "agent_message",
author: item.kind === "user_message" ? "You (eval prompt)" : "Agent",
body: scrub(item.body),
streaming: false,
});
} else if (item.kind === "tool_activity") {
const operationId = operation(item.operationId);
const status = ["ok", "denied", "running"].includes(item.status)
? item.status
: "running";
const result = {
outcome: status,
detail: "Tool payload withheld from public replay.",
};
const recordId = `public-call-${turnIndex + 1}-${items.length + 1}`;
items.push({
...base,
kind: "tool_activity",
operationId,
status,
summary: `${operationId}: ${status}`,
input: { detail: "Arguments withheld from public replay." },
result,
evidenceRef: { section: "calls", recordId },
});
if (status !== "running")
evidence.calls.push({
id: recordId,
turnId,
operationId,
version: 1,
providerRequest: operationId,
dispatchedCommand: operationId,
outcome: status,
result,
redactions: ["arguments", "result payload", "provider identities"],
threadAnchorId: base.id,
});
}
// Provider activity, reasoning, raw events, file refs and unrecognized
// future item kinds are deliberately not part of the public contract.
}
return {
id: turnId,
ordinal: turnIndex + 1,
mode: "replay",
toolCallCount: items.filter((item) => item.kind === "tool_activity")
.length,
at: timestamp(turn.at),
stoppedByUser: turn.stoppedByUser === true,
items,
};
});
if (!turns.some((turn) => turn.items.length > 0)) {
turns.length = 0;
turns.push({
id: "public-turn-1",
ordinal: 1,
mode: "replay",
toolCallCount: 0,
at: timestamp(artifact.snapshot?.createdAt),
stoppedByUser: false,
items: [
{
id: "public-notice",
at: timestamp(artifact.snapshot?.createdAt),
kind: "system_notice",
glyph: "–",
text: "No publishable conversation was recorded for this attempt. See the checks and the access-controlled artifact for diagnostics.",
evidenceRef: { section: "runner", recordId: "public-notice" },
},
],
});
}
return {
schema: "paperclip.capability.issue-thread-view.v1",
sessionId: "public-report",
mode: "replay",
identity: {
agentLabel: "Recorded agent",
runnerLabel: "Recorded runner",
runnerAttached: false,
controlPlaneLabel: "Mock Paperclip",
controlPlaneTooltip: PUBLIC_CHAT_NOTICE,
replaySource: "live",
},
issue: {
identifier: "EVAL",
title: scrub(evalCase.title || evalCase.id),
status: [
"backlog",
"todo",
"in_progress",
"in_review",
"done",
"blocked",
"cancelled",
].includes(source?.issue?.status)
? source.issue.status
: "in_review",
priority: "medium",
assignee: null,
runState: "Read-only public replay",
scenarioId: evalCase.id,
fixtureProfile: evalCase.id,
},
turns,
composer: {
state: "disabled",
helper: null,
reason: "Read-only eval report",
pendingInteractionId: null,
},
evidence,
connection: { state: "closed", attempt: 0 },
replay: null,
renderedAt: timestamp(source?.renderedAt || artifact.snapshot?.createdAt),
};
}

273
packages/paperclip-runner/scripts/public-eval-chat.test.mjs Normal file
View File

@ -0,0 +1,273 @@
import assert from "node:assert/strict";
import { mkdtemp, mkdir, writeFile, cp, rm, symlink } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
import {
publicChatView,
publicText,
PUBLIC_CHAT_SCHEMA,
PUBLIC_CHAT_NOTICE,
} from "./public-eval-chat.mjs";
import {
publicViewerShell,
validatePublicChatPayload,
trustedViewerFiles,
} from "./public-eval-viewer.mjs";
import { validatePublicProtocolEvalReport } from "./publish-runner-protocol-eval-history.mjs";
function artifact() {
return {
providerSessionId: "private-session-canary",
snapshot: {
networkEvidence: {
realPaperclipRequests: 0,
childPaperclipEnvironmentKeys: [],
},
},
issueThread: {
schema: "paperclip.capability.issue-thread-view.v1",
issue: { status: "blocked" },
turns: [
{
items: [
{ kind: "user_message", body: "Please block this task." },
{
kind: "agent_message",
body: "Blocked. private-session-canary https://private.example/path",
privateField: "private-field-canary",
},
{
kind: "tool_activity",
operationId: "block_task",
status: "ok",
input: { secret: "argument-canary" },
result: { token: "result-canary" },
},
{ kind: "thinking", body: "reasoning-canary" },
{ kind: "future_kind", body: "future-canary" },
],
},
],
},
};
}
function payload() {
return {
attemptId: "attempt-01",
caseId: "block-task",
disposition: "pass",
passed: true,
checks: [],
publication: { schema: PUBLIC_CHAT_SCHEMA, notice: PUBLIC_CHAT_NOTICE },
view: publicChatView(artifact(), { id: "block-task" }),
devtools: null,
navigation: { suiteHref: "../../index.html", previous: null, next: null },
run: {
model: "test",
provider: "test",
sessionId: "public-report",
effectiveModelHistory: [],
managedProfile: null,
acpxProfile: null,
usage: null,
},
};
}
test("projects only isolated recorded messages and bounded tool facts", () => {
const view = publicChatView(artifact(), { id: "block-task" });
assert.equal(view.issue.status, "blocked");
assert.deepEqual(
view.turns[0].items.map((item) => item.kind),
["user_message", "agent_message", "tool_activity"],
);
assert.match(view.turns[0].items[1].body, /Blocked/);
assert.doesNotMatch(JSON.stringify(view), /canary|private\.example/);
assert.deepEqual(view.turns[0].items[2].input, {
detail: "Arguments withheld from public replay.",
});
validatePublicChatPayload(payload());
for (const networkEvidence of [
undefined,
{ realPaperclipRequests: 1, childPaperclipEnvironmentKeys: [] },
{
realPaperclipRequests: 0,
childPaperclipEnvironmentKeys: ["PAPERCLIP_API_KEY"],
},
]) {
const source = artifact();
source.snapshot.networkEvidence = networkEvidence;
const unavailable = publicChatView(source, { id: "block-task" });
assert.equal(unavailable.turns[0].items[0].kind, "system_notice");
assert.doesNotMatch(JSON.stringify(unavailable), /Please block|Blocked\./);
}
});
test("scrubs credentials and private references before truncating text", () => {
for (const secret of [
"sk-" + "a".repeat(32),
"ghp_" + "b".repeat(32),
"Bearer abcdef123456",
"password=secret-canary",
"/Users/someone/private.txt",
"arn:aws:service:region:account:resource",
"-----BEGIN PRIVATE KEY-----\n" +
"x".repeat(41_000) +
"\n-----END PRIVATE KEY-----",
]) {
assert.equal(publicText(secret), "[redacted]");
}
assert.match(publicText("a".repeat(41_000)), /\[truncated\]$/);
});
test("turns containing only withheld items still show the missing-recording notice", () => {
const source = artifact();
source.issueThread.turns = [
{ items: [{ kind: "thinking", body: "private-reasoning" }] },
{ items: [] },
];
const view = publicChatView(source, { id: "missing" });
assert.equal(view.turns.length, 1);
assert.equal(view.turns[0].items[0].kind, "system_notice");
assert.match(view.turns[0].items[0].text, /No publishable conversation/);
});
test("trusted viewer rejects index and assets symlinks outside its root", async () => {
const root = await mkdtemp(join(tmpdir(), "eval-viewer-symlinks-"));
try {
const viewer = join(root, "viewer");
const outside = join(root, "outside");
await mkdir(viewer);
await mkdir(outside);
await writeFile(
join(outside, "index.html"),
'<script type="module"></script>',
);
await mkdir(join(outside, "assets"));
await writeFile(join(outside, "assets/app.js"), "// outside canary");
await symlink(join(outside, "index.html"), join(viewer, "index.html"));
await symlink(join(outside, "assets"), join(viewer, "assets"));
await assert.rejects(trustedViewerFiles(viewer), /symlinks/);
await rm(join(viewer, "index.html"));
await writeFile(
join(viewer, "index.html"),
'<script type="module"></script>',
);
await assert.rejects(trustedViewerFiles(viewer), /symlinks/);
} finally {
await rm(root, { recursive: true, force: true });
}
});
test("fails closed on unknown fields, raw tools, private evidence and identities", () => {
for (const mutate of [
(p) => {
p.view.turns[0].items[0].extra = "unprojected";
},
(p) => {
p.view.turns[0].items[2].input = { password: "oops" };
},
(p) => {
p.view.evidence.calls[0].result.detail = "raw-result";
},
(p) => {
p.view.evidence.state.push({ secret: "raw-state" });
},
(p) => {
p.run.providerSessionId = "private-session";
},
(p) => {
p.view.turns[0].items[1].body = "sk-" + "x".repeat(30);
},
(p) => {
p.devtools = {};
},
(p) => {
p.view.composer.state = "ready";
},
]) {
const value = payload();
mutate(value);
assert.throws(() => validatePublicChatPayload(value));
}
});
test("publisher permits only the exact trusted shell/assets and valid local navigation", async () => {
const root = await mkdtemp(join(tmpdir(), "eval-chat-contract-"));
try {
const viewer = join(root, "trusted");
const report = join(root, "report");
await mkdir(join(viewer, "assets"), { recursive: true });
await mkdir(join(report, "attempts/attempt-01"), { recursive: true });
const index =
'<!doctype html><html><head><script type="module" src="./assets/app.js"></script><link rel="stylesheet" href="./assets/app.css"></head><body><div id="root"></div></body></html>';
await writeFile(join(viewer, "index.html"), index);
await writeFile(join(viewer, "assets/app.js"), "// trusted build");
await writeFile(join(viewer, "assets/app.css"), ":root {}");
await cp(join(viewer, "assets"), join(report, "viewer/assets"), {
recursive: true,
});
await writeFile(
join(report, "index.html"),
'<a href="attempts/attempt-01/index.html">PASS</a>',
);
await writeFile(
join(report, "campaign.json"),
JSON.stringify({
schema: "paperclip.runner-protocol-eval.campaign/v1",
campaignId: "gha-42-1",
}),
);
const page = join(report, "attempts/attempt-01/index.html");
const writePayload = async (value) =>
writeFile(
page,
publicViewerShell(
index,
JSON.stringify(value).replaceAll("<", "\\u003c"),
),
);
await writePayload(payload());
await validatePublicProtocolEvalReport(report, { viewerRoot: viewer });
await assert.rejects(validatePublicProtocolEvalReport(report));
await writeFile(
join(report, "viewer/assets/app.js"),
"// substituted build",
);
await assert.rejects(
validatePublicProtocolEvalReport(report, { viewerRoot: viewer }),
/trusted/,
);
await writeFile(join(report, "viewer/assets/app.js"), "// trusted build");
await writeFile(
page,
publicViewerShell(index, JSON.stringify(payload())) +
"<script>alert(1)</script>",
);
await assert.rejects(
validatePublicProtocolEvalReport(report, { viewerRoot: viewer }),
/trusted shell/,
);
const escaped = payload();
escaped.view.turns[0].items[0].body =
'</script><script>alert("not executable")</script>';
await writePayload(escaped);
await validatePublicProtocolEvalReport(report, { viewerRoot: viewer });
const broken = payload();
broken.navigation.next = {
label: "Next attempt",
href: "../missing/index.html",
};
await writePayload(broken);
await assert.rejects(
validatePublicProtocolEvalReport(report, { viewerRoot: viewer }),
/link|reference/i,
);
await symlink(viewer, join(root, "symlink"));
await assert.rejects(trustedViewerFiles(join(root, "symlink")));
} finally {
await rm(root, { recursive: true, force: true });
}
});

234
packages/paperclip-runner/scripts/public-eval-viewer.mjs Normal file
View File

@ -0,0 +1,234 @@
import { readFile, readdir, lstat } from "node:fs/promises";
import { join } from "node:path";
import { PUBLIC_CHAT_SCHEMA, SECRET_TEXT } from "./public-eval-chat.mjs";
export const PUBLIC_VIEWER_CSP =
"default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'none'; connect-src 'none'; form-action 'none'; base-uri 'none'";
export const PUBLIC_VIEWER_DATA =
/<script type="application\/json" id="paperclip-eval-report">([^<]*)<\/script>/u;
const ASSET = /^[A-Za-z0-9][A-Za-z0-9._-]*\.(?:js|css|woff2)$/;
export function publicViewerShell(index, encodedPayload) {
return index
.replaceAll('"./assets/', '"../../viewer/assets/')
.replace(
"<head>",
`<head>\n <meta http-equiv="Content-Security-Policy" content="${PUBLIC_VIEWER_CSP}">`,
)
.replace(
'<script type="module"',
`<script type="application/json" id="paperclip-eval-report">${encodedPayload}</script>\n <script type="module"`,
);
}
export async function trustedViewerFiles(viewerRoot) {
const rootStat = viewerRoot ? await lstat(viewerRoot) : null;
if (!rootStat || rootStat.isSymbolicLink() || !rootStat.isDirectory())
throw new Error(
"A trusted viewer build is required for public chat reports",
);
const indexStat = await lstat(join(viewerRoot, "index.html"));
const assetsStat = await lstat(join(viewerRoot, "assets"));
if (
indexStat.isSymbolicLink() ||
!indexStat.isFile() ||
assetsStat.isSymbolicLink() ||
!assetsStat.isDirectory()
)
throw new Error("Trusted viewer must not use symlinks");
const index = await readFile(join(viewerRoot, "index.html"), "utf8");
const files = new Map();
for (const entry of await readdir(join(viewerRoot, "assets"), {
withFileTypes: true,
})) {
if (!entry.isFile() || entry.isSymbolicLink() || !ASSET.test(entry.name))
throw new Error("Unexpected trusted viewer asset");
files.set(
`viewer/assets/${entry.name}`,
await readFile(join(viewerRoot, "assets", entry.name)),
);
}
if (
![...files.keys()].some((name) => name.endsWith(".js")) ||
!index.includes('<script type="module"')
)
throw new Error("Incomplete trusted viewer build");
return { index, files };
}
export function validatePublicChatPayload(payload) {
if (
payload?.publication?.schema !== PUBLIC_CHAT_SCHEMA ||
payload.view?.sessionId !== "public-report" ||
payload.view?.composer?.state !== "disabled" ||
payload.view?.connection?.state !== "closed" ||
payload.devtools !== null
)
throw new Error(
"Public attempt must contain the read-only public chat projection",
);
const allowed = new Set([
"attemptId",
"caseId",
"disposition",
"passed",
"checks",
"view",
"devtools",
"navigation",
"run",
"publication",
]);
if (Object.keys(payload).some((key) => !allowed.has(key)))
throw new Error("Unknown public chat payload field");
const fields = (value, names) => {
if (
!value ||
typeof value !== "object" ||
Array.isArray(value) ||
Object.keys(value).some((key) => !names.split(" ").includes(key))
)
throw new Error("Unknown public chat projection field");
};
fields(payload.publication, "schema notice");
fields(payload.navigation, "suiteHref previous next");
for (const link of [payload.navigation.previous, payload.navigation.next])
if (link !== null) fields(link, "label href");
fields(
payload.run,
"model provider driver providerVersion runnerProvider acpxAgent acpxProfile requestedModel effectiveModelHistory configuration sessionId providerSessionId agentVersion managedProfile retainedSession retainedSessionStatus fixtureDigest runnerPackageDigest runnerdDigest startedAt finishedAt durationMs runnerBuild initialRevision finalRevision usage",
);
if (
payload.run.effectiveModelHistory?.length ||
payload.run.managedProfile != null ||
payload.run.acpxProfile != null
)
throw new Error("Public replay contains private provider metadata");
if (payload.run.usage !== null)
fields(
payload.run.usage,
"agentTurns providerRequests inputTokens outputTokens cachedInputTokens reasoningTokens providerReportedCostNanodollars estimatedCostNanodollars pricingVersion",
);
fields(
payload.view,
"schema sessionId mode identity issue turns composer evidence connection replay renderedAt",
);
fields(
payload.view.identity,
"agentLabel runnerLabel runnerAttached controlPlaneLabel controlPlaneTooltip replaySource",
);
fields(
payload.view.issue,
"identifier title status priority assignee runState scenarioId fixtureProfile",
);
fields(payload.view.composer, "state helper reason pendingInteractionId");
fields(payload.view.connection, "state attempt");
fields(
payload.view.evidence,
"tools calls authorization control_plane runner state traceability parity",
);
for (const check of payload.checks) {
fields(
check,
"id kind passed detail evidenceRefs title description definition anchor",
);
fields(check.definition, "id kind");
fields(check.anchor, "kind id");
if (check.evidenceRefs.length)
throw new Error("Public replay contains raw evidence references");
}
const visit = (value, key = "") => {
if (typeof value === "string") {
if (
/(?:sessionId|providerSessionId)$/i.test(key) &&
!["public-report", "unknown", "redacted"].includes(value)
)
throw new Error("Public replay contains a private session identity");
for (const pattern of SECRET_TEXT) {
pattern.lastIndex = 0;
if (pattern.test(value))
throw new Error(
"Public replay contains credential or private reference material",
);
}
} else if (value && typeof value === "object") {
for (const [name, child] of Object.entries(value)) {
if (
/^(?:managedProfile|acpxProfile|providerTrace|mockState|stateHistory|trace|environment|env|apiKey|accessToken|password|secret)$/i.test(
name,
) &&
child != null
)
throw new Error("Public replay contains a private field");
visit(child, name);
}
}
};
visit(payload);
for (const section of [
"tools",
"authorization",
"control_plane",
"runner",
"state",
"traceability",
"parity",
]) {
if (
!Array.isArray(payload.view.evidence?.[section]) ||
payload.view.evidence[section].length
)
throw new Error("Public replay contains unprojected evidence");
}
for (const call of payload.view.evidence.calls) {
fields(
call,
"id turnId operationId version providerRequest dispatchedCommand outcome result redactions threadAnchorId",
);
fields(call.result, "outcome detail");
if (call.result.detail !== "Tool payload withheld from public replay.")
throw new Error("Public replay contains raw call evidence");
}
for (const turn of payload.view.turns ?? []) {
fields(turn, "id ordinal mode toolCallCount at stoppedByUser items");
for (const item of turn.items ?? []) {
if (
![
"user_message",
"agent_message",
"tool_activity",
"system_notice",
].includes(item.kind)
)
throw new Error("Public replay contains an unprojected item");
const shapes = {
user_message: "kind id at author body streaming",
agent_message: "kind id at author body streaming",
tool_activity:
"kind id at operationId status summary input result evidenceRef",
system_notice: "kind id at glyph text evidenceRef",
};
fields(item, shapes[item.kind]);
if (item.evidenceRef) fields(item.evidenceRef, "section recordId");
if (
item.kind === "tool_activity" &&
(JSON.stringify(item.input) !==
JSON.stringify({
detail: "Arguments withheld from public replay.",
}) ||
Object.keys(item.result).sort().join(",") !== "detail,outcome" ||
item.result.detail !== "Tool payload withheld from public replay.")
)
throw new Error("Public replay contains a raw tool payload");
}
}
}
export function validatePublicViewerPage(content, trustedIndex) {
const match = content.match(PUBLIC_VIEWER_DATA);
if (!match || publicViewerShell(trustedIndex, match[1]) !== content)
throw new Error("Public viewer page differs from the trusted shell");
const payload = JSON.parse(match[1]);
validatePublicChatPayload(payload);
return payload;
}

227
packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs
View File

@ -2,6 +2,7 @@
import { createHash } from "node:crypto";
import { execFile } from "node:child_process";
import {
appendFile,
lstat,
mkdtemp,
mkdir,
@ -13,13 +14,23 @@ import {
import { tmpdir } from "node:os";
import { extname, join, relative, resolve, sep } from "node:path";
import { promisify } from "node:util";
import { enrichProtocolEvalHistory } from "./runner-protocol-eval-metrics.mjs";
import { renderProtocolEvalHistoryIndex } from "./runner-protocol-eval-history-view.mjs";
export { renderProtocolEvalHistoryIndex } from "./runner-protocol-eval-history-view.mjs";
import {
trustedViewerFiles,
validatePublicViewerPage,
} from "./public-eval-viewer.mjs";
const execFileAsync = promisify(execFile);
const SAFE_CAMPAIGN = /^gha-[1-9][0-9]*-[1-9][0-9]*$/;
const SAFE_CAMPAIGN =
/^gha-[1-9][0-9]*-[1-9][0-9]*(?:-report-[a-z0-9][a-z0-9-]{0,39})?$/;
const SAFE_REPORT_PATHS = [
/^(?:index|latest|inventory|real-server)\.html$/,
/^tests\/[A-Za-z0-9][A-Za-z0-9._-]{0,199}\.html$/,
/^attempts\/[A-Za-z0-9][A-Za-z0-9._-]{0,199}\.html$/,
/^attempts\/[A-Za-z0-9][A-Za-z0-9._-]{0,199}\/index\.html$/,
/^viewer\/assets\/[A-Za-z0-9][A-Za-z0-9._-]*\.(?:js|css|woff2)$/,
/^campaign\.json$/,
];
const CREDENTIAL_PATTERNS = [
@ -38,21 +49,11 @@ const ACTIVE_HTML_PATTERNS = [
/javascript\s*:/iu,
/(?:src|href)\s*=\s*["'](?:https?:)?\/\//iu,
];
const MAX_HISTORY_CAMPAIGNS = 200;
function json(value) {
return `${JSON.stringify(value, null, 2)}\n`;
}
function html(value) {
return String(value ?? "")
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
async function loadObject(path) {
const value = JSON.parse(await readFile(path, "utf8"));
if (value === null || Array.isArray(value) || typeof value !== "object") {
@ -136,9 +137,16 @@ function internalHtmlHrefs(content) {
.filter((href) => href && !href.startsWith("#"));
}
export async function validatePublicProtocolEvalReport(reportRoot) {
export async function validatePublicProtocolEvalReport(
reportRoot,
{ viewerRoot } = {},
) {
const root = resolve(reportRoot);
const files = await relativeFiles(root);
const hasChat = files.some((file) =>
/^attempts\/[^/]+\/index\.html$/.test(file),
);
const viewer = hasChat ? await trustedViewerFiles(viewerRoot) : null;
if (!files.includes("index.html") || !files.includes("campaign.json")) {
throw new Error(
"Public protocol eval report requires index.html and campaign.json",
@ -157,21 +165,44 @@ export async function validatePublicProtocolEvalReport(reportRoot) {
`Public protocol eval file exceeds its size boundary: ${file}`,
);
}
if (file.startsWith("viewer/")) {
const expected = viewer?.files.get(file);
if (!expected || !expected.equals(await readFile(absolute)))
throw new Error(
`Public viewer asset differs from trusted build: ${file}`,
);
continue;
}
const content = await readFile(absolute, "utf8");
for (const pattern of CREDENTIAL_PATTERNS) {
if (pattern.test(content))
throw new Error(
`Public report contains credential/session material: ${file}`,
);
const richAttempt = /^attempts\/[^/]+\/index\.html$/.test(file);
const payload = richAttempt
? validatePublicViewerPage(content, viewer.index)
: null;
if (!richAttempt) {
for (const pattern of CREDENTIAL_PATTERNS) {
if (pattern.test(content))
throw new Error(
`Public report contains credential/session material: ${file}`,
);
}
if (extname(file) !== ".html") continue;
for (const pattern of ACTIVE_HTML_PATTERNS) {
if (pattern.test(content))
throw new Error(
`Public report contains active or remote HTML: ${file}`,
);
}
}
if (extname(file) !== ".html") continue;
for (const pattern of ACTIVE_HTML_PATTERNS) {
if (pattern.test(content))
throw new Error(
`Public report contains active or remote HTML: ${file}`,
);
}
for (const href of internalHtmlHrefs(content)) {
const navigation = payload
? [
payload.navigation?.suiteHref,
payload.navigation?.previous?.href,
payload.navigation?.next?.href,
].filter(Boolean)
: [];
for (const href of [...internalHtmlHrefs(content), ...navigation]) {
if (typeof href !== "string" || /[?:\\]|^\/|^[a-z]+:/i.test(href))
throw new Error(`Unsafe report navigation in ${file}`);
const clean = href.split("#", 1)[0].split("?", 1)[0];
const target = resolve(
root,
@ -189,6 +220,15 @@ export async function validatePublicProtocolEvalReport(reportRoot) {
}
}
}
if (viewer) {
for (const file of viewer.files.keys())
if (!files.includes(file))
throw new Error(`Missing public viewer asset: ${file}`);
if (files.some((file) => /^attempts\/[^/]+\.html$/.test(file)))
throw new Error(
"Chat Evalbook must not mix in legacy plain attempt pages",
);
}
const campaign = await loadObject(join(root, "campaign.json"));
if (
campaign.schema !== "paperclip.runner-protocol-eval.campaign/v1" ||
@ -199,11 +239,17 @@ export async function validatePublicProtocolEvalReport(reportRoot) {
return { files, campaign };
}
export async function createProtocolEvalBundleManifest(reportRoot, campaignId) {
export async function createProtocolEvalBundleManifest(
reportRoot,
campaignId,
{ viewerRoot } = {},
) {
if (!SAFE_CAMPAIGN.test(campaignId))
throw new Error("Unsafe protocol eval campaign ID");
const { files, campaign } =
await validatePublicProtocolEvalReport(reportRoot);
const { files, campaign } = await validatePublicProtocolEvalReport(
reportRoot,
{ viewerRoot },
);
if (campaign.campaignId !== campaignId)
throw new Error("Report campaign ID does not match publication target");
const entries = await Promise.all(
@ -250,6 +296,9 @@ export function protocolEvalHistoryRecord(campaign, publicRoot) {
totals: campaign.totals,
rosters: campaign.rosters,
source: campaign.source,
...(campaign.reportRevision
? { reportRevision: campaign.reportRevision }
: {}),
};
}
@ -266,30 +315,29 @@ export function mergeProtocolEvalHistory(history, record) {
);
}
const campaigns = existing
? history.campaigns
? [...history.campaigns]
: [...history.campaigns, record];
campaigns.sort((left, right) =>
right.generatedAt.localeCompare(left.generatedAt),
);
const latest = campaigns[0] ?? null;
const activityAt = (campaign) =>
campaign.reportRevision?.renderedAt ?? campaign.generatedAt;
const activityOrder = (left, right) =>
activityAt(right).localeCompare(activityAt(left));
campaigns.sort(activityOrder);
// Report revisions are discoverable history entries, never qualification runs.
const qualifications = campaigns
.filter((campaign) => !campaign.reportRevision)
.sort((left, right) => right.generatedAt.localeCompare(left.generatedAt));
const latest = qualifications[0] ?? null;
const latestGreen =
campaigns.find((campaign) => campaign.complete && campaign.allPassed) ??
null;
const retained = campaigns.slice(0, MAX_HISTORY_CAMPAIGNS);
if (
latestGreen &&
!retained.some(
(campaign) => campaign.campaignId === latestGreen.campaignId,
)
) {
retained[retained.length - 1] = latestGreen;
}
qualifications.find(
(campaign) => campaign.complete && campaign.allPassed,
) ?? null;
return {
schema: history.schema,
updatedAt: new Date().toISOString(),
latestCampaignId: latest?.campaignId ?? null,
latestGreenCampaignId: latestGreen?.campaignId ?? null,
campaigns: retained,
campaigns,
...(history.analytics ? { analytics: history.analytics } : {}),
};
}
@ -323,44 +371,6 @@ export function buildProtocolEvalPointers(history) {
};
}
function date(value) {
return new Intl.DateTimeFormat("en-US", {
dateStyle: "medium",
timeStyle: "short",
timeZone: "UTC",
}).format(new Date(value));
}
export function renderProtocolEvalHistoryIndex(history) {
const rows = history.campaigns.length
? history.campaigns
.map((campaign) => {
const status =
campaign.complete && campaign.allPassed ? "passed" : "failed";
const rosters = campaign.rosters
.map(
(roster) =>
`${html(roster.model)} · ${roster.passed}/${roster.selected}`,
)
.join("<br>");
return `<tr><td><a href="${html(campaign.publicUrl)}"><code>${html(campaign.campaignId)}</code></a><small>${html(date(campaign.generatedAt))} UTC</small></td><td><span class="status ${status}">${status}</span></td><td><strong>${html(campaign.totals.passed)}/${html(campaign.totals.selected)}</strong><small>${html(campaign.totals.behaviorFailures)} behavior · ${html(campaign.totals.infrastructureFailures)} infrastructure</small></td><td>${rosters}</td><td><code>${html(campaign.source?.paperclip?.sha?.slice(0, 8) ?? "unknown")}</code><small>evals ${html(campaign.source?.evals?.sha?.slice(0, 8) ?? "unknown")}</small></td><td><a href="${html(campaign.publicUrl)}">Open Evalbook →</a></td></tr>`;
})
.join("")
: '<tr><td colspan="6" class="empty">No campaigns have been published yet.</td></tr>';
const latest = history.campaigns.find(
(campaign) => campaign.campaignId === history.latestCampaignId,
);
const latestGreen = history.campaigns.find(
(campaign) => campaign.campaignId === history.latestGreenCampaignId,
);
return `<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width"><meta name="color-scheme" content="light dark"><title>Runner protocol eval campaigns · Paperclip</title>
<style>:root{color-scheme:light dark;--bg:#fff;--fg:#172019;--muted:#667069;--line:#dfe3dc;--raised:#f8f9f6;--pass:#17603a;--pass-bg:#e5f3e9;--fail:#942f2b;--fail-bg:#f9e5e3} @media(prefers-color-scheme:dark){:root{--bg:#141413;--fg:#fafafa;--muted:#aaa;--line:#ffffff1f;--raised:#1c1c1b;--pass:#65d58c;--pass-bg:#22c55e1f;--fail:#ff7770;--fail-bg:#dc26262e}} *{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--fg);font:14px/1.5 ui-sans-serif,system-ui,sans-serif}main{width:min(1560px,calc(100% - 48px));margin:48px auto 72px}h1{margin:0;font-size:clamp(34px,4vw,56px);line-height:1.05;letter-spacing:-.04em}p{max-width:760px;color:var(--muted);font-size:16px}a{color:inherit;text-underline-offset:3px}.pointers{display:flex;gap:10px;margin:28px 0 18px}.pointers a{padding:8px 11px;border:1px solid var(--line);border-radius:8px;background:var(--raised);text-decoration:none}.table{overflow:auto;border:1px solid var(--line);border-radius:12px}table{width:100%;border-collapse:collapse}th,td{padding:13px 14px;border-bottom:1px solid var(--line);text-align:left;vertical-align:top}th{background:var(--raised);color:var(--muted);font-size:10px;text-transform:uppercase;letter-spacing:.06em}tr:last-child td{border:0}small{display:block;margin-top:4px;color:var(--muted);font-size:10px}.status{display:inline-block;padding:3px 8px;border-radius:99px;font-size:10px;font-weight:750;text-transform:uppercase}.passed{color:var(--pass);background:var(--pass-bg)}.failed{color:var(--fail);background:var(--fail-bg)}.empty{padding:48px;text-align:center;color:var(--muted)}footer{margin-top:18px;color:var(--muted);font-size:11px}@media(max-width:700px){main{width:calc(100% - 28px);margin-top:28px}}</style></head>
<body><main><div><small>Paperclip quality engineering</small><h1>Runner protocol eval campaigns</h1><p>Versioned direct live-runner Evalbook reports. Full provider transcripts, session identifiers, state, and raw tool evidence remain in access-controlled workflow artifacts.</p></div>
<nav class="pointers">${latest ? `<a href="${html(latest.publicUrl)}">Latest · ${html(latest.campaignId)}</a>` : ""}${latestGreen ? `<a href="${html(latestGreen.publicUrl)}">Latest green · ${html(latestGreen.campaignId)}</a>` : ""}</nav>
<div class="table"><table><thead><tr><th>Campaign</th><th>Status</th><th>Cells</th><th>Models / rosters</th><th>Source</th><th></th></tr></thead><tbody>${rows}</tbody></table></div><footer>Updated ${html(date(history.updatedAt))} UTC · Immutable campaign bundles · Canonical Evalbook layout with public-safe evidence projections</footer></main></body></html>`;
}
function awsObject(bucket, key) {
return `s3://${bucket}/${key}`;
}
@ -435,19 +445,31 @@ async function uploadImmutableReport(bucket, prefix, reportRoot) {
);
}
export async function publishProtocolEvalHistory({ reportRoot, destination }) {
export async function publishProtocolEvalHistory({
reportRoot,
destination,
viewerRoot,
}) {
const validatedDestination =
validateProtocolEvalHistoryDestination(destination);
const { campaign } = await validatePublicProtocolEvalReport(reportRoot);
const { campaign } = await validatePublicProtocolEvalReport(reportRoot, {
viewerRoot,
});
const viewer = await trustedViewerFiles(viewerRoot);
const stylesheet = viewer.index.match(/<link rel="stylesheet" crossorigin href="\.\/(assets\/[A-Za-z0-9._-]+\.css)">/)?.[1];
if (!stylesheet || !viewer.files.get(`viewer/${stylesheet}`)?.includes(".evalbook-site"))
throw new Error("Published history requires the same-run Runner Lab site theme");
const stylesheetHref = `campaigns/${campaign.campaignId}/viewer/${stylesheet}`;
const manifest = await createProtocolEvalBundleManifest(
reportRoot,
campaign.campaignId,
{ viewerRoot },
);
const temporary = await mkdtemp(
join(tmpdir(), "runner-protocol-eval-history-"),
);
const historyKey = `${validatedDestination.prefix}/history.json`;
const history = mergeProtocolEvalHistory(
const mergedHistory = mergeProtocolEvalHistory(
(await downloadJson(
validatedDestination.bucket,
historyKey,
@ -458,6 +480,15 @@ export async function publishProtocolEvalHistory({ reportRoot, destination }) {
`${validatedDestination.publicBaseUrl}/${validatedDestination.prefix}`,
),
);
const history = await enrichProtocolEvalHistory(mergedHistory, {
currentCampaign: campaign,
loadCampaign: async (id) => {
if (!SAFE_CAMPAIGN.test(id)) throw new Error("Unsafe historical campaign ID");
return downloadJson(validatedDestination.bucket,
`${validatedDestination.prefix}/campaigns/${id}/campaign.json`,
join(temporary, `${id}.json`));
},
});
const campaignPrefix = `${validatedDestination.prefix}/campaigns/${campaign.campaignId}`;
const manifestKey = `${campaignPrefix}/bundle-manifest.json`;
const existing = await downloadJson(
@ -502,7 +533,7 @@ export async function publishProtocolEvalHistory({ reportRoot, destination }) {
);
}
const index = join(temporary, "index.html");
await writeFile(index, renderProtocolEvalHistoryIndex(history));
await writeFile(index, renderProtocolEvalHistoryIndex(history, stylesheetHref));
await uploadFile(
validatedDestination.bucket,
`${validatedDestination.prefix}/index.html`,
@ -513,11 +544,31 @@ export async function publishProtocolEvalHistory({ reportRoot, destination }) {
campaignId: campaign.campaignId,
bundleDigest: manifest.bundleDigest,
historySize: history.campaigns.length,
reportUrl: `${validatedDestination.publicBaseUrl}/${campaignPrefix}/index.html`,
historyUrl: `${validatedDestination.publicBaseUrl}/${validatedDestination.prefix}/index.html`,
};
}
export async function writeProtocolEvalPublicationLinks(result, environment = process.env) {
const { campaignId, reportUrl, historyUrl } = result;
if (!SAFE_CAMPAIGN.test(campaignId)) throw new Error("Invalid published campaign ID");
const safeUrl = (value) => {
const url = new URL(value);
if (url.protocol !== "https:" || url.username || url.password || /[\r\n<>]/.test(value))
throw new Error("Invalid published report URL");
return url.href;
};
const report = safeUrl(reportUrl);
const history = safeUrl(historyUrl);
if (environment.GITHUB_OUTPUT)
await appendFile(environment.GITHUB_OUTPUT, `report_url=${report}\nhistory_url=${history}\n`);
if (environment.GITHUB_STEP_SUMMARY)
await appendFile(environment.GITHUB_STEP_SUMMARY, `## Published Runner Evalbook\n\n[Open this run's Evalbook](<${report}>) · [All eval runs](<${history}>)\n\nCampaign: \`${campaignId}\`\n\nPublic replay uses the Runner Lab theme; full evidence is in the workflow artifact.\n`);
}
async function main() {
const result = await publishProtocolEvalHistory({
viewerRoot: process.env.PAPERCLIP_RUNNER_PROTOCOL_EVAL_VIEWER_DIR,
reportRoot: resolve(
process.env.PAPERCLIP_RUNNER_PROTOCOL_EVAL_PUBLIC_REPORT_DIR ??
"runner-protocol-eval-public-report",
@ -531,9 +582,11 @@ async function main() {
process.env.RUNNER_PROTOCOL_EVAL_HISTORY_PUBLIC_BASE_URL ?? "",
},
});
await writeProtocolEvalPublicationLinks(result);
console.log(
`Published immutable protocol eval campaign ${result.campaignId} (${result.bundleDigest}) and ${result.historySize} history record(s)`,
);
console.log(`Evalbook: ${result.reportUrl}\nRun history: ${result.historyUrl}`);
}
if (

87
packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.test.mjs
View File

@ -1,5 +1,5 @@
import assert from "node:assert/strict";
import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
@ -14,9 +14,28 @@ import {
renderProtocolEvalHistoryIndex,
validateProtocolEvalHistoryDestination,
validatePublicProtocolEvalReport,
writeProtocolEvalPublicationLinks,
} from "./publish-runner-protocol-eval-history.mjs";
const roots = [];
test("successful publication exposes exact report and history links to Actions", async () => {
const root = await mkdtemp(join(tmpdir(), "evalbook-publication-links-"));
roots.push(root);
const environment = { GITHUB_OUTPUT: join(root, "outputs"), GITHUB_STEP_SUMMARY: join(root, "summary") };
const result = {
campaignId: "gha-42-2",
reportUrl: "https://reports.example/runner-protocol-evals/campaigns/gha-42-2/index.html",
historyUrl: "https://reports.example/runner-protocol-evals/index.html",
};
await writeProtocolEvalPublicationLinks(result, environment);
assert.equal(await readFile(environment.GITHUB_OUTPUT, "utf8"), `report_url=${result.reportUrl}\nhistory_url=${result.historyUrl}\n`);
const summary = await readFile(environment.GITHUB_STEP_SUMMARY, "utf8");
assert.ok(summary.includes(`[Open this run's Evalbook](<${result.reportUrl}>)`));
assert.ok(summary.includes(`[All eval runs](<${result.historyUrl}>)`));
await assert.rejects(writeProtocolEvalPublicationLinks({ ...result, reportUrl: "https://example.test/\nreport_url=bad" }, environment));
assert.throws(() => renderProtocolEvalHistoryIndex(emptyProtocolEvalHistory(), "https://untrusted.example/style.css"));
});
test.afterEach(async () => {
await Promise.all(
roots.splice(0).map((root) => rm(root, { recursive: true, force: true })),
@ -111,12 +130,13 @@ test("keeps publication to the canonical static Evalbook surface", () => {
"inventory.html",
"tests/get-context.html",
"attempts/attempt-01.html",
"attempts/attempt-01/index.html",
"viewer/assets/index-build.js",
"campaign.json",
]) {
assert.equal(isPublicProtocolEvalPath(file), true, file);
}
for (const file of [
"attempts/attempt-01/index.html",
"runs/attempt/artifact.json",
"provider-trace.log",
"../secret",
@ -177,7 +197,10 @@ test("retains immutable history and independent latest-green pointers", () => {
const pointers = buildProtocolEvalPointers(history);
assert.equal(pointers.latest.campaign.campaignId, "gha-43-1");
assert.equal(pointers.latestGreen.campaign.campaignId, "gha-42-1");
const index = renderProtocolEvalHistoryIndex(history);
const index = renderProtocolEvalHistoryIndex(history, "campaigns/gha-43-1/viewer/assets/index.css");
assert.match(index, /class="evalbook-site"/);
assert.match(index, /href="campaigns\/gha-43-1\/viewer\/assets\/index.css"/);
assert.doesNotMatch(index, /<style>|color-scheme:light/);
assert.match(index, /Runner protocol eval campaigns/);
assert.match(index, /Open Evalbook/);
assert.match(index, /34\/35/);
@ -188,7 +211,61 @@ test("retains immutable history and independent latest-green pointers", () => {
);
});
test("retains the latest green pointer outside the 200 newest campaigns", () => {
test("report refreshes never replace qualification pointers or evict older measurements", () => {
const record = (value) =>
protocolEvalHistoryRecord(
value,
"https://reports.example/runner-protocol-evals",
);
let history = mergeProtocolEvalHistory(
emptyProtocolEvalHistory(),
record(campaign()),
);
history = mergeProtocolEvalHistory(
history,
record(
campaign({
campaignId: "gha-43-1",
generatedAt: "2026-09-06T00:00:00.000Z",
allPassed: false,
}),
),
);
for (let index = 0; index < 205; index++) {
history = mergeProtocolEvalHistory(
history,
record(
campaign({
campaignId: `gha-42-1-report-refresh-${index}`,
// Defend even against an incorrectly timestamped refresh producer.
generatedAt: "2026-09-07T00:00:00.000Z",
reportRevision: {
sourceCampaignId: "gha-42-1",
renderedAt: "2026-09-07T00:00:00.000Z",
providerCalls: 0,
},
}),
),
);
}
assert.equal(history.latestCampaignId, "gha-43-1");
assert.equal(history.latestGreenCampaignId, "gha-42-1");
assert.equal(
buildProtocolEvalPointers(history).latest.campaign.campaignId,
"gha-43-1",
);
assert.equal(
buildProtocolEvalPointers(history).latestGreen.campaign.campaignId,
"gha-42-1",
);
assert.equal(history.campaigns.length, 207);
assert.match(
renderProtocolEvalHistoryIndex(history, "campaigns/gha-42-1-report-chat-v1/viewer/assets/index.css"),
/Report refresh · no new model calls/,
);
});
test("retains every run and the latest green pointer beyond 200 campaigns", () => {
const green = protocolEvalHistoryRecord(
campaign(),
"https://reports.example/runner-protocol-evals",
@ -210,7 +287,7 @@ test("retains the latest green pointer outside the 200 newest campaigns", () =>
);
}
assert.equal(history.campaigns.length, 200);
assert.equal(history.campaigns.length, 202);
assert.equal(history.latestCampaignId, "gha-243-1");
assert.equal(history.latestGreenCampaignId, "gha-42-1");
assert.equal(history.campaigns.at(-1).campaignId, "gha-42-1");

106
packages/paperclip-runner/scripts/refresh-runner-protocol-eval-report.mjs Normal file
View File

@ -0,0 +1,106 @@
#!/usr/bin/env node
// Re-render immutable recorded evidence; this command never invokes a model.
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdir, readFile, writeFile, lstat, readdir } from "node:fs/promises";
import { join, resolve } from "node:path";
import { sanitizeProtocolEvalRuns } from "./runner-protocol-eval-campaign.mjs";
import { validatePublicProtocolEvalReport } from "./publish-runner-protocol-eval-history.mjs";
import { sumAttemptCosts } from "./runner-protocol-eval-metrics.mjs";
export async function refreshProtocolEvalReport({
sourceRoot,
evalsRoot,
viewerRoot,
outputRoot,
revision,
selection,
renderedAt = new Date().toISOString(),
}) {
if (!/^[a-z0-9][a-z0-9-]{0,39}$/.test(revision ?? ""))
throw new Error("A safe, unique --revision is required");
if (await lstat(outputRoot).catch(() => null))
throw new Error("Report refresh output must be a new directory");
const campaign = JSON.parse(
await readFile(join(sourceRoot, "campaign.json"), "utf8"),
);
if (!/^gha-[1-9][0-9]*-[1-9][0-9]*$/.test(campaign.campaignId ?? ""))
throw new Error("Expected an original Actions campaign");
const program = join(
evalsRoot,
"evals/paperclip-runner/tools/eval_program.py",
);
const rendererDigest = createHash("sha256")
.update(await readFile(program))
.digest("hex");
await mkdir(outputRoot, { recursive: true });
const runsRoot = join(outputRoot, "public-runs");
const reportRoot = join(outputRoot, "report");
await sanitizeProtocolEvalRuns({
runsRoot: join(sourceRoot, "runs"),
publicRunsRoot: runsRoot,
});
execFileSync(
"python3",
[
program,
"report",
"--runs-root",
runsRoot,
"--output",
reportRoot,
"--viewer-root",
viewerRoot,
"--public-viewer",
"--inventory",
join(evalsRoot, "evals/paperclip-runner/inventory.json"),
"--coverage-matrix",
join(evalsRoot, "evals/paperclip-runner/coverage-matrix.json"),
],
{ stdio: "inherit" },
);
const refreshed = {
...campaign,
...(selection ? { selection } : {}),
// Recover retry-inclusive cost from retained evidence, not just winning cells.
costs: sumAttemptCosts(await Promise.all((await readdir(runsRoot)).map(async (id) =>
JSON.parse(await readFile(join(runsRoot, id, "artifact.json"), "utf8")).usage))),
campaignId: `${campaign.campaignId}-report-${revision}`,
// A presentation refresh is not a new model measurement.
generatedAt: campaign.generatedAt,
reportRevision: {
sourceCampaignId: campaign.campaignId,
sourceGeneratedAt: campaign.generatedAt,
renderedAt,
rendererDigest,
providerCalls: 0,
},
};
await writeFile(
join(reportRoot, "campaign.json"),
`${JSON.stringify(refreshed, null, 2)}\n`,
);
await validatePublicProtocolEvalReport(reportRoot, { viewerRoot });
return { reportRoot, campaignId: refreshed.campaignId, providerCalls: 0 };
}
if (
process.argv[1] &&
resolve(process.argv[1]) === resolve(import.meta.filename)
) {
const arg = (name) => {
const index = process.argv.indexOf(name);
if (index < 0 || !process.argv[index + 1])
throw new Error(`Missing ${name}`);
return process.argv[index + 1];
};
console.log(
await refreshProtocolEvalReport({
sourceRoot: resolve(arg("--source")),
evalsRoot: resolve(arg("--evals-root")),
viewerRoot: resolve(arg("--viewer-root")),
outputRoot: resolve(arg("--output")),
revision: arg("--revision"),
}),
);
}

11
packages/paperclip-runner/scripts/render-runner-workflow-evalbook.mjs
View File

@ -309,6 +309,15 @@ export async function renderRunnerWorkflowWithCanonicalEvalbook({
environment = process.env,
}) {
const program = await resolveCanonicalEvalProgram(packageRoot, environment);
const viewerRoot = resolve(
environment.PAPERCLIP_EVAL_VIEWER_ROOT ??
resolve(packageRoot, "dist-issue-thread"),
);
await access(resolve(viewerRoot, "index.html")).catch(() => {
throw new Error(
"Evalbook requires the chat viewer. Run pnpm --filter @paperclipai/paperclip-runner build:issue-thread first.",
);
});
const runsRoot = resolve(outputDirectory, "evalbook-runs");
await rm(runsRoot, { recursive: true, force: true });
const attempts = await writeRunnerWorkflowEvalbookAttempts({
@ -331,6 +340,8 @@ export async function renderRunnerWorkflowWithCanonicalEvalbook({
runsRoot,
"--output",
outputDirectory,
"--viewer-root",
viewerRoot,
]);
const programBytes = await readFile(program);
const manifest = {

62
packages/paperclip-runner/scripts/runner-protocol-eval-campaign.mjs
View File

@ -11,6 +11,12 @@ import {
writeFile,
} from "node:fs/promises";
import { basename, dirname, join, relative, resolve, sep } from "node:path";
import { sumAttemptCosts } from "./runner-protocol-eval-metrics.mjs";
import {
publicChatView,
PUBLIC_CHAT_SCHEMA,
PUBLIC_CHAT_NOTICE,
} from "./public-eval-chat.mjs";
const SAFE_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/;
const ATTEMPT_FILES = new Set([
@ -97,7 +103,9 @@ async function maintainedRosterSelection(programRoot) {
return basename(rosterPath);
});
if (selected.length === 0 || new Set(selected).size !== selected.length) {
throw new Error("Maintained live campaign must contain unique enabled rosters");
throw new Error(
"Maintained live campaign must contain unique enabled rosters",
);
}
return new Set(selected);
}
@ -120,8 +128,7 @@ export async function buildProtocolEvalCatalog({
const programRoot = resolve(evalsRoot, "evals/paperclip-runner");
const rosterRoot = resolve(programRoot, "rosters");
const requested = parseRosterSelection(rosterSelection);
const selected =
requested ?? (await maintainedRosterSelection(programRoot));
const selected = requested ?? (await maintainedRosterSelection(programRoot));
const rosterFiles = (await readdir(rosterRoot, { withFileTypes: true }))
.filter(
(entry) =>
@ -201,6 +208,7 @@ export async function buildProtocolEvalCatalog({
schema: "paperclip.runner-protocol-eval.catalog/v1",
campaignId,
source,
selection: { kind: requested === null ? "maintained_full" : "subset", rosters: rosterSelection },
rosters,
cells,
matrices: shards.map((include) => ({ include })),
@ -432,6 +440,7 @@ export async function aggregateProtocolEvalCampaign({
}
const results = [];
const attemptUsages = [];
for (const cell of catalog.cells) {
const retained = retainedByCell.get(cell.cellId);
const attemptIds = retained?.attemptIds?.length
@ -445,6 +454,10 @@ export async function aggregateProtocolEvalCampaign({
}),
];
const finalAttemptId = attemptIds.at(-1);
for (const attemptId of attemptIds) {
const attempt = await loadObject(join(runsOut, attemptId, "artifact.json"));
attemptUsages.push(attempt.usage);
}
const [score, artifact] = await Promise.all([
loadObject(join(runsOut, finalAttemptId, "score.json")),
loadObject(join(runsOut, finalAttemptId, "artifact.json")),
@ -484,6 +497,8 @@ export async function aggregateProtocolEvalCampaign({
schema: "paperclip.runner-protocol-eval.campaign/v1",
campaignId: catalog.campaignId,
generatedAt,
selection: catalog.selection,
costs: sumAttemptCosts(attemptUsages),
source: {
paperclip: source.paperclip,
evals: source.evals,
@ -509,7 +524,8 @@ export async function aggregateProtocolEvalCampaign({
return campaign;
}
function publicArtifact(artifact) {
function publicArtifact(artifact, evalCase) {
const issueThread = publicChatView(artifact, evalCase);
const model = artifact.snapshot?.providerModel ?? {};
const infrastructure = artifact.infrastructureFailure;
const providerVersion =
@ -525,12 +541,29 @@ function publicArtifact(artifact) {
provider: artifact.provider,
driver: artifact.driver,
providerVersion,
retainedSession: false,
retainedSession: null,
retainedSessionStatus: "redacted from the public report",
usage: safeUsage(artifact.usage),
turn: { status: artifact.turn?.status ?? "failed" },
timing: {
startedAt:
typeof artifact.timing?.startedAt === "string"
? artifact.timing.startedAt
: null,
finishedAt:
typeof artifact.timing?.finishedAt === "string"
? artifact.timing.finishedAt
: null,
durationMs: Number.isFinite(artifact.timing?.durationMs)
? artifact.timing.durationMs
: null,
},
turn: {
status: artifact.turn?.status ?? "failed",
turnId: issueThread.turns.at(-1).id,
},
snapshot: {
createdAt: artifact.snapshot?.createdAt ?? artifact.createdAt,
sessionId: "public-report",
providerModel: {
id: model.id ?? artifact.requestedModel,
provider: model.provider ?? artifact.provider,
@ -539,6 +572,8 @@ function publicArtifact(artifact) {
evidence: [],
},
devtools: { revisions: [] },
publication: { schema: PUBLIC_CHAT_SCHEMA, notice: PUBLIC_CHAT_NOTICE },
issueThread,
...(infrastructure && typeof infrastructure === "object"
? {
infrastructureFailure: {
@ -622,15 +657,22 @@ export async function sanitizeProtocolEvalRuns({ runsRoot, publicRunsRoot }) {
await Promise.all([
writeFile(
join(destination, "artifact.json"),
json(publicArtifact(artifact)),
json(publicArtifact(artifact, evalCase)),
{ mode: 0o600 },
),
writeFile(join(destination, "score.json"), json(publicScore(score)), {
mode: 0o600,
}),
writeFile(join(destination, "case.json"), json(evalCase), {
mode: 0o600,
}),
writeFile(
join(destination, "case.json"),
json({
id: evalCase.id,
checks: (evalCase.checks ?? []).map(({ id, kind }) => ({ id, kind })),
}),
{
mode: 0o600,
},
),
writeFile(join(destination, "config.json"), json(publicConfig(config)), {
mode: 0o600,
}),

39
packages/paperclip-runner/scripts/runner-protocol-eval-campaign.test.mjs
View File

@ -111,6 +111,7 @@ test("catalogs roster plus case cells and emits bounded balanced shards", async
maxParallel: 80,
});
assert.equal(catalog.cells.length, 1);
assert.equal(catalog.selection.kind, "maintained_full");
assert.equal(catalog.cells[0].credentialName, "OPENROUTER_API_KEY");
assert.equal(catalog.maxParallelPerShard, 40);
assert.equal(catalog.matrices[0].include.length, 1);
@ -185,6 +186,7 @@ test("all selects the maintained enabled campaign and explicit diagnostics can s
diagnostic.rosters.map((roster) => roster.rosterId),
["protocol-live-disabled-model"],
);
assert.equal(diagnostic.selection.kind, "subset");
});
test("all fails closed when the maintained campaign is missing", async () => {
@ -252,6 +254,37 @@ test("aggregates retained attempts and synthesizes missing cells as infrastructu
);
});
test("campaign cost includes failed attempts before a successful retry", async () => {
const { root, config, evalCase } = await fixture();
const catalog = await buildProtocolEvalCatalog({ evalsRoot: root, campaignId: "gha-42-1" });
const catalogPath = join(root, "catalog.json");
const download = join(root, "downloads/cell");
await mkdir(download, { recursive: true });
await writeFile(catalogPath, JSON.stringify(catalog));
await writeFile(join(download, "cell.json"), JSON.stringify({
cellId: catalog.cells[0].cellId, caseId: evalCase.id,
rosterFile: catalog.cells[0].rosterFile, exitCode: 0,
}));
for (const [attemptId, amount, passed] of [["attempt-01", 2, false], ["attempt-02", 3, true]]) {
const directory = join(download, "runs", attemptId);
await mkdir(directory, { recursive: true });
for (const [file, value] of Object.entries({
"artifact.json": { attemptId, usage: { estimatedCostNanodollars: amount } },
"score.json": { attemptId, caseId: evalCase.id, passed, disposition: passed ? "passed" : "infrastructure_failure" },
"case.json": evalCase, "config.json": config,
})) await writeFile(join(directory, file), JSON.stringify(value));
}
const result = await aggregateProtocolEvalCampaign({
catalogPath, downloadsRoot: join(root, "downloads"), evalsRoot: root,
runsOut: join(root, "merged"), campaignOut: join(root, "campaign.json"), source: {},
});
assert.equal(result.totals.passed, 1);
assert.equal(result.results[0].usage.estimatedCostNanodollars, 3);
assert.equal(result.costs.estimated.nanodollars, 5);
assert.equal(result.costs.attempts, 2);
assert.equal(result.selection.kind, "maintained_full");
});
test("rejects downloaded cells that were not declared by the immutable catalog", async () => {
const { root } = await fixture();
const catalog = await buildProtocolEvalCatalog({
@ -286,7 +319,7 @@ test("rejects downloaded cells that were not declared by the immutable catalog",
);
});
test("public run projection removes provider sessions, traces, transcripts, evidence, and state", async () => {
test("public run projection removes raw evidence and gives unverified recordings an empty chat view", async () => {
const { root, config, evalCase } = await fixture();
const attemptId = "get-task-context-opencode-gha-42-1-attempt-01";
const source = join(root, "raw-runs", attemptId);
@ -357,12 +390,14 @@ test("public run projection removes provider sessions, traces, transcripts, evid
);
assert.doesNotMatch(
serialized,
/private-session|private transcript|private-turn|issueThread|trace/,
/private-session|private transcript|private-turn|"trace":/,
);
const artifact = JSON.parse(serialized);
assert.deepEqual(artifact.snapshot.transcript, []);
assert.deepEqual(artifact.snapshot.evidence, []);
assert.deepEqual(artifact.devtools.revisions, []);
assert.equal(artifact.issueThread.composer.state, "disabled");
assert.equal(artifact.issueThread.turns[0].items[0].kind, "system_notice");
const score = JSON.parse(
await readFile(join(root, "public-runs", attemptId, "score.json"), "utf8"),
);

123
packages/paperclip-runner/scripts/runner-protocol-eval-history-view.mjs Normal file
View File

@ -0,0 +1,123 @@
import { compareCampaignAnalytics } from "./runner-protocol-eval-metrics.mjs";
const escape = (value) => String(value ?? "").replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;").replaceAll('"', "&quot;").replaceAll("'", "&#39;");
const date = (value) => new Intl.DateTimeFormat("en-US", { dateStyle: "medium", timeStyle: "short", timeZone: "UTC" }).format(new Date(value));
const percentage = (run) => run.totals.selected ? 100 * run.totals.passed / run.totals.selected : null;
const dollars = (value) => `$${(value / 1e9).toFixed(6)}`;
export function costLabel(costs, field = "estimated") {
const metric = costs?.[field];
if (!Number.isFinite(metric?.nanodollars) || metric.nanodollars < 0) return "Unknown";
const incomplete = metric.recordedAttempts < costs.attempts;
return `${incomplete ? "≥ " : ""}${dollars(metric.nanodollars)}`;
}
function costCell(costs) {
const coverage = costs?.estimated?.recordedAttempts ?? 0;
const attempts = costs?.attempts ?? 0;
return `<strong>${escape(costLabel(costs))}</strong><small>Estimated · ${coverage}/${attempts} entries</small><small>Provider list: ${escape(costLabel(costs, "providerReported"))}</small><small>${costs?.scope === "all_attempts" ? "All attempts, including retries" : "Historical final attempts only"}</small>`;
}
function commit(repository, sha, label) {
if (!/^[a-f0-9]{40}$/.test(sha ?? "")) return `<small>${label}: unknown</small>`;
return `<small>${label}: <a href="https://github.com/${repository}/commit/${sha}" title="${sha}"><code>${sha.slice(0, 8)}</code></a></small>`;
}
function sourceLinks(run) {
const url = run.source?.workflowRunUrl;
const workflow = /^https:\/\/github\.com\/paperclipai\/paperclip\/actions\/runs\/[1-9][0-9]*$/.test(url ?? "")
? `<small><a href="${url}">GitHub Actions ↗</a></small>` : "";
return `${commit("paperclipai/paperclip", run.source?.paperclip?.sha, "Paperclip")}${commit("paperclipai/paperclip-evals", run.source?.evals?.sha, "Evals")}${workflow}<small>${escape(run.source?.paperclip?.ref ?? "")}</small>`;
}
function chart(runs, analytics, kind, id) {
const cost = kind === "cost";
const label = cost ? "Estimated cost over time (USD)" : "Pass rate over time (%)";
const values = runs.map((run) => cost ? analytics[run.campaignId]?.costs?.estimated?.nanodollars : percentage(run));
const max = cost ? Math.max(1, ...values.filter(Number.isFinite)) : 100;
const timestamps = runs.map((run) => Date.parse(run.generatedAt));
const elapsed = timestamps.at(-1) - timestamps[0];
const x = (index) => elapsed > 0 ? 48 + 504 * (timestamps[index] - timestamps[0]) / elapsed : 300;
const y = (value) => 145 - value / max * 116;
const scopes = cost ? ["all_attempts", "final_attempts"] : ["pass"];
const series = scopes.map((scope) => {
let drawing = false;
const path = values.map((value, index) => {
const present = Number.isFinite(value) && value >= 0 && (!cost || analytics[runs[index].campaignId]?.costs?.scope === scope);
if (!present) { drawing = false; return ""; }
const point = `${drawing ? "L" : "M"}${x(index).toFixed(2)},${y(value).toFixed(2)}`;
drawing = true;
return point;
}).join(" ");
return `<path class="trend-line ${scope}" d="${path}"/>`;
}).join("");
const points = values.map((value, index) => {
if (!Number.isFinite(value) || value < 0) return "";
const run = runs[index];
const metric = analytics[run.campaignId];
const description = `${date(run.generatedAt)} UTC · ${run.campaignId} · ${cost ? `${costLabel(metric?.costs)} (${metric?.costs?.scope})` : `${value.toFixed(1)}% (${run.totals.passed}/${run.totals.selected})`}`;
return `<a href="${escape(run.publicUrl)}" aria-label="${escape(description)}"><circle class="trend-point ${cost ? metric?.costs?.scope : "pass"}" cx="${x(index).toFixed(2)}" cy="${y(value).toFixed(2)}" r="4"><title>${escape(description)}</title></circle></a>`;
}).join("");
const tick = (run) => `${run.generatedAt.slice(5, 16).replace("T", " ")} UTC`;
return `<figure><figcaption>${label}</figcaption><svg class="trend-chart" viewBox="0 0 600 186" role="img" aria-labelledby="${id}"><title id="${id}">${label}. Each point links to its recorded run.</title><path class="trend-axis" d="M48,25V145H552"/><text x="40" y="33" text-anchor="end">${cost ? `$${(max / 1e9).toFixed(2)}` : "100%"}</text><text x="40" y="149" text-anchor="end">0</text>${series}${points}<text x="48" y="176">${escape(tick(runs[0]))}</text><text x="552" y="176" text-anchor="end">${escape(tick(runs.at(-1)))}</text></svg></figure>`;
}
function changeCell(run, analytics, previous) {
if (run.reportRevision) return '<span class="muted">Presentation only</span>';
if (!previous) return '<span class="muted">No matching baseline</span>';
const change = compareCampaignAnalytics(analytics[run.campaignId], analytics[previous.campaignId]);
if (!change) return '<span class="muted">Suite changed</span>';
const failures = (cells, destination) => cells.map((cell) => `<li><a href="${escape(destination.publicUrl)}tests/${encodeURIComponent(cell.caseId)}.html">${escape(cell.caseId)}</a> · ${escape(cell.rosterId)}${cell.disposition === "infrastructure_failure" ? " · infrastructure" : ""}</li>`).join("");
const counts = `${change.regressions.length} regressions · ${change.recoveries.length} recoveries`;
return `<details><summary>${counts}</summary><small>Versus <a href="${escape(previous.publicUrl)}">${escape(previous.campaignId)}</a></small>${change.regressions.length ? `<strong>Previously passing → failing</strong><ul>${failures(change.regressions, run)}</ul>` : ""}${change.recoveries.length ? `<strong>Previously failing → passing</strong><ul>${failures(change.recoveries, run)}</ul>` : ""}</details>`;
}
export function renderProtocolEvalHistoryIndex(history, stylesheetHref) {
if (!/^campaigns\/gha-[a-z0-9-]+\/viewer\/assets\/[A-Za-z0-9._-]+\.css$/.test(stylesheetHref ?? ""))
throw new Error("History requires an immutable campaign's Runner Lab stylesheet");
for (const run of history.campaigns) {
if (!["selected", "passed", "behaviorFailures", "infrastructureFailures"].every((field) => Number.isSafeInteger(run.totals?.[field]) && run.totals[field] >= 0)
|| !run.rosters.every((roster) => Number.isSafeInteger(roster.selected) && Number.isSafeInteger(roster.passed)))
throw new Error("History requires numeric recorded counts");
const url = new URL(run.publicUrl);
if (url.protocol !== "https:" || url.username || url.password)
throw new Error("History report links must be credential-free HTTPS URLs");
}
const analytics = history.analytics ?? {};
const presentations = new Map();
for (const refresh of history.campaigns.filter((run) => run.reportRevision)
.sort((a, b) => a.reportRevision.renderedAt.localeCompare(b.reportRevision.renderedAt)))
presentations.set(refresh.reportRevision.sourceCampaignId, refresh);
const measurements = history.campaigns.filter((run) => !run.reportRevision)
.map((run) => presentations.has(run.campaignId)
? { ...run, originalPublicUrl: run.publicUrl, publicUrl: presentations.get(run.campaignId).publicUrl }
: run)
.sort((a, b) => a.generatedAt.localeCompare(b.generatedAt) || a.campaignId.localeCompare(b.campaignId));
const groups = new Map();
const baselines = new Map();
for (const run of measurements) {
const key = analytics[run.campaignId]?.suiteKey;
if (!key || !run.complete) continue;
const group = groups.get(key) ?? [];
baselines.set(run.campaignId, group.at(-1));
group.push(run);
groups.set(key, group);
}
const ordered = [...groups.entries()].sort(([, a], [, b]) => b.at(-1).totals.selected - a.at(-1).totals.selected || b.at(-1).generatedAt.localeCompare(a.at(-1).generatedAt));
const trends = ordered.map(([key, runs], index) => {
const latest = runs.at(-1);
return `<details class="trend-group" ${index === 0 ? "open" : ""}><summary>${latest.totals.selected} cells · ${latest.rosters.length} configurations · ${runs.length} recorded runs · evals ${escape(latest.source.evals.sha.slice(0, 8))}</summary><div class="trend-grid">${chart(runs, analytics, "pass", `pass-${key}`)}${chart(runs, analytics, "cost", `cost-${key}`)}</div><p class="muted">Cost: solid cyan includes retries; dashed purple is historical final-attempt cost. Missing cost is a gap, not zero. Hover or select a point to inspect its run.</p></details>`;
}).join("") || '<p class="muted">Comparable run metadata has not been recorded yet.</p>';
const row = (run) => {
const metric = analytics[run.reportRevision?.sourceCampaignId ?? run.campaignId];
const status = !run.complete ? "incomplete" : run.allPassed ? "passed" : "failed";
const models = run.rosters.map((roster) => `<li>${escape(roster.model)} · ${roster.passed}/${roster.selected}<small>${escape(roster.driver)} · ${escape(roster.rosterId)}</small></li>`).join("");
const scope = metric?.selection === "maintained_full" ? "Full maintained suite" : metric?.selection === "subset" ? "Selected subset" : `${run.totals.selected} recorded cells`;
return `<tr><td><a href="${escape(run.publicUrl)}"><code>${escape(run.campaignId)}</code></a><small>${escape(date(run.generatedAt))} UTC</small>${run.reportRevision ? `<small>Report refresh · no new model calls · source ${escape(run.reportRevision.sourceCampaignId)}</small>` : `<small>${scope}</small>`}<small><a href="${escape(run.publicUrl)}">Open Evalbook →</a></small></td><td><span class="status ${status}">${status}</span><strong class="pass-rate">${run.totals.passed}/${run.totals.selected} · ${percentage(run)?.toFixed(1) ?? "—"}%</strong><small>${run.totals.behaviorFailures} behavior · ${run.totals.infrastructureFailures} infrastructure</small><details><summary>${run.rosters.length} model configurations</summary><ul>${models}</ul></details></td><td>${changeCell(run, analytics, baselines.get(run.campaignId))}</td><td>${run.reportRevision ? '<small>No additional model cost</small>' : costCell(metric?.costs)}</td><td>${sourceLinks(run)}</td></tr>`;
};
const table = (runs) => `<div class="table history-table" role="region" aria-label="Recorded eval runs" tabindex="0"><table><thead><tr><th>Run</th><th>Pass / fail</th><th>Change vs matching suite</th><th>Cost (USD)</th><th>Exact source</th></tr></thead><tbody>${runs.map(row).join("") || '<tr><td colspan="5">No campaigns have been published yet.</td></tr>'}</tbody></table></div>`;
const refreshes = history.campaigns.filter((run) => run.reportRevision);
const latest = measurements.find((run) => run.campaignId === history.latestCampaignId);
const green = measurements.find((run) => run.campaignId === history.latestGreenCampaignId);
return `<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width"><title>Runner protocol eval campaigns · Paperclip</title><link rel="stylesheet" href="${escape(stylesheetHref)}"></head><body class="evalbook-site"><main><header class="top"><a href="index.html">paperclip-runner evals</a><span class="badge">Run history</span></header><h1>Runner protocol eval campaigns</h1><p class="muted">Every recorded campaign, its cost, and its exact source. Public chat replays are linked below; full provider evidence remains in access-controlled workflow artifacts.</p><nav class="pointers">${latest ? `<a href="${escape(latest.publicUrl)}">Latest · ${escape(latest.campaignId)}</a>` : ""}${green ? `<a href="${escape(green.publicUrl)}">Latest green · ${escape(green.campaignId)}</a>` : ""}<a href="history.json">Download history JSON</a></nav><h2>Like-for-like trends</h2><p class="muted">Only identical cells, model configurations, and eval-suite SHAs are compared. A changed suite starts a separate series. Report refreshes never count as new measurements. Regressions distinguish model behavior from infrastructure failures.</p>${trends}<h2>All runs · ${measurements.length}</h2><p class="muted">Estimated cost and provider list cost are alternatives, not additive. ≥ means some entries lack usage. Historical final-only cost excludes retry spending. Commit links expose the full SHA on hover.</p>${table([...measurements].reverse())}${refreshes.length ? `<details><summary>Report refreshes · ${refreshes.length} (no new measurements)</summary>${table(refreshes)}</details>` : ""}<footer>Updated ${escape(date(history.updatedAt))} UTC · All run records retained · Immutable campaign bundles</footer></main></body></html>`;
}

97
packages/paperclip-runner/scripts/runner-protocol-eval-metrics.mjs Normal file
View File

@ -0,0 +1,97 @@
import { createHash } from "node:crypto";
import { isDeepStrictEqual } from "node:util";
const SHA = /^[a-f0-9]{40}$/;
const ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/;
// Provider list cost and our pricing estimate are alternative measurements,
// never additive. Missing usage is unknown, not a zero-dollar call.
export function sumAttemptCosts(usages, scope = "all_attempts") {
const metric = (field) => {
const values = usages.map((usage) => usage?.[field]).filter((value) =>
typeof value === "number" && Number.isFinite(value) && value >= 0);
return {
nanodollars: values.length ? values.reduce((sum, value) => sum + value, 0) : null,
recordedAttempts: values.length,
};
};
return {
scope,
attempts: usages.length,
estimated: metric("estimatedCostNanodollars"),
providerReported: metric("providerReportedCostNanodollars"),
};
}
export function campaignAnalytics(campaign) {
const results = campaign.results ?? [];
const cells = results.map((result) => [result.cellId, result.model, result.provider, result.driver]);
cells.sort((a, b) => JSON.stringify(a).localeCompare(JSON.stringify(b)));
const comparable = SHA.test(campaign.source?.evals?.sha ?? "")
&& cells.length > 0 && cells.length === campaign.totals?.selected
&& results.every((result) => typeof result.passed === "boolean")
&& results.filter((result) => result.passed).length === campaign.totals.passed
&& new Set(cells.map(([id]) => id)).size === cells.length
&& cells.every((cell) => cell.every((value) => typeof value === "string" && value.length > 0));
const suiteKey = comparable ? createHash("sha256")
.update(JSON.stringify([campaign.source.evals.sha, cells])).digest("hex") : null;
const suppliedCosts = campaign.costs;
const validCosts = suppliedCosts && ["all_attempts", "final_attempts"].includes(suppliedCosts.scope)
&& Number.isSafeInteger(suppliedCosts.attempts) && suppliedCosts.attempts >= 0
&& ["estimated", "providerReported"].every((field) => {
const metric = suppliedCosts[field];
return Number.isSafeInteger(metric?.recordedAttempts) && metric.recordedAttempts >= 0
&& metric.recordedAttempts <= suppliedCosts.attempts
&& (metric.nanodollars === null || typeof metric.nanodollars === "number" && Number.isFinite(metric.nanodollars) && metric.nanodollars >= 0);
});
return {
schema: "paperclip.runner-protocol-eval.analytics/v1",
suiteKey,
selection: campaign.selection?.kind === "maintained_full" ? "maintained_full"
: campaign.selection?.kind === "subset" ? "subset" : "recorded",
costs: validCosts ? {
scope: suppliedCosts.scope, attempts: suppliedCosts.attempts,
estimated: { nanodollars: suppliedCosts.estimated.nanodollars, recordedAttempts: suppliedCosts.estimated.recordedAttempts },
providerReported: { nanodollars: suppliedCosts.providerReported.nanodollars, recordedAttempts: suppliedCosts.providerReported.recordedAttempts },
} : sumAttemptCosts(results.map((result) => result.usage), "final_attempts"),
failedCells: results.filter((result) => !result.passed).map((result) => ({
cellId: ID.test(result.cellId) ? result.cellId : "unknown",
caseId: ID.test(result.caseId) ? result.caseId : "unknown",
rosterId: ID.test(result.rosterId) ? result.rosterId : "unknown",
disposition: result.disposition === "behavior_failure" ? "behavior_failure" : "infrastructure_failure",
})),
};
}
export function compareCampaignAnalytics(current, previous) {
if (!current?.suiteKey || current.suiteKey !== previous?.suiteKey) return null;
const before = new Set(previous.failedCells.map((cell) => cell.cellId));
const after = new Set(current.failedCells.map((cell) => cell.cellId));
return {
regressions: current.failedCells.filter((cell) => !before.has(cell.cellId)),
recoveries: previous.failedCells.filter((cell) => !after.has(cell.cellId)),
};
}
// History records and campaign bundles stay immutable. This separate, derived
// projection can be backfilled from old bundles or enriched by a report refresh.
export async function enrichProtocolEvalHistory(history, { currentCampaign, loadCampaign }) {
const analytics = { ...history.analytics };
const sourceId = currentCampaign.reportRevision?.sourceCampaignId ?? currentCampaign.campaignId;
for (const record of history.campaigns.filter((item) => !item.reportRevision)) {
const refresh = record.campaignId === sourceId ? currentCampaign : null;
if (!refresh && analytics[record.campaignId]?.schema === "paperclip.runner-protocol-eval.analytics/v1") continue;
const campaign = refresh ?? await loadCampaign(record.campaignId);
if (!campaign) continue;
if (!isDeepStrictEqual(campaign.source, record.source)
|| !isDeepStrictEqual(campaign.totals, record.totals))
throw new Error(`Campaign analytics do not match immutable history: ${record.campaignId}`);
const projected = campaignAnalytics(campaign);
const previous = analytics[record.campaignId];
// Never downgrade an all-attempt measurement with a historical final-only one.
if (previous?.costs?.scope === "all_attempts" && projected.costs.scope !== "all_attempts")
projected.costs = previous.costs;
analytics[record.campaignId] = projected;
}
return { ...history, analytics };
}

83
packages/paperclip-runner/scripts/runner-protocol-eval-metrics.test.mjs Normal file
View File

@ -0,0 +1,83 @@
import assert from "node:assert/strict";
import test from "node:test";
import { campaignAnalytics, compareCampaignAnalytics, enrichProtocolEvalHistory, sumAttemptCosts } from "./runner-protocol-eval-metrics.mjs";
import { costLabel, renderProtocolEvalHistoryIndex } from "./runner-protocol-eval-history-view.mjs";
import { emptyProtocolEvalHistory, mergeProtocolEvalHistory, protocolEvalHistoryRecord } from "./publish-runner-protocol-eval-history.mjs";
function campaign(id, failed = [], overrides = {}) {
const results = ["context", "create", "finish"].map((caseId) => ({
cellId: `codex--${caseId}`, caseId, rosterId: "codex", model: "mini", provider: "codex", driver: "codex_app_server",
passed: !failed.includes(caseId), disposition: failed.includes(caseId) ? "behavior_failure" : "passed",
usage: { estimatedCostNanodollars: 1_000_000, providerReportedCostNanodollars: 2_000_000 },
}));
return {
campaignId: `gha-${id}-1`, generatedAt: `2026-09-${String(id).padStart(2, "0")}T00:00:00.000Z`,
source: { paperclip: { sha: "a".repeat(40), ref: "refs/heads/master" }, evals: { sha: "b".repeat(40) }, workflowRunUrl: `https://github.com/paperclipai/paperclip/actions/runs/${id}` },
complete: true, allPassed: failed.length === 0, results,
selection: { kind: "maintained_full" },
totals: { selected: 3, passed: 3 - failed.length, behaviorFailures: failed.length, infrastructureFailures: 0 },
rosters: [{ rosterId: "codex", model: "mini", driver: "codex_app_server", selected: 3, passed: 3 - failed.length }],
...overrides,
};
}
test("costs include retries without adding provider list cost to estimates or inventing missing usage", () => {
const costs = sumAttemptCosts([{ estimatedCostNanodollars: 2, providerReportedCostNanodollars: 7 }, { estimatedCostNanodollars: 3 }, {}, { estimatedCostNanodollars: -1 }, { estimatedCostNanodollars: Infinity }]);
assert.equal(costs.scope, "all_attempts");
assert.deepEqual(costs.estimated, { nanodollars: 5, recordedAttempts: 2 });
assert.deepEqual(costs.providerReported, { nanodollars: 7, recordedAttempts: 1 });
assert.equal(costs.attempts, 5);
assert.match(costLabel(costs), /^≥ /);
assert.equal(costLabel(sumAttemptCosts([{}, {}])), "Unknown");
assert.equal(costLabel(sumAttemptCosts([{ estimatedCostNanodollars: 0 }])), "$0.000000");
});
test("regressions and recoveries compare exact suite membership and evals SHA, not count alone", () => {
const before = campaignAnalytics(campaign(1, ["context"]));
const after = campaignAnalytics(campaign(2, ["create"]));
const change = compareCampaignAnalytics(after, before);
assert.deepEqual(change.regressions.map((cell) => cell.caseId), ["create"]);
assert.deepEqual(change.recoveries.map((cell) => cell.caseId), ["context"]);
const differentSha = campaign(2, [], { source: { evals: { sha: "c".repeat(40) } } });
assert.equal(compareCampaignAnalytics(campaignAnalytics(differentSha), before), null);
const changedMembership = campaign(2);
changedMembership.results[0].cellId = "codex--different-case";
assert.equal(compareCampaignAnalytics(campaignAnalytics(changedMembership), before), null);
const reordered = campaign(1, ["context"]);
reordered.results.reverse();
assert.equal(campaignAnalytics(reordered).suiteKey, before.suiteKey);
});
test("history analytics backfill without rewriting records and refreshes enrich original costs", async () => {
const source = campaign(1);
const record = protocolEvalHistoryRecord(source, "https://reports.example/runner-protocol-evals");
let history = mergeProtocolEvalHistory(emptyProtocolEvalHistory(), record);
history = await enrichProtocolEvalHistory(history, { currentCampaign: source, loadCampaign: () => assert.fail("current campaign already available") });
assert.equal(history.analytics[source.campaignId].costs.scope, "final_attempts");
const refresh = { ...source, campaignId: "gha-1-1-report-theme", reportRevision: { sourceCampaignId: source.campaignId, renderedAt: "2026-09-04T00:00:00.000Z" }, costs: sumAttemptCosts([...source.results.map((r) => r.usage), { estimatedCostNanodollars: 500_000 }]) };
history = mergeProtocolEvalHistory(history, protocolEvalHistoryRecord(refresh, "https://reports.example/runner-protocol-evals"));
history = await enrichProtocolEvalHistory(history, { currentCampaign: refresh, loadCampaign: () => assert.fail("refresh already available") });
assert.equal(history.analytics[source.campaignId].costs.estimated.nanodollars, 3_500_000);
assert.deepEqual(history.campaigns.find((run) => run.campaignId === source.campaignId), record);
assert.equal(history.latestCampaignId, source.campaignId);
await assert.rejects(enrichProtocolEvalHistory(history, { currentCampaign: { ...refresh, totals: { ...source.totals, passed: 0 } }, loadCampaign: () => null }), /do not match/);
});
test("history graphs exclude refreshes, link exact SHAs and Actions, and open the latest presentation", async () => {
const first = campaign(1, ["context"]);
const second = campaign(2, ["create"]);
const refresh = { ...second, campaignId: "gha-2-1-report-theme", reportRevision: { sourceCampaignId: second.campaignId, renderedAt: "2026-09-04T00:00:00.000Z" } };
let history = emptyProtocolEvalHistory();
for (const item of [first, second, refresh]) history = mergeProtocolEvalHistory(history, protocolEvalHistoryRecord(item, "https://reports.example/runner-protocol-evals"));
history = await enrichProtocolEvalHistory(history, { currentCampaign: refresh, loadCampaign: (id) => id === first.campaignId ? first : second });
const html = renderProtocolEvalHistoryIndex(history, "campaigns/gha-2-1-report-theme/viewer/assets/index.css");
assert.match(html, /All runs · 2/);
assert.match(html, /2 recorded runs/);
assert.match(html, /1 regressions · 1 recoveries/);
assert.match(html, /Full maintained suite/);
assert.ok(html.includes(`https://github.com/paperclipai/paperclip/commit/${"a".repeat(40)}`));
assert.ok(html.includes(`https://github.com/paperclipai/paperclip-evals/commit/${"b".repeat(40)}`));
assert.match(html, /actions\/runs\/2/);
assert.match(html, /href="https:\/\/reports.example\/runner-protocol-evals\/campaigns\/gha-2-1-report-theme\/">Latest · gha-2-1/);
assert.doesNotMatch(html, /<script|NaN|Infinity/);
});

17
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs
View File

@ -136,6 +136,19 @@ test("publishes only the separately sanitized Evalbook through trusted OIDC code
/Upload access-controlled canonical Evalbook and raw attempts/u,
);
assert.match(report, /Upload publisher-only sanitized Evalbook/u);
assert.match(
report,
/verify-runner-evalbook-viewer\.mjs --report-root runner-protocol-merged\/public-report/u,
);
assert.match(
report,
/verify-runner-evalbook-viewer\.mjs --report-root runner-protocol-merged\/report/u,
);
assert.match(
report,
/--viewer-root runner-protocol-build\/extracted\/dist-issue-thread\s*\\\n\s*--public-viewer/u,
);
assert.equal([...report.matchAll(/--viewer-root /gu)].length, 2);
const publisher = workflow.slice(workflow.indexOf(" publish_history:"));
assert.match(publisher, /ref: \$\{\{ github\.sha \}\}/u);
@ -143,6 +156,10 @@ test("publishes only the separately sanitized Evalbook through trusted OIDC code
assert.match(publisher, /runner-protocol-eval-public-/u);
assert.match(publisher, /publish-runner-protocol-eval-history\.mjs/u);
assert.match(publisher, /runner-protocol-evals/u);
assert.match(publisher, /runner-protocol-viewer-/u);
assert.match(publisher, /PAPERCLIP_RUNNER_PROTOCOL_EVAL_VIEWER_DIR/u);
assert.match(publisher, /url: \$\{\{ steps\.publish\.outputs\.report_url \}\}/u);
assert.match(publisher, /Publish versioned report and refresh the root index\n\s+id: publish/u);
assert.doesNotMatch(publisher, /(?:OPENAI|ANTHROPIC|OPENROUTER)_API_KEY/u);
assert.doesNotMatch(publisher, /paperclipai\/paperclip-evals/u);
assert.doesNotMatch(publisher, /downloaded-runner-protocol-evals/u);

185
packages/paperclip-runner/scripts/verify-runner-evalbook-viewer.mjs Normal file
View File

@ -0,0 +1,185 @@
#!/usr/bin/env node
// Execute the built viewer against actual generated pages, without a runner API.
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { readFile, readdir, mkdir } from "node:fs/promises";
import { resolve, join, extname, sep } from "node:path";
import { chromium, expect } from "@playwright/test";
const arg = (name) => process.argv[process.argv.indexOf(name) + 1];
if (!process.argv.includes("--report-root"))
throw new Error("--report-root is required");
const root = resolve(arg("--report-root"));
const screenshots = process.argv.includes("--screenshots")
? resolve(arg("--screenshots"))
: null;
const samples = new Map();
for (const entry of await readdir(join(root, "attempts"), {
withFileTypes: true,
})) {
if (!entry.isDirectory()) continue;
const route = `attempts/${entry.name}/index.html`;
const html = await readFile(join(root, route), "utf8");
const encoded =
html.match(
/<script type="application\/json" id="paperclip-eval-report">([^<]*)<\/script>/u,
)?.[1] ??
html.match(/window\.__PAPERCLIP_EVAL_REPORT__=(.*?);<\/script>/su)?.[1];
assert.ok(encoded, `Attempt lacks canonical viewer payload: ${route}`);
const payload = JSON.parse(encoded);
const messages = payload.view.turns
.flatMap((turn) => turn.items)
.filter((item) => ["user_message", "agent_message"].includes(item.kind));
const kind = !messages.length
? "missing-recording"
: payload.passed
? "passed"
: "failed";
if (!samples.has(kind)) samples.set(kind, { route, payload });
}
assert.ok(samples.size, "No attempt pages to verify");
const server = createServer(async (req, res) => {
try {
const url = new URL(req.url, "http://report.invalid");
const path = resolve(root, `.${decodeURIComponent(url.pathname)}`);
if (!path.startsWith(`${root}${sep}`)) {
res.writeHead(403).end();
return;
}
const mime = {
".html": "text/html",
".js": "text/javascript",
".css": "text/css",
".woff2": "font/woff2",
};
res.writeHead(200, {
"Content-Type": mime[extname(path)] ?? "application/octet-stream",
});
res.end(await readFile(path));
} catch {
res.writeHead(404).end();
}
});
await new Promise((done) => server.listen(0, "127.0.0.1", done));
const origin = `http://127.0.0.1:${server.address().port}`;
let browser;
try {
browser = await chromium.launch({ headless: true });
if (screenshots) await mkdir(screenshots, { recursive: true });
for (const [kind, { route, payload }] of samples) {
const page = await browser.newPage({
viewport: { width: 1440, height: 1000 },
});
const errors = [];
const requests = [];
page.on("pageerror", (error) => errors.push(error.message));
page.on("request", (request) => requests.push(request.url()));
await page.goto(`${origin}/${route}`, { waitUntil: "networkidle" });
await page.locator(".pit-thread").waitFor({ state: "visible" });
const chatBackground = await page.evaluate(() => getComputedStyle(document.body).backgroundColor);
assert.equal(
await page.locator("h1").textContent(),
payload.view.issue.title,
);
assert.equal(
await page
.getByRole("button", { name: "Reset scenario", exact: true })
.count(),
0,
);
assert.equal(
await page.getByText("Loading company state…", { exact: true }).count(),
0,
);
assert.ok(
await page.getByText(payload.run.model, { exact: false }).count(),
"Model missing from inspector",
);
const tool = page
.locator('[data-thread-item="tool_activity"] summary')
.first();
if (await tool.count()) {
await tool.click();
assert.ok(
await page.locator('[data-thread-item="tool_activity"][open]').count(),
);
if (payload.publication)
assert.ok(
await page
.getByText("Arguments withheld from public replay.", {
exact: false,
})
.count(),
);
if (payload.devtools || payload.publication) {
const evidenceLink = page
.locator('[data-thread-item="tool_activity"][open]')
.first()
.getByRole("button", { name: "View in Evidence", exact: true });
const evidenceTab = page.getByRole("tab", { name: "Evidence", exact: true });
await evidenceLink.click();
await expect(evidenceTab).toHaveAttribute("aria-selected", "true");
await expect(page.locator('[data-highlighted="true"]').first()).toBeVisible();
await expect(page.getByRole("heading", { name: "Assertions", exact: true })).toHaveCount(0);
// Reopening the same record after selecting another tab must work too.
await page.getByRole("tab", { name: "Eval", exact: true }).click();
await evidenceLink.click();
await expect(evidenceTab).toHaveAttribute("aria-selected", "true");
await page.getByRole("tab", { name: "Eval", exact: true }).click();
}
}
if (payload.publication) {
assert.ok(
await page
.getByText(payload.publication.notice, { exact: true })
.count(),
);
assert.ok(
requests.every(
(url) => url.startsWith(origin) && !url.includes("/api/"),
),
"Replay attempted a remote/API request",
);
}
assert.deepEqual(errors, [], `Viewer errors for ${kind}`);
if (screenshots)
await page.screenshot({
path: join(screenshots, `${kind}.png`),
fullPage: true,
});
await page
.getByRole("link", { name: "← All results", exact: true })
.click();
assert.equal(new URL(page.url()).pathname, "/index.html");
await expect(page.locator("body")).toHaveClass("evalbook-site");
const assertSiteTheme = async () => {
assert.equal(await page.evaluate(() => getComputedStyle(document.body).backgroundColor), chatBackground);
assert.equal(await page.evaluate(() => getComputedStyle(document.documentElement).colorScheme), "dark");
};
await assertSiteTheme();
if (screenshots) await page.screenshot({ path: join(screenshots, "index.png"), fullPage: true });
await page.locator(".matrix tbody th a").first().click();
await expect(page.getByText("Test design", { exact: true })).toBeVisible();
await assertSiteTheme();
await page.getByText("Full authored test definition", { exact: true }).click();
if (screenshots) await page.screenshot({ path: join(screenshots, "test-design.png"), fullPage: true });
await page.getByRole("link", { name: "← All results", exact: true }).click();
await page.getByRole("link", { name: "Latest", exact: true }).click();
await assertSiteTheme();
await page.getByRole("link", { name: "Overview", exact: true }).click();
await assertSiteTheme();
await page.goto(`${origin}/${route}`, { waitUntil: "networkidle" });
await page.reload({ waitUntil: "networkidle" });
await page.locator(".pit-thread").waitFor({ state: "visible" });
assert.deepEqual(errors, [], `Reload errors for ${kind}`);
await page.setViewportSize({ width: 390, height: 844 });
await page.locator(".pit-thread").waitFor({ state: "visible" });
await page.close();
console.log(
`Verified ${kind}: chat, read-only controls, navigation, reload and narrow viewport`,
);
}
} finally {
await browser?.close();
await new Promise((done) => server.close(done));
}