From d2e940f4c1a24bc39485ba7dac1227dd9ecee5b0 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 12 Sep 2026 11:33:20 -0700 Subject: [PATCH 1/4] ci: run release Runner protocol and Rust checks in parallel (#13326) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud deploys verified images from merged source commits. > - Cloud readiness waits for every release verification check. > - Runner verification currently runs long TypeScript tests before Rust checks. > - These checks can run on independent runners with their own build directories. > - This PR runs them in parallel while preserving all checks and the shared dependency cache. ## Linked Issues or Issue Description Refs #13194. Related prior work: #13142 and #13259. A search found no duplicate parallel release-check change. **What existing behavior does this improve?** Time from merge to Cloud source verification and deployment readiness. **Current behavior** Recent successful runs take roughly 13 minutes from merge to deployable. In run 34705914878, Runner verification took 11m23s. Protocol tests finished before Rust tests and API authority checks started. **Proposed behavior** Run protocol and Rust verification in two matrix jobs. Cloud readiness still requires both jobs to pass. **Reason and benefit** Remove the serial dependency between independent checks. Expected improvement is about 2โ€“3 minutes on a typical cached run, until the image build or server tests become the longest job. This is an estimate; post-merge timing will confirm it. **Breaking changes** Individual release Runner job names gain a lane suffix. Cloud source and readiness marker names stay the same. PR runner routing is unchanged. ## What Changed - Split release Runner checks into protocol and Rust lanes. Keep every constituent of `check:all` exactly once. - Restore the existing Rust dependency cache in both lanes. Allow only the Rust lane to save it after warming both build profiles. - Add coverage and cache authorization regressions. Document the parallel verification and single cache writer. ## Verification - Passed 477 workflow and source-verification tests with `node --test .github/scripts/tests/*.test.mjs scripts/cloud-source-verification.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Passed `actionlint`, `git diff --check`, and the private AWS routing regression suite. - Passed local `pnpm -r typecheck` and the standalone `check:runner && check:api-authority` lane, including all 1,671 API tests before the protocol lane had built TypeScript output. - The broad local protocol run under Node 25 had four failures. The two affected files passed under CI's Node 24.19.0: 67 passed, 6 platform skips. - Local `pnpm test:run` aborted when disk space ran out; local `pnpm build` could not run afterward. These are local verification limits. [Linux CI run 34710421424](https://github.com/paperclipai/paperclip/actions/runs/34710421424) passed full typecheck, all grouped tests, native verification, build, release dry run, and browser checks. Native protocol CI passed 1,986 tests, plus 1,671 API tests and the Rust suites. - Latest-head Greptile is 5/5 with no open findings. All 33 current-head checks are successful or intentionally skipped. ## Risks - Uses one additional short-lived verification runner per release verification. The existing AWS exact-master restriction remains in place. - The Rust lane warms debug dependencies so its cache save also serves protocol tests. Both lanes always rebuild workspace code. - A workflow regression could omit a check. The new coverage test compares the matrix checks directly with `check:all`; Cloud readiness depends on the complete reusable workflow. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../tests/release-runner-cache.test.mjs | 47 ++++++++++++++++--- .github/workflows/release-verify.yml | 31 ++++++++++-- doc/RELEASE-AUTOMATION-SETUP.md | 32 ++++++++----- .../release-verify-workflow.test.mjs | 10 ++-- 4 files changed, 92 insertions(+), 28 deletions(-) diff --git a/.github/scripts/tests/release-runner-cache.test.mjs b/.github/scripts/tests/release-runner-cache.test.mjs index 9e4b8fe715..7e35ec09c9 100644 --- a/.github/scripts/tests/release-runner-cache.test.mjs +++ b/.github/scripts/tests/release-runner-cache.test.mjs @@ -1,6 +1,7 @@ import test from "node:test"; import assert from "node:assert/strict"; import { readFileSync } from "node:fs"; +import { runInNewContext } from "node:vm"; const workflow = readFileSync(new URL("../../workflows/release-verify.yml", import.meta.url), "utf8"); const runner = workflow.split(" verify_paperclip_runner:")[1].split(" build:")[0]; @@ -22,16 +23,48 @@ test("the shared cache excludes workspace artifacts and only restores or saves t assert.match(runner, /cache-workspace-crates: false/); assert.match(runner, /cache-bin: false/); const saveIf = runner.match(/^\s*save-if: (.+)$/m)?.[1]; - assert.equal(saveIf, "${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}"); + assert.equal(saveIf, "${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }}"); const cacheStep = runner.split(" - name: Cache Runner Rust dependencies")[1].split(" - name: Install dependencies")[0]; - assert.equal(cacheStep.match(/^\s*if: (.+)$/m)?.[1], saveIf); + assert.equal(cacheStep.match(/^\s*if: (.+)$/m)?.[1], saveIf.replace("matrix.lane == 'rust' && ", "")); assert.doesNotMatch(runner, /cache-on-failure: true|cache-all-crates: true/); }); -test("cache hits cannot bypass Runner verification", () => { - const verify = runner.split(" - name: Verify Paperclip Runner")[1]; - assert.match(verify, /run: pnpm --filter @paperclipai\/paperclip-runner check:all/); - assert.doesNotMatch(verify, /if:|continue-on-error:/); - assert.ok(runner.indexOf("Cache Runner Rust dependencies") < runner.indexOf(" - name: Verify Paperclip Runner\n")); +test("parallel lanes cover check:all exactly once and never bypass verification", () => { + const scripts = JSON.parse(readFileSync(new URL("../../../packages/paperclip-runner/package.json", import.meta.url))).scripts; + const checks = [...runner.matchAll(/^ checks: (.+)$/gm)].flatMap(([, value]) => value.split(" ")); + assert.deepEqual(checks, scripts["check:all"].split(" && ").map((command) => command.replace(/^pnpm run /, ""))); + assert.deepEqual([...runner.matchAll(/^ - lane: (.+)$/gm)].map(([, value]) => value), ["protocol", "rust"]); + assert.match(runner, /fail-fast: false/); + assert.doesNotMatch(runner, /max-parallel: 1|^ needs:|continue-on-error:/m); + const verify = runner.split(" - name: Verify Paperclip Runner\n")[1].split(" - name: Warm debug")[0]; + assert.match(verify, /RUNNER_CHECKS: \$\{\{ matrix.checks \}\}/); + assert.match(verify, /set -euo pipefail/); + assert.match(verify, /for check in \$RUNNER_CHECKS; do\s+pnpm --filter @paperclipai\/paperclip-runner "\$check"\s+done/); + assert.doesNotMatch(verify, /if:|cache-hit/); assert.doesNotMatch(runner, /id-token: write|packages: write|secrets: inherit/); }); + +test("only the trusted Rust lane writes, and warms both build profiles before saving", () => { + const cache = runner.split(" - name: Cache Runner Rust dependencies")[1].split(" - name: Install dependencies")[0]; + const warm = runner.split(" - name: Warm debug dependencies for the shared Runner cache")[1]; + const expr = (body, field) => body.match(new RegExp(`^ +${field}: \\$\\{\\{ (.+) \\}\\}$`, "m"))[1]; + assert.equal(expr(cache, "save-if"), expr(warm, "if")); + assert.match(warm, /run: pnpm --filter @paperclipai\/paperclip-runner build:rust/); + const sha = "a".repeat(40); + const base = { repository: "paperclipai/paperclip", event_name: "push", ref: "refs/heads/master", sha }; + for (const lane of ["protocol", "rust"]) { + for (const [overrides, ref, trusted] of [ + [{}, sha, true], + [{ event_name: "pull_request", ref: "refs/pull/1/merge" }, sha, false], + [{ event_name: "pull_request_target" }, sha, false], + [{ event_name: "workflow_dispatch" }, sha, false], + [{ repository: "someone/paperclip" }, sha, false], + [{ ref: "refs/heads/feature" }, sha, false], + [{}, "b".repeat(40), false], + ]) { + const context = { matrix: { lane }, github: { ...base, ...overrides }, inputs: { ref } }; + assert.equal(runInNewContext(expr(cache, "if"), context), trusted); + assert.equal(runInNewContext(expr(cache, "save-if"), context), trusted && lane === "rust"); + } + } +}); diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 442d198f20..505624a6b0 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -262,12 +262,21 @@ jobs: run: pnpm test:runner-workflow-evals verify_paperclip_runner: - name: Verify Paperclip Runner + name: Verify Paperclip Runner (${{ matrix.lane }}) runs-on: ${{ vars.AWS_POST_MERGE_CI_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha != '' && inputs.ref == github.sha && 'runs-on/fleet=paperclip-post-merge-x64/env=public-ci' || 'ubuntu-latest' }} timeout-minutes: 20 permissions: contents: read + strategy: + fail-fast: false + matrix: + include: + - lane: protocol + checks: check:eval-kernel check:protocol + - lane: rust + checks: check:runner check:api-authority + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 @@ -305,15 +314,27 @@ jobs: # dependencies; never restore installed executables from cargo/bin. cache-workspace-crates: false cache-bin: false - # The step guard also restricts restores. Save only after a successful - # master-push verification of that push's exact commit. - save-if: ${{ github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} + # Both lanes restore the existing dependency cache. Only the Rust + # lane saves it, after warming both release and debug dependencies. + save-if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} - name: Install dependencies run: pnpm install --no-frozen-lockfile - name: Verify Paperclip Runner - run: pnpm --filter @paperclipai/paperclip-runner check:all + env: + RUNNER_CHECKS: ${{ matrix.checks }} + run: | + set -euo pipefail + for check in $RUNNER_CHECKS; do + pnpm --filter @paperclipai/paperclip-runner "$check" + done + + - name: Warm debug dependencies for the shared Runner cache + # Protocol tests need debug binaries. Populate their dependencies in + # the sole cache writer, so a cold save also serves the protocol lane. + if: ${{ matrix.lane == 'rust' && github.repository == 'paperclipai/paperclip' && github.event_name == 'push' && github.ref == 'refs/heads/master' && inputs.ref == github.sha }} + run: pnpm --filter @paperclipai/paperclip-runner build:rust build: name: Build diff --git a/doc/RELEASE-AUTOMATION-SETUP.md b/doc/RELEASE-AUTOMATION-SETUP.md index 43b6fee7bd..4817e5d7fc 100644 --- a/doc/RELEASE-AUTOMATION-SETUP.md +++ b/doc/RELEASE-AUTOMATION-SETUP.md @@ -338,19 +338,27 @@ Check: ## Runner verification dependency cache -`release-verify.yml` caches Cargo dependencies for its `Verify Paperclip Runner` -job using a pinned Rust Cache action. It selects the compiler from the Runner -package's `rust-toolchain.toml` before computing the cache key. Compiler and Cargo -metadata changes select a new cache; the `release-runner-v1` shared key lets -callers of this reusable verification workflow reuse the same dependency cache. +`release-verify.yml` runs `Verify Paperclip Runner` on two independent runners. +The protocol lane runs `check:eval-kernel` and `check:protocol`. The Rust lane +runs `check:runner` and `check:api-authority`. Together they retain every check +in `check:all`; both lanes must pass before Cloud source verification or +readiness can succeed. A failed lane does not cancel the other lane. -Workspace crates and installed Cargo binaries are excluded. Every run still -builds the Runner workspace and runs `check:all`, including the Rust and -TypeScript tests. Only an own-repository master-push run verifying that push's exact -SHA can restore the cache, and only a successful run saves it. PR, tag, and -manual candidate verification compile without this cache. A miss or eviction costs compilation time but does not change the checks. -To discard old dependency caches, increment the shared-key version and let the -next successful master verification warm it again. +Both lanes restore Cargo dependencies with the pinned Rust Cache action. The +compiler comes from the Runner package's `rust-toolchain.toml` before the action +computes its key. Compiler and Cargo metadata changes select a new cache. The +existing `release-runner-v1` shared key avoids separate copies for these lanes. +Only the Rust lane saves this cache. After verification it also runs `build:rust` +to warm the debug dependencies used by the protocol lane; its own tests already +warm release dependencies. The cache writer is shorter than the protocol lane. + +Workspace crates and installed Cargo binaries are excluded. Every run rebuilds +workspace code and runs all assigned checks, including on a cache hit. Only an +own-repository master-push run verifying that push's exact SHA can restore the +cache, and only a successful Rust lane saves it. PR, tag, and manual candidate +verification compile without this cache. A miss or eviction costs compilation +time but does not change the checks. To discard old dependency caches, increment +the shared-key version and let the next successful master verification warm it. The trust boundary is the protected master branch, not the cache-key text. GitHub does not let master restore caches created by a child branch, sibling diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 702edcab8f..d93b4bf5da 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -221,10 +221,12 @@ test("release verify workflow covers the same split test surface as stable PR ve ); assert.match(verifyWorkflow, /pnpm -r typecheck/); assert.match(verifyWorkflow, /pnpm build/); - assert.match( - verifyWorkflow, - /pnpm --filter @paperclipai\/paperclip-runner check:all/, - ); + const runnerScripts = JSON.parse(readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8")).scripts; + const runnerChecks = [...verifyWorkflow.matchAll(/^ checks: (.+)$/gm)] + .flatMap(([, checks]) => checks.split(" ")); + assert.deepEqual(runnerChecks, runnerScripts["check:all"].split(" && ") + .map((command) => command.replace(/^pnpm run /, ""))); + assert.match(verifyWorkflow, /pnpm --filter @paperclipai\/paperclip-runner "\$check"/); assert.match(verifyWorkflow, /runner_workflow_evals:/); assert.match(verifyWorkflow, /runner_chaos_evals:/); assert.match( From ed50a39c3f35abcd29b31e536a960a12d6ddb48e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 12 Sep 2026 13:34:45 -0500 Subject: [PATCH 2/4] fix: preserve NUL characters in run-event payloads (#13325) Co-Authored-By: Paperclip --- doc/run-log-events.md | 11 ++++ packages/db/src/run-event-payload.test.ts | 47 +++++++++++++++ packages/db/src/run-event-payload.ts | 60 +++++++++++++++++++ .../db/src/schema/heartbeat_run_events.ts | 4 +- .../runner-prp-coordinator.test.ts | 58 +++++++++++++++++- 5 files changed, 176 insertions(+), 4 deletions(-) create mode 100644 packages/db/src/run-event-payload.test.ts create mode 100644 packages/db/src/run-event-payload.ts diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 2e069de502..d6f98564e5 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -13,6 +13,17 @@ the PRP `eventType`, source instance, source event ID, source sequence, protocol schema version, and a SHA-256 digest of the canonical source envelope. Its payload is `{ "prpEvent": }`. +PostgreSQL JSONB cannot represent NUL (U+0000), which can occur in command +output such as Vite virtual-module paths. The run-event payload column uses a +lossless storage codec for these events: the JSONB projection renders NUL as +the literal `\u0000`, and the reserved `$paperclipRunEventJsonV1` field contains +the original serialized JSON as a doubly escaped string. Ordinary payloads +retain their existing representation. Drizzle reads restore the exact original +payload before replay, hash validation, redaction, or API presentation. SQL +queries can still inspect ordinary routing fields in the projection; raw SQL +readers of the whole payload must apply `decodeRunEventPayload`. The column +remains JSONB and requires no schema migration. + The writer locks the native `heartbeat_runs` row and allocates the existing per-run `seq` cursor. A byte-equivalent retry reuses the first row; a changed retry or source-sequence gap is rejected. Company, issue, agent, run, session, diff --git a/packages/db/src/run-event-payload.test.ts b/packages/db/src/run-event-payload.test.ts new file mode 100644 index 0000000000..f577e21647 --- /dev/null +++ b/packages/db/src/run-event-payload.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from "vitest"; +import { decodeRunEventPayload, encodeRunEventPayload } from "./run-event-payload.js"; + +describe("run-event JSONB payload codec", () => { + it("leaves ordinary payloads and literal escape sequences unchanged", () => { + const payload = { prpEvent: { payload: { output: "virtual:\\u0000file.js", emoji: "๐Ÿงช" } } }; + const encoded = encodeRunEventPayload(payload); + expect(encoded).toBe(JSON.stringify(payload)); + expect(decodeRunEventPayload(encoded)).toEqual(payload); + expect(decodeRunEventPayload(payload)).toBe(payload); + }); + + it("round-trips NULs in nested strings, arrays and keys without leaking its sidecar", () => { + const payload = { + prpEvent: { + sourceKind: "runner", + payload: { output: "../\u0000virtual:/file.js", items: [null, true, 1, "\u0000"] }, + }, + "\u0000key": "a", + "\\u0000key": "b", + }; + const encoded = encodeRunEventPayload(payload); + const stored = JSON.parse(encoded); + expect(stored.prpEvent.sourceKind).toBe("runner"); + expect(stored.prpEvent.payload.output).toBe("../\\u0000virtual:/file.js"); + // No actual NUL survives anywhere in the object sent to PostgreSQL. + function assertNoNul(value: unknown): void { + if (typeof value === "string") expect(value).not.toContain("\u0000"); + if (value !== null && typeof value === "object") { + for (const [key, entry] of Object.entries(value)) { + expect(key).not.toContain("\u0000"); + assertNoNul(entry); + } + } + } + assertNoNul(stored); + expect(decodeRunEventPayload(encoded)).toEqual(payload); + expect(decodeRunEventPayload(stored)).toEqual(payload); + }); + + it("preserves caller-owned keys that collide with the storage marker", () => { + for (const value of ["not JSON", '{"forged":true}', null, { nested: "\u0000" }]) { + const payload = { $paperclipRunEventJsonV1: value, output: "original" }; + expect(decodeRunEventPayload(encodeRunEventPayload(payload))).toEqual(payload); + } + }); +}); diff --git a/packages/db/src/run-event-payload.ts b/packages/db/src/run-event-payload.ts new file mode 100644 index 0000000000..ee945c47e7 --- /dev/null +++ b/packages/db/src/run-event-payload.ts @@ -0,0 +1,60 @@ +import { customType } from "drizzle-orm/pg-core"; + +// Reserved only in the on-disk representation, never in a decoded event. +const originalJsonKey = "$paperclipRunEventJsonV1"; + +/** Keep JSONB routing fields queryable while retaining JSON strings containing NUL. */ +export function encodeRunEventPayload(payload: Record): string { + const originalJson = JSON.stringify(payload); + if (!originalJson.includes("\\u0000") && !originalJson.includes(originalJsonKey)) { + return originalJson; + } + + const original = JSON.parse(originalJson) as Record; + let needsEncoding = Object.hasOwn(original, originalJsonKey); + function projectString(value: string): string { + if (!value.includes("\u0000")) return value; + needsEncoding = true; + return value.replaceAll("\u0000", "\\u0000"); + } + function project(value: unknown): unknown { + if (typeof value === "string") return projectString(value); + if (Array.isArray(value)) return value.map(project); + if (value !== null && typeof value === "object") { + return Object.fromEntries(Object.entries(value).map(([key, entry]) => [ + projectString(key), project(entry), + ])); + } + return value; + } + + const projection = project(original) as Record; + if (!needsEncoding) return originalJson; + // JSON.stringify escapes the original JSON a second time: PostgreSQL receives + // literal backslashes, not an unsupported U+0000. Decode before hashing/replay. + return JSON.stringify({ ...projection, [originalJsonKey]: originalJson }); +} + +export function decodeRunEventPayload(value: string | Record): Record { + const payload = typeof value === "string" ? JSON.parse(value) as Record : value; + if (Object.hasOwn(payload, originalJsonKey)) { + const originalJson = payload[originalJsonKey]; + if (typeof originalJson !== "string") throw new Error("Invalid run-event payload encoding"); + const original: unknown = JSON.parse(originalJson); + if (original === null || typeof original !== "object" || Array.isArray(original)) { + throw new Error("Invalid run-event payload encoding"); + } + return original as Record; + } + return payload; +} + +// The SQL type stays JSONB; existing rows and SQL routing queries are unchanged. +export const runEventPayload = customType<{ + data: Record; + driverData: string; +}>({ + dataType: () => "jsonb", + toDriver: encodeRunEventPayload, + fromDriver: decodeRunEventPayload, +}); diff --git a/packages/db/src/schema/heartbeat_run_events.ts b/packages/db/src/schema/heartbeat_run_events.ts index 2bbab19c54..71da8bd3d9 100644 --- a/packages/db/src/schema/heartbeat_run_events.ts +++ b/packages/db/src/schema/heartbeat_run_events.ts @@ -5,7 +5,6 @@ import { text, timestamp, integer, - jsonb, index, bigserial, bigint, @@ -14,6 +13,7 @@ import { import { companies } from "./companies.js"; import { agents } from "./agents.js"; import { heartbeatRuns } from "./heartbeat_runs.js"; +import { runEventPayload } from "../run-event-payload.js"; export const heartbeatRunEvents = pgTable( "heartbeat_run_events", @@ -28,7 +28,7 @@ export const heartbeatRunEvents = pgTable( level: text("level"), color: text("color"), message: text("message"), - payload: jsonb("payload").$type>(), + payload: runEventPayload("payload"), sourceInstanceId: text("source_instance_id"), sourceEventId: text("source_event_id"), sourceSeq: bigint("source_seq", { mode: "number" }), diff --git a/server/src/services/native-runtime/runner-prp-coordinator.test.ts b/server/src/services/native-runtime/runner-prp-coordinator.test.ts index fd9f58c642..34b3050540 100644 --- a/server/src/services/native-runtime/runner-prp-coordinator.test.ts +++ b/server/src/services/native-runtime/runner-prp-coordinator.test.ts @@ -1,10 +1,10 @@ -import { randomUUID } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { mkdtempSync, rmSync } from "node:fs"; import { createServer } from "node:http"; import { tmpdir } from "node:os"; import { resolve } from "node:path"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { @@ -36,6 +36,7 @@ import { import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js"; import { runnerPrpCoordinator } from "./runner-prp-coordinator.js"; import { PaperclipRunnerSemanticAuthority } from "./runner-semantic-authority.js"; +import { nativeSha256 } from "./canonical.js"; const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported @@ -339,6 +340,59 @@ describeEmbeddedPostgres("hidden runner PRP coordinator", () => { }); }); + it("persists NUL-containing command output without changing replay identity", async () => { + const seed = await seedNativeRun(); + const nativeStore = store(seed); + const output = "transforming (6) ../\u0000virtual:/@storybook/builder-vite/storybook-stories.js"; + const payload = { + schema: "paperclip.tool.execution.v1", + executionId: "storybook-build", + transport: "process", + operation: "execute", + status: "completed", + output, + outputBytes: Buffer.byteLength(output), + outputTruncated: false, + outputDigest: `sha256:${createHash("sha256").update(output).digest("hex")}`, + exitCode: 0, + }; + const event: PrpEvent = { + ...runnerEvent(seed), + eventType: "tool.execution.completed", + payload, + }; + + // The provider's valid JSON cannot be inserted directly into PostgreSQL JSONB. + await expect(db.execute(sql`select ${JSON.stringify(event)}::jsonb`)) + .rejects.toMatchObject({ cause: { code: "22P05" } }); + await expect(nativeStore.appendEvent(event)).resolves.toMatchObject({ + disposition: "committed", + cursor: 1, + }); + const [row] = await db.select().from(heartbeatRunEvents) + .where(eq(heartbeatRunEvents.runId, seed.runId)); + expect(row.payload).toEqual({ prpEvent: event }); + expect(row.sourcePayloadSha256).toBe(`sha256:${nativeSha256(row.payload?.prpEvent)}`); + + // Existing SQL selectors still see the event's ordinary routing fields. + const [projection] = await db.select({ + sourceKind: sql`${heartbeatRunEvents.payload}->'prpEvent'->>'sourceKind'`, + }).from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, seed.runId)); + expect(projection.sourceKind).toBe("runner"); + await expect(nativeStore.appendEvent(event)).resolves.toMatchObject({ + disposition: "duplicate", + cursor: 1, + }); + await expect(nativeStore.appendEvent({ + ...event, + payload: { ...payload, output: output.replaceAll("\u0000", "\\u0000") }, + })).rejects.toBeInstanceOf(NativeSessionProtocolIntegrityError); + await expect(nativeStore.appendEvent(runnerEvent(seed, 2))).resolves.toMatchObject({ + disposition: "committed", + cursor: 2, + }); + }); + it("persists events and results idempotently and leases finalization", async () => { const seed = await seedNativeRun(); const nativeStore = store(seed); From 8df3ee2cf5e903dafab3b1a575c3ef13094e5aef Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 12 Sep 2026 11:36:20 -0700 Subject: [PATCH 3/4] ci: rebalance serialized tests with current suite durations (#13328) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud deployments wait for verified source commits. > - Verification splits serialized server tests across independent runners. > - The shard duration estimates came from August and no longer match current tests. > - Stale estimates put much more work on one runner than the others. > - This PR refreshes the estimates from a complete successful run to balance the existing runners. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time spent waiting for the slowest serialized server-test shard in PR and release verification. **Current behavior** In [Cloud readiness run 34705914878](https://github.com/paperclipai/paperclip/actions/runs/34705914878), the five serialized shards spent 479, 371, 322, 390, and 365 seconds running tests. The recovery suite had a 55-second estimate but now takes about 156 seconds including process overhead. **Proposed behavior** Use fresh per-suite measurements with the existing deterministic duration balancer. Applying the same measured costs to the new assignment gives 385, 385, 386, 385, and 385 seconds. This predicts about 93 seconds less waiting for the slowest shard, before runner/setup overhead. Live CI will confirm the result. **Reason and benefit** Use the existing runners more evenly. No extra runner, test parallelism, cache, timeout, or routing change is needed. **Breaking changes** Suite-to-shard assignments change. The full suite set, assertions, and per-suite process isolation stay the same. **Additional context** Searched related CI and shard PRs. This updates the existing duration manifest, without duplicating a pending sharding implementation. ## What Changed - Refresh all 145 serialized suite weights from the same successful release verification run. - Record source job IDs and the measurement method in the manifest. Durations include process startup, imports, collection, tests, and shutdown. ## Verification - Passed all 30 shard and release-workflow tests: `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`. - Confirmed every measured suite appears exactly once across all five source logs. - Compared old and new assignments using the same measured weights. The maximum fell from 478862ms to 385551ms. - In [PR CI run 34710696242](https://github.com/paperclipai/paperclip/actions/runs/34710696242), all five serialized jobs passed in 7m05sโ€“7m27s including setup. The measured assignment is now balanced in a live run. - The same run passed full typecheck, all grouped tests, native verification, build, release dry run, and browser checks. Local full-suite verification on this base was limited by disk exhaustion; local typecheck and targeted shard tests passed. - Latest-head Greptile is 5/5 with no open findings. Every current-head CI check must be green or intentionally skipped before merge. ## Risks - Individual durations vary with load and future test changes. These estimates affect assignment only; missing or renamed suites receive the existing median weight. - Both PR and release verification read this manifest, so both receive the new assignments. Each suite still runs in its own serialized Vitest process. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- scripts/serialized-shard-durations.json | 281 ++++++++++++------------ 1 file changed, 146 insertions(+), 135 deletions(-) diff --git a/scripts/serialized-shard-durations.json b/scripts/serialized-shard-durations.json index 4e6fe71ee7..477e9acfcd 100644 --- a/scripts/serialized-shard-durations.json +++ b/scripts/serialized-shard-durations.json @@ -1,140 +1,151 @@ { - "$comment": "Per-suite wall-clock durations (ms) for the serialized route/authz vitest lane, used by scripts/run-vitest-stable.mjs to balance suites across the PR shard matrix. Sampled from a real PR run of .github/workflows/pr.yml (actions run 32012408876, 2026-08-17) by diffing consecutive '[test:run] ' label timestamps in the 'Run serialized server test shard' logs - that captures each suite's true serial cost including the per-suite vitest spawn overhead, not just the vitest-reported test time. Suites missing here get the median weight, so the manifest only needs occasional refreshes.", + "$comment": "Per-suite wall-clock durations (ms) for serialized route/authz verification, used by scripts/run-vitest-stable.mjs in PR and release shards. Refreshed from successful Cloud readiness run 34705914878 (2026-09-12), jobs 103585803684, 103585803688, 103585803716, 103585803734, and 103585803754. Each weight spans consecutive [test:run] suite-start timestamps; the last suite ends at the first post-job cleanup timestamp. This includes each isolated Vitest process startup, import, collection, tests, and shutdown. All 145 suites are sampled exactly once. New or renamed suites use the median weight. Refresh from a successful complete run when observed shard times drift.", "unit": "ms", "durations": { - "server/src/__tests__/access-routes-permissions-upgrade.test.ts": 11105, - "server/src/__tests__/activity-routes.test.ts": 3181, - "server/src/__tests__/adapter-model-refresh-routes.test.ts": 3934, - "server/src/__tests__/adapter-routes-authz.test.ts": 4391, - "server/src/__tests__/adapter-routes.test.ts": 5343, - "server/src/__tests__/agent-action-audit-routes.test.ts": 11458, - "server/src/__tests__/agent-adapter-validation-routes.test.ts": 5758, - "server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 3818, - "server/src/__tests__/agent-device-login-routes.test.ts": 6486, - "server/src/__tests__/agent-instructions-routes.test.ts": 5532, - "server/src/__tests__/agent-live-run-routes.test.ts": 5498, - "server/src/__tests__/agent-permissions-routes.test.ts": 8978, - "server/src/__tests__/agent-secrets-routes.test.ts": 11780, - "server/src/__tests__/agent-skills-routes.test.ts": 7404, - "server/src/__tests__/agent-test-environment-routes.test.ts": 5845, - "server/src/__tests__/approval-routes-idempotency.test.ts": 6142, - "server/src/__tests__/assets.test.ts": 3555, - "server/src/__tests__/auth-routes.test.ts": 2094, - "server/src/__tests__/auth-session-route.test.ts": 3070, - "server/src/__tests__/authz-company-access.test.ts": 2893, - "server/src/__tests__/authz-existence-oracle-guard.test.ts": 1121, - "server/src/__tests__/authz-secret-context.test.ts": 2398, - "server/src/__tests__/board-chat-route-feature-flag.test.ts": 1032, - "server/src/__tests__/bootstrap-claim-routes.test.ts": 4748, - "server/src/__tests__/built-in-agent-routes.test.ts": 4015, - "server/src/__tests__/cases-routes.test.ts": 12174, - "server/src/__tests__/cli-auth-routes.test.ts": 5228, - "server/src/__tests__/cloud-routes.test.ts": 2146, - "server/src/__tests__/companies-route-cross-company-authz.test.ts": 4314, - "server/src/__tests__/companies-route-path-guard.test.ts": 3004, - "server/src/__tests__/company-branding-route.test.ts": 3920, - "server/src/__tests__/company-import-transfer-routes.test.ts": 8905, - "server/src/__tests__/company-portability-routes.test.ts": 3034, - "server/src/__tests__/company-portability.test.ts": 4800, - "server/src/__tests__/company-search-extract-routes.test.ts": 7621, - "server/src/__tests__/company-search-rate-limit-routes.test.ts": 7587, - "server/src/__tests__/company-skill-policy-routes.test.ts": 6894, - "server/src/__tests__/company-skills-import-authz-routes.test.ts": 12068, - "server/src/__tests__/company-skills-routes.test.ts": 7484, - "server/src/__tests__/company-user-directory-route.test.ts": 4728, - "server/src/__tests__/costs-service.test.ts": 8319, - "server/src/__tests__/decision-queues-routes.test.ts": 6956, - "server/src/__tests__/document-annotation-routes.test.ts": 4375, - "server/src/__tests__/environment-custom-image-routes.test.ts": 4387, - "server/src/__tests__/environment-instance-routes.test.ts": 4527, - "server/src/__tests__/environment-routes.test.ts": 4780, - "server/src/__tests__/environment-selection-route-guards.test.ts": 4208, - "server/src/__tests__/execution-workspaces-routes.test.ts": 3450, - "server/src/__tests__/express5-auth-wildcard.test.ts": 1105, - "server/src/__tests__/external-object-routes.test.ts": 6231, - "server/src/__tests__/folders-routes.test.ts": 2802, - "server/src/__tests__/health-dev-server-token.test.ts": 2541, - "server/src/__tests__/health.test.ts": 2090, - "server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 14289, - "server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 13747, - "server/src/__tests__/heartbeat-process-recovery.test.ts": 55336, - "server/src/__tests__/inbox-agent-policy-routes.test.ts": 10057, - "server/src/__tests__/inbox-archive-routes.test.ts": 9644, - "server/src/__tests__/instance-database-backups-routes.test.ts": 2962, - "server/src/__tests__/instance-settings-routes.test.ts": 5356, - "server/src/__tests__/invite-accept-existing-member.test.ts": 4812, - "server/src/__tests__/invite-accept-gateway-defaults.test.ts": 10423, - "server/src/__tests__/invite-accept-replay.test.ts": 5116, - "server/src/__tests__/invite-create-route.test.ts": 4640, - "server/src/__tests__/invite-expiry.test.ts": 7291, - "server/src/__tests__/invite-join-manager.test.ts": 7124, - "server/src/__tests__/invite-list-route.test.ts": 7580, - "server/src/__tests__/invite-logo-route.test.ts": 5104, - "server/src/__tests__/invite-onboarding-text.test.ts": 7095, - "server/src/__tests__/invite-rate-limit-route.test.ts": 7446, - "server/src/__tests__/invite-summary-route.test.ts": 7598, - "server/src/__tests__/invite-test-resolution-route.test.ts": 6135, - "server/src/__tests__/invite-url-public-base-url.test.ts": 3482, - "server/src/__tests__/issue-activity-events-routes.test.ts": 6254, - "server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 13283, - "server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 5442, - "server/src/__tests__/issue-assignee-invokability-routes.test.ts": 4602, - "server/src/__tests__/issue-attachment-routes.test.ts": 5267, - "server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 10902, - "server/src/__tests__/issue-closed-workspace-routes.test.ts": 6589, - "server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 11053, - "server/src/__tests__/issue-comment-cancel-routes.test.ts": 4687, - "server/src/__tests__/issue-comment-reopen-routes.test.ts": 4009, - "server/src/__tests__/issue-create-deduplication-routes.test.ts": 11885, - "server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 5701, - "server/src/__tests__/issue-document-restore-routes.test.ts": 5902, - "server/src/__tests__/issue-execution-policy-routes.test.ts": 6571, - "server/src/__tests__/issue-feedback-routes.test.ts": 3978, - "server/src/__tests__/issue-identifier-routes.test.ts": 11276, - "server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 12518, - "server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 10919, - "server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 12493, - "server/src/__tests__/issue-scheduled-retry-routes.test.ts": 11559, - "server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 11281, - "server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 11437, - "server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 11043, - "server/src/__tests__/issue-telemetry-routes.test.ts": 4745, - "server/src/__tests__/issue-thread-interaction-routes.test.ts": 8825, - "server/src/__tests__/issue-tree-control-routes.test.ts": 2689, - "server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 6552, - "server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 11098, - "server/src/__tests__/issue-watchdogs-routes.test.ts": 15909, - "server/src/__tests__/issue-workspace-command-authz.test.ts": 3660, - "server/src/__tests__/issues-checkout-wakeup.test.ts": 1053, - "server/src/__tests__/issues-goal-context-routes.test.ts": 5356, - "server/src/__tests__/issues-service.test.ts": 37910, - "server/src/__tests__/llms-routes.test.ts": 2370, - "server/src/__tests__/low-trust-red-team-routes.test.ts": 25333, - "server/src/__tests__/multilingual-issues-routes.test.ts": 11329, - "server/src/__tests__/onboarding-seed-route.test.ts": 8981, - "server/src/__tests__/openapi-routes.test.ts": 3234, - "server/src/__tests__/openclaw-invite-prompt-route.test.ts": 3938, - "server/src/__tests__/opencode-local-adapter-environment.test.ts": 1071, - "server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 11054, - "server/src/__tests__/pipelines-routes.test.ts": 13514, - "server/src/__tests__/plugin-install-route-security.test.ts": 10977, - "server/src/__tests__/plugin-routes-authz.test.ts": 3628, - "server/src/__tests__/plugin-scoped-api-routes.test.ts": 3535, - "server/src/__tests__/project-goal-telemetry-routes.test.ts": 4566, - "server/src/__tests__/project-routes-env.test.ts": 3141, - "server/src/__tests__/projects-list-archived-routes.test.ts": 10943, - "server/src/__tests__/redaction.test.ts": 973, - "server/src/__tests__/resource-memberships-routes.test.ts": 11206, - "server/src/__tests__/routine-document-annotation-routes.test.ts": 2824, - "server/src/__tests__/routines-e2e.test.ts": 12006, - "server/src/__tests__/routines-routes.test.ts": 4019, - "server/src/__tests__/secret-proposals-routes.test.ts": 15670, - "server/src/__tests__/secrets-routes.test.ts": 4757, - "server/src/__tests__/sidebar-preferences-routes.test.ts": 3080, - "server/src/__tests__/summary-slot-routes.test.ts": 3858, - "server/src/__tests__/teams-catalog-routes.test.ts": 3463, - "server/src/__tests__/user-profile-routes.test.ts": 6441, - "server/src/__tests__/workspace-runtime-routes-authz.test.ts": 3296, - "server/src/__tests__/workspace-runtime-service-authz.test.ts": 6251 + "server/src/__tests__/access-routes-hidden-floor.test.ts": 11187, + "server/src/__tests__/access-routes-permissions-upgrade.test.ts": 18919, + "server/src/__tests__/activity-routes.test.ts": 5963, + "server/src/__tests__/adapter-auth-signal-routes.test.ts": 14105, + "server/src/__tests__/adapter-model-refresh-routes.test.ts": 10257, + "server/src/__tests__/adapter-routes-authz.test.ts": 7991, + "server/src/__tests__/adapter-routes.test.ts": 8580, + "server/src/__tests__/agent-action-audit-routes.test.ts": 17869, + "server/src/__tests__/agent-adapter-validation-routes.test.ts": 24906, + "server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 9297, + "server/src/__tests__/agent-device-login-routes.test.ts": 25216, + "server/src/__tests__/agent-hire-idempotency-routes.test.ts": 16903, + "server/src/__tests__/agent-instructions-routes.test.ts": 15221, + "server/src/__tests__/agent-live-run-routes.test.ts": 30747, + "server/src/__tests__/agent-permissions-routes.test.ts": 9849, + "server/src/__tests__/agent-secrets-routes.test.ts": 16629, + "server/src/__tests__/agent-skills-routes.test.ts": 24106, + "server/src/__tests__/agent-test-environment-routes.test.ts": 20121, + "server/src/__tests__/approval-routes-idempotency.test.ts": 6366, + "server/src/__tests__/artifact-review-document-routes.test.ts": 11415, + "server/src/__tests__/assets.test.ts": 7738, + "server/src/__tests__/auth-routes.test.ts": 3471, + "server/src/__tests__/auth-session-route.test.ts": 4282, + "server/src/__tests__/authz-company-access.test.ts": 3348, + "server/src/__tests__/authz-existence-oracle-guard.test.ts": 312, + "server/src/__tests__/authz-secret-context.test.ts": 3899, + "server/src/__tests__/board-chat-route-feature-flag.test.ts": 1251, + "server/src/__tests__/bootstrap-claim-routes.test.ts": 5821, + "server/src/__tests__/built-in-agent-routes.test.ts": 6607, + "server/src/__tests__/cases-routes.test.ts": 17725, + "server/src/__tests__/cli-auth-routes.test.ts": 8422, + "server/src/__tests__/cloud-routes.test.ts": 3475, + "server/src/__tests__/companies-route-cross-company-authz.test.ts": 10608, + "server/src/__tests__/companies-route-path-guard.test.ts": 3520, + "server/src/__tests__/company-branding-route.test.ts": 6319, + "server/src/__tests__/company-import-transfer-routes.test.ts": 10852, + "server/src/__tests__/company-portability-routes.test.ts": 5037, + "server/src/__tests__/company-portability.test.ts": 6030, + "server/src/__tests__/company-search-extract-routes.test.ts": 11382, + "server/src/__tests__/company-search-rate-limit-routes.test.ts": 11592, + "server/src/__tests__/company-skill-policy-routes.test.ts": 9969, + "server/src/__tests__/company-skills-import-authz-routes.test.ts": 16725, + "server/src/__tests__/company-skills-routes.test.ts": 6666, + "server/src/__tests__/company-user-directory-route.test.ts": 5254, + "server/src/__tests__/costs-service.test.ts": 10178, + "server/src/__tests__/decision-queues-routes.test.ts": 9909, + "server/src/__tests__/document-annotation-routes.test.ts": 12372, + "server/src/__tests__/environment-custom-image-routes.test.ts": 8712, + "server/src/__tests__/environment-instance-routes.test.ts": 8738, + "server/src/__tests__/environment-routes.test.ts": 5462, + "server/src/__tests__/environment-selection-route-guards.test.ts": 9749, + "server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 12231, + "server/src/__tests__/execution-workspaces-routes.test.ts": 6598, + "server/src/__tests__/express5-auth-wildcard.test.ts": 1098, + "server/src/__tests__/external-object-routes.test.ts": 14559, + "server/src/__tests__/folders-routes.test.ts": 3122, + "server/src/__tests__/health-dev-server-token.test.ts": 3808, + "server/src/__tests__/health.test.ts": 3664, + "server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 22771, + "server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 26411, + "server/src/__tests__/heartbeat-process-recovery.test.ts": 165058, + "server/src/__tests__/inbox-agent-policy-routes.test.ts": 17418, + "server/src/__tests__/inbox-archive-routes.test.ts": 18276, + "server/src/__tests__/instance-database-backups-routes.test.ts": 3602, + "server/src/__tests__/instance-settings-routes.test.ts": 3263, + "server/src/__tests__/invite-accept-existing-member.test.ts": 5373, + "server/src/__tests__/invite-accept-gateway-defaults.test.ts": 16661, + "server/src/__tests__/invite-accept-replay.test.ts": 11438, + "server/src/__tests__/invite-create-route.test.ts": 5108, + "server/src/__tests__/invite-expiry.test.ts": 11055, + "server/src/__tests__/invite-join-manager.test.ts": 10519, + "server/src/__tests__/invite-list-route.test.ts": 10911, + "server/src/__tests__/invite-logo-route.test.ts": 10811, + "server/src/__tests__/invite-onboarding-text.test.ts": 9920, + "server/src/__tests__/invite-rate-limit-route.test.ts": 11305, + "server/src/__tests__/invite-summary-route.test.ts": 12300, + "server/src/__tests__/invite-test-resolution-route.test.ts": 10497, + "server/src/__tests__/invite-url-public-base-url.test.ts": 5781, + "server/src/__tests__/issue-activity-events-routes.test.ts": 13914, + "server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 50945, + "server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 9454, + "server/src/__tests__/issue-assignee-invokability-routes.test.ts": 9340, + "server/src/__tests__/issue-attachment-routes.test.ts": 18688, + "server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 17348, + "server/src/__tests__/issue-closed-workspace-routes.test.ts": 9378, + "server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 16687, + "server/src/__tests__/issue-comment-cancel-routes.test.ts": 9120, + "server/src/__tests__/issue-comment-reopen-routes.test.ts": 9744, + "server/src/__tests__/issue-create-deduplication-routes.test.ts": 18131, + "server/src/__tests__/issue-created-from-routes.test.ts": 18051, + "server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 12754, + "server/src/__tests__/issue-document-restore-routes.test.ts": 11341, + "server/src/__tests__/issue-execution-policy-routes.test.ts": 17384, + "server/src/__tests__/issue-feedback-routes.test.ts": 10562, + "server/src/__tests__/issue-identifier-routes.test.ts": 17386, + "server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 18762, + "server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 17534, + "server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 19600, + "server/src/__tests__/issue-queued-comments-routes.test.ts": 28267, + "server/src/__tests__/issue-scheduled-retry-routes.test.ts": 17617, + "server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 17166, + "server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 18751, + "server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 18263, + "server/src/__tests__/issue-telemetry-routes.test.ts": 9904, + "server/src/__tests__/issue-thread-interaction-routes.test.ts": 39373, + "server/src/__tests__/issue-tree-control-routes.test.ts": 3616, + "server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 15540, + "server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 17422, + "server/src/__tests__/issue-watchdogs-routes.test.ts": 22439, + "server/src/__tests__/issue-workspace-command-authz.test.ts": 9358, + "server/src/__tests__/issues-checkout-wakeup.test.ts": 1064, + "server/src/__tests__/issues-goal-context-routes.test.ts": 9431, + "server/src/__tests__/issues-service.test.ts": 76576, + "server/src/__tests__/llms-routes.test.ts": 3399, + "server/src/__tests__/low-trust-red-team-routes.test.ts": 27945, + "server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 3624, + "server/src/__tests__/multilingual-issues-routes.test.ts": 17623, + "server/src/__tests__/onboarding-seed-route.test.ts": 13936, + "server/src/__tests__/openapi-routes.test.ts": 4089, + "server/src/__tests__/openclaw-invite-prompt-route.test.ts": 5131, + "server/src/__tests__/opencode-local-adapter-environment.test.ts": 8676, + "server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 16917, + "server/src/__tests__/pipelines-routes.test.ts": 20408, + "server/src/__tests__/plugin-install-route-security.test.ts": 13924, + "server/src/__tests__/plugin-routes-authz.test.ts": 5693, + "server/src/__tests__/plugin-scoped-api-routes.test.ts": 6245, + "server/src/__tests__/project-goal-telemetry-routes.test.ts": 6735, + "server/src/__tests__/project-routes-env.test.ts": 5412, + "server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6840, + "server/src/__tests__/projects-list-archived-routes.test.ts": 17390, + "server/src/__tests__/redaction.test.ts": 1125, + "server/src/__tests__/resource-memberships-routes.test.ts": 17692, + "server/src/__tests__/routine-document-annotation-routes.test.ts": 3749, + "server/src/__tests__/routines-e2e.test.ts": 20181, + "server/src/__tests__/routines-routes.test.ts": 6778, + "server/src/__tests__/secret-proposals-routes.test.ts": 21824, + "server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 11110, + "server/src/__tests__/secrets-routes.test.ts": 10410, + "server/src/__tests__/sidebar-preferences-routes.test.ts": 4495, + "server/src/__tests__/summary-slot-routes.test.ts": 6451, + "server/src/__tests__/teams-catalog-routes.test.ts": 4948, + "server/src/__tests__/user-profile-routes.test.ts": 9198, + "server/src/__tests__/workspace-command-authz.test.ts": 1044, + "server/src/__tests__/workspace-runtime-routes-authz.test.ts": 6915, + "server/src/__tests__/workspace-runtime-service-authz.test.ts": 10428 } } From 7435b2ee9cc6c71e733fd77fee523d67b8b2badf Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 12 Sep 2026 11:57:32 -0700 Subject: [PATCH 4/4] ci: cache compiled Docker Rust dependencies separately from source (#13329) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Cloud deploys images that contain the native Rust Runner. > - The image already builds that Runner before copying ordinary app source. > - A Rust source change still invalidates its entire compiled dependency layer. > - Compiled dependencies can survive source changes when their recipe is unchanged. > - This PR adds a separate locked dependency build before compiling the real workspace. ## Linked Issues or Issue Description Refs #13195. A search of related Docker and Cargo cache PRs found no duplicate dependency-recipe change. **What existing behavior does this improve?** Docker image build time after Rust source or embedded protocol changes. **Current behavior** The `runner-build` stage compiles dependencies and workspace code in one layer. In Cloud readiness run 34698143548, that stage took about 3m48s when its cache was unavailable. **Proposed behavior** Generate a recipe with pinned cargo-chef 0.1.73. Build locked release dependencies in `runner-deps`, then copy and compile real Rust source and embedded protocol inputs in `runner-build`. Source edits can reuse the dependency layer from the existing registry cache. **Reason and benefit** Reduce dependency recompilation during source changes and merge bursts. Expected savings are roughly 2โ€“4 minutes when the old native layer would miss but dependency layers are available. Full cold builds also pay for the recipe tool installation. Ordinary app-only cache hits gain little from this change. **Breaking changes** None to the shipped application or image tags. The recipe tool and compiled dependencies remain in build stages. ## What Changed - Install a pinned recipe generator with its locked dependencies and the existing package-owned compiler. - Add recipe planning and compiled dependency stages. Use the same release profile, package, binary, and lockfile enforcement as the real native build. - Remove generated source stubs before copying actual source. Preserve protocol inputs, timestamp normalization, binary staging, and application checks. - Add Docker cache wiring regressions and update the Docker cache documentation. - Run a two-build probe in Docker Runner check. It requires dependency reuse, changed real binary metadata after a source edit, and a changed recipe after a dependency declaration edit. It uses a disposable tracked-source context and exports only small metadata files. ## Verification - Passed all five Docker build-stamp and dependency-cache tests with `pnpm exec vitest run server/src/__tests__/docker-build-stamp.test.ts`. - Passed the local ARM64 `docker buildx build --target runner-build --progress plain`. Local Docker then hit storage errors during a runtime probe; cache invalidation verification continues on GitHub-hosted Linux. - Passed `bash -n scripts/check-docker-runner-cache.sh`, `actionlint`, and `git diff --check`. - Passed a [Linux AMD64 cache probe](https://github.com/paperclipai/paperclip/actions/runs/34711042199) against the PR source: dependencies compiled in 3m49s for the baseline and were `CACHED` after a source edit; real source compilation took about 37 seconds. Binary metadata changed and dependency declaration changes altered the recipe. The permanent probe is also running in latest-head Docker Runner check. - Passed latest-head [Docker Runner check](https://github.com/paperclipai/paperclip/actions/runs/34711145160), including the permanent source/dependency invalidation probe. - Passed full [PR verification](https://github.com/paperclipai/paperclip/actions/runs/34711145352/attempts/2): typecheck, all grouped tests, native verification, build, release dry run, and browser checks. One unrelated signoff-policy browser test failed waiting for a heartbeat run on attempt 1; only that failed shard and dependent checks were retried, and passed. - Latest-head Greptile is 5/5 with no unresolved findings. Full local tests/build were limited by local disk exhaustion; Linux CI completed those checks. ## Risks - The two-build CI probe has a 20-minute job limit to cover the cold build and source rebuild. It adds no AWS routing. - A fully cold build must install cargo-chef and populate the dependency layer. Both become reusable registry layers; no Actions cache is added. - The recipe and final build must keep the same compiler, build profile, package, binary, and directory layout. A source-change rebuild probe checks real cache reuse and binary invalidation. - Dependency or compiler changes still require rebuilding dependencies. Existing image verification and full-SHA publication gates remain unchanged. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/docker-runner-check.yml | 13 ++-- Dockerfile | 22 ++++++- doc/DOCKER.md | 32 +++++++--- scripts/check-docker-runner-cache.sh | 60 +++++++++++++++++++ .../src/__tests__/docker-build-stamp.test.ts | 29 +++++++++ 5 files changed, 141 insertions(+), 15 deletions(-) create mode 100644 scripts/check-docker-runner-cache.sh diff --git a/.github/workflows/docker-runner-check.yml b/.github/workflows/docker-runner-check.yml index e9705c3b5e..7c3b7820b5 100644 --- a/.github/workflows/docker-runner-check.yml +++ b/.github/workflows/docker-runner-check.yml @@ -6,6 +6,7 @@ on: - .github/workflows/docker-runner-check.yml - Dockerfile - .dockerignore + - scripts/check-docker-runner-cache.sh - packages/paperclip-runner/rust-toolchain.toml - packages/paperclip-runner/runner/** - packages/paperclip-runner/protocol/** @@ -20,7 +21,7 @@ jobs: runner: name: Compile isolated native Runner runs-on: ubuntu-latest - timeout-minutes: 15 + timeout-minutes: 20 permissions: contents: read steps: @@ -30,8 +31,8 @@ jobs: persist-credentials: false - name: Set up Docker Buildx uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 - # Compile the real target with the real .dockerignore. This catches new - # Cargo or embedded protocol inputs that the isolated COPY set omits. - # No registry credentials, cache imports/exports, or image publication. - - name: Compile the Runner from its isolated Docker context - run: docker buildx build --target runner-build --progress plain . + # Compile the real target, then change source in a disposable context. + # Require dependency-layer reuse and changed metadata from the real binary. + # No registry credentials, external cache, or image publication. + - name: Verify native build and dependency cache reuse + run: bash scripts/check-docker-runner-cache.sh diff --git a/Dockerfile b/Dockerfile index b349251238..a3d046c96e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -88,7 +88,27 @@ RUN set -eux; \ COPY packages/paperclip-runner/rust-toolchain.toml /tmp/runner-toolchain/rust-toolchain.toml RUN cd /tmp/runner-toolchain && rustup show -FROM rust-toolchain AS runner-build +# Pin the recipe generator and its dependency lockfile. It is a build-only tool +# and uses the same package-owned compiler as both native build stages. +FROM rust-toolchain AS rust-chef +RUN cd /tmp/runner-toolchain && cargo install cargo-chef --version 0.1.73 --locked + +FROM rust-chef AS runner-plan +WORKDIR /app/packages/paperclip-runner +COPY packages/paperclip-runner/rust-toolchain.toml ./ +COPY packages/paperclip-runner/runner ./runner +RUN cd runner && cargo chef prepare --recipe-path /tmp/runner-recipe.json + +FROM rust-chef AS runner-deps +WORKDIR /app/packages/paperclip-runner/runner +COPY packages/paperclip-runner/rust-toolchain.toml ../ +# The recipe changes only when dependency manifests, the lockfile, or target +# metadata change. Source edits can reuse this compiled dependency layer. +COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json +RUN cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd --recipe-path /tmp/runner-recipe.json \ + && find . -mindepth 1 -maxdepth 1 ! -name target -exec rm -rf {} + + +FROM runner-deps AS runner-build WORKDIR /app/packages/paperclip-runner # Rust embeds protocol schemas and fixtures with include_str!. Keep those # alongside the complete Cargo workspace so every compile-time input keys diff --git a/doc/DOCKER.md b/doc/DOCKER.md index 757a2c3094..2905bfe957 100644 --- a/doc/DOCKER.md +++ b/doc/DOCKER.md @@ -342,11 +342,20 @@ Notes: ## Native Runner build cache The image compiles the native Runner in `runner-build`, before copying the -application source. That stage includes the pinned Rust compiler, the complete -Cargo workspace and lockfile, and the protocol schemas and fixtures embedded -by Rust. Changes to those inputs rebuild the native binary. Ordinary server or -UI changes can reuse it through the existing registry cache (`mode=max`). Each -platform gets its own native build; no cross-architecture binary is reused. +application source. A pinned `cargo-chef` generates a dependency recipe in +`runner-plan`. The separate `runner-deps` stage compiles that recipe with the +package-owned Rust compiler. Both the dependency build and the real binary use +the release profile and locked Cargo dependencies. The recipe stage never +modifies source in the checkout. + +Changes to Rust source or embedded protocol inputs rebuild the real binary but +can reuse compiled dependencies when the recipe is unchanged. Dependency +manifests, the Cargo lockfile, target metadata, or compiler changes invalidate +the relevant cache. Ordinary server or UI changes can reuse the entire native +build through the existing registry cache (`mode=max`). Each platform gets its +own native build; no cross-architecture binary is reused. No additional GitHub +Actions cache is created. A cold build also installs the recipe generator and +compiles dependencies, so the savings apply after those layers are available. The application build inherits that stage and still runs the normal server build, including Cargo, binary staging, and generated-contract checks. Rust @@ -357,6 +366,13 @@ directory as before. Cache misses only cost compilation time. Pull requests that change the Dockerfile, Docker ignore rules, or Runner native inputs also build the isolated `runner-build` target in `Docker Runner check`. -This compiles against the actual reduced context and catches missing embedded -inputs before the post-merge image build. It uses a GitHub-hosted runner with -read-only repository access and does not publish images or cache artifacts. +The check runs `bash scripts/check-docker-runner-cache.sh` against a disposable +copy of tracked source and the actual Docker ignore rules. It compiles a baseline, +changes a Rust metadata constant, and rebuilds. It requires a cached dependency +build, an unchanged dependency recipe, and changed metadata from the real binary. +It also verifies that a dependency declaration change alters the recipe. The +probe exports only small metadata files, avoiding a large image import into the +Docker daemon. It catches missing embedded inputs before the post-merge build. +It uses a GitHub-hosted runner with read-only repository access and does not +publish images or cache artifacts. Allow up to 20 minutes for its cold build and +source rebuild. diff --git a/scripts/check-docker-runner-cache.sh b/scripts/check-docker-runner-cache.sh new file mode 100644 index 0000000000..0c8caf81d5 --- /dev/null +++ b/scripts/check-docker-runner-cache.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Build the real Docker target twice in a disposable copy of tracked source. +# Export only metadata, avoiding a multi-gigabyte test image in the daemon. +set -euo pipefail +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")" +trap 'rm -rf "$probe_dir"' EXIT +mkdir "$probe_dir/context" +cd "$repo_root" +git ls-files -z | tar -cf - --null -T - | tar -xf - -C "$probe_dir/context" +cd "$probe_dir/context" +export PROBE_DIR="$probe_dir" +cp Dockerfile "$probe_dir/cache-probe.Dockerfile" +cat >> "$probe_dir/cache-probe.Dockerfile" <<'DOCKER' +FROM runner-build AS cache-proof +RUN ./runner/target/release/paperclip-runnerd --build-metadata > /metadata.json +FROM scratch AS cache-proof-export +COPY --from=cache-proof /metadata.json /metadata.json +COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json +FROM scratch AS recipe-proof-export +COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json +DOCKER +build_proof() { + docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$1" --progress plain . 2>&1 | tee "$probe_dir/$1.log" +} +build_proof baseline +python3 - <<'CHECK' +from pathlib import Path +p=Path('packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs') +s=p.read_text(); needle='paperclip-runner/runnerd-build-metadata/v1' +assert s.count(needle)==1 +p.write_text(s.replace(needle,needle+'-cache-probe')) +CHECK +build_proof source-change +python3 - <<'CHECK' +import os,json,re +from pathlib import Path +root=Path(os.environ['PROBE_DIR']) +before=json.loads((root/'baseline/metadata.json').read_text()) +after=json.loads((root/'source-change/metadata.json').read_text()) +assert before['schema']=='paperclip-runner/runnerd-build-metadata/v1' +assert after['schema']==before['schema']+'-cache-probe' +assert (root/'baseline/recipe.json').read_bytes()==(root/'source-change/recipe.json').read_bytes() +log=(root/'source-change.log').read_text() +step=re.search(r'#(\d+) \[runner-deps[^\n]+ RUN cargo chef cook',log)[1] +assert f'#{step} CACHED' in log +assert 'Compiling paperclip-runner-core' in log +print('PASS: unchanged dependency recipe and cached cook layer; real binary changed.') +p=Path('packages/paperclip-runner/runner/Cargo.toml') +s=p.read_text(); assert 'serde_json = "1.0"' in s +p.write_text(s.replace('serde_json = "1.0"','serde_json = ">=1.0.0, <2.0.0"')) +CHECK +docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target recipe-proof-export --output "type=local,dest=$probe_dir/manifest-change" --progress plain . +python3 - <<'CHECK' +from pathlib import Path +import os +root=Path(os.environ['PROBE_DIR']) +assert (root/'source-change/recipe.json').read_bytes()!=(root/'manifest-change/recipe.json').read_bytes() +print('PASS: dependency declaration change invalidates the recipe.') +CHECK diff --git a/server/src/__tests__/docker-build-stamp.test.ts b/server/src/__tests__/docker-build-stamp.test.ts index 4794989feb..76862952c8 100644 --- a/server/src/__tests__/docker-build-stamp.test.ts +++ b/server/src/__tests__/docker-build-stamp.test.ts @@ -76,3 +76,32 @@ describe("docker build-stamp wiring", () => { ).toBeGreaterThanOrEqual(2); }); }); + + +describe("Docker Rust dependency cache", () => { + it("caches the locked dependency recipe separately from source and per-build metadata", () => { + const chef = stageBody(dockerfile, "rust-chef"); + const planner = stageBody(dockerfile, "runner-plan"); + const dependencies = stageBody(dockerfile, "runner-deps"); + expect(chef).toContain("FROM rust-toolchain AS rust-chef"); + expect(chef).toMatch(/cargo install cargo-chef --version \d+\.\d+\.\d+ --locked/); + expect(planner).toContain("COPY packages/paperclip-runner/runner ./runner"); + expect(planner).toContain("cargo chef prepare --recipe-path /tmp/runner-recipe.json"); + expect(dependencies).toContain("FROM rust-chef AS runner-deps"); + expect(dependencies).toContain("COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json"); + expect(dependencies).toContain("cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd"); + expect(dependencies).not.toMatch(/COPY .*\.\/runner|COPY .*\.\/protocol|COPY \. \.|PAPERCLIP_BUILD_COMMIT/); + }); + + it("rebuilds real workspace code and embedded protocol inputs after cooking dependencies", () => { + const native = stageBody(dockerfile, "runner-build"); + expect(native).toContain("FROM runner-deps AS runner-build"); + for (const source of ["runner", "protocol"]) { + expect(native.indexOf(`COPY packages/paperclip-runner/${source} ./${source}`)) + .toBeLessThan(native.indexOf("cargo build --release")); + expect(native).toContain(`COPY packages/paperclip-runner/${source} ./${source}`); + } + expect(native).toContain("cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd"); + expect(stageBody(dockerfile, "build")).toContain("FROM runner-build AS build"); + }); +});