[codex] Parallelize release verify workflow (#9168)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Releases publish the same app and package set that operators install, so release verification should keep full release-strength coverage. > - The release workflow currently verifies stable and canary releases with one serial job that typechecks, runs all tests, and builds. > - The PR workflow already proves the test surface can be split into grouped general suites and serialized shards without changing coverage. > - This pull request extracts the release verify work into a reusable workflow and fans out the independent lanes. > - The benefit is faster stable and canary release verification while preserving the existing publish and preview gates. ## Linked Issues or Issue Description No public GitHub issue exists for this CI improvement. **Subsystem affected** Cross-cutting (multiple of the above) **Problem or motivation** Release verification spends most of its wall time in a single serial test step even though the same stable test surface is already partitioned for PR CI. Stable dispatches and master-push canaries therefore wait on one long runner after setup, typecheck, tests, and build run sequentially. **Proposed solution** Add a reusable release verification workflow with parallel typecheck, grouped general tests, serialized test shards, and build lanes. Have both stable and canary release verification call it with the ref they need to verify. **Alternatives considered** Keeping the serial `pnpm test:run` job preserves the old shape but keeps stable and canary releases waiting on one long runner. Skipping verification when a source SHA already has green CI would be faster, but adds stale-check and lookup risk beyond this change. **Roadmap alignment** No overlapping item found in `ROADMAP.md`; this is release CI maintenance. **Additional context** The new workflow keeps the release-strength full `pnpm -r typecheck`, uses the existing stable test grouping/sharding entry points, and leaves publish/preview jobs unchanged. ## What Changed - Added `.github/workflows/release-verify.yml` as a `workflow_call` workflow accepting a `ref` input. - Split release verification into parallel `typecheck`, `general_tests`, `serialized_tests`, and `build` jobs with 20-minute lane timeouts. - Mirrored the PR workflow's stable test partition: `general-server` shards 1-3, `general-workspaces-a`, `general-workspaces-b`, and four serialized shards. - Replaced `release.yml` `verify_canary` and `verify_stable` job bodies with calls to the reusable workflow while leaving publish and preview jobs unchanged. - Added a Node test that guards the release workflow delegation and split verify surface. ## Verification - `actionlint 1.7.12 .github/workflows/release.yml .github/workflows/release-verify.yml` - `node ./scripts/release-package-map.mjs check` - `node --test ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/run-vitest-stable-shard.test.mjs` - `git diff --check` ## Risks - Release verification now starts more jobs per release event, increasing total runner setup/install minutes. This matches the existing PR CI tradeoff and should reduce release wall time substantially. - The called workflow checks out the requested ref shallowly. That is intentional for verify lanes; publish and preview jobs still retain their existing full-history checkouts. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5-class coding agent in local tool-use mode with shell execution, repository editing, GitHub connector access, and medium reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
GitHub
parent
d3e26a8d02
commit
cec0fc249a
|
|
@ -58,6 +58,9 @@ jobs:
|
|||
- name: Test general-server shard partition
|
||||
run: node --test ./scripts/__tests__/run-vitest-stable-shard.test.mjs
|
||||
|
||||
- name: Test release verify workflow wiring
|
||||
run: node --test ./scripts/__tests__/release-verify-workflow.test.mjs
|
||||
|
||||
- name: Test standalone package build concurrency
|
||||
run: node --test ./scripts/__tests__/build-standalone-concurrency.test.mjs
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,175 @@
|
|||
name: Release Verify
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
ref:
|
||||
description: Commit SHA, branch, or tag to verify
|
||||
required: true
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
typecheck:
|
||||
name: Typecheck
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
- name: Validate release package manifest
|
||||
run: node ./scripts/release-package-map.mjs check
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Typecheck
|
||||
run: pnpm -r typecheck
|
||||
|
||||
general_tests:
|
||||
name: General tests (${{ matrix.group_label }})
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- group: general-server
|
||||
group_label: server (1/3)
|
||||
shard_index: 0
|
||||
shard_count: 3
|
||||
- group: general-server
|
||||
group_label: server (2/3)
|
||||
shard_index: 1
|
||||
shard_count: 3
|
||||
- group: general-server
|
||||
group_label: server (3/3)
|
||||
shard_index: 2
|
||||
shard_count: 3
|
||||
- group: general-workspaces-a
|
||||
group_label: workspaces-a
|
||||
- group: general-workspaces-b
|
||||
group_label: workspaces-b
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Run grouped general test suites
|
||||
run: |
|
||||
if [ -n "${{ matrix.shard_count }}" ]; then
|
||||
pnpm test:run:general -- --group '${{ matrix.group }}' \
|
||||
--shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
|
||||
else
|
||||
pnpm test:run:general -- --group '${{ matrix.group }}'
|
||||
fi
|
||||
|
||||
serialized_tests:
|
||||
name: Serialized tests (${{ matrix.shard_label }})
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- shard_index: 0
|
||||
shard_count: 4
|
||||
shard_label: 1/4
|
||||
- shard_index: 1
|
||||
shard_count: 4
|
||||
shard_label: 2/4
|
||||
- shard_index: 2
|
||||
shard_count: 4
|
||||
shard_label: 3/4
|
||||
- shard_index: 3
|
||||
shard_count: 4
|
||||
shard_label: 4/4
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Run serialized server test shard
|
||||
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
|
||||
|
||||
build:
|
||||
name: Build
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
|
|
@ -28,42 +28,9 @@ concurrency:
|
|||
jobs:
|
||||
verify_canary:
|
||||
if: github.event_name == 'push'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
- name: Validate release package manifest
|
||||
run: node ./scripts/release-package-map.mjs check
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Typecheck
|
||||
run: pnpm -r typecheck
|
||||
|
||||
- name: Run tests
|
||||
run: pnpm test:run
|
||||
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
uses: ./.github/workflows/release-verify.yml
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
|
||||
publish_canary:
|
||||
if: github.event_name == 'push'
|
||||
|
|
@ -122,43 +89,9 @@ jobs:
|
|||
|
||||
verify_stable:
|
||||
if: github.event_name == 'workflow_dispatch'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
ref: ${{ inputs.source_ref }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v6
|
||||
with:
|
||||
version: 9.15.4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: 24
|
||||
cache: pnpm
|
||||
|
||||
- name: Validate release package manifest
|
||||
run: node ./scripts/release-package-map.mjs check
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --no-frozen-lockfile
|
||||
|
||||
- name: Typecheck
|
||||
run: pnpm -r typecheck
|
||||
|
||||
- name: Run tests
|
||||
run: pnpm test:run
|
||||
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
uses: ./.github/workflows/release-verify.yml
|
||||
with:
|
||||
ref: ${{ inputs.source_ref }}
|
||||
|
||||
preview_stable:
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.dry_run
|
||||
|
|
|
|||
|
|
@ -0,0 +1,52 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import test from "node:test";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
|
||||
function readWorkflow(name) {
|
||||
return readFileSync(path.join(repoRoot, ".github/workflows", name), "utf8");
|
||||
}
|
||||
|
||||
test("release workflow delegates stable and canary verification to the reusable workflow", () => {
|
||||
const releaseWorkflow = readWorkflow("release.yml");
|
||||
|
||||
assert.match(
|
||||
releaseWorkflow,
|
||||
/verify_canary:\n\s+if: github\.event_name == 'push'\n\s+uses: \.\/\.github\/workflows\/release-verify\.yml\n\s+with:\n\s+ref: \$\{\{ github\.sha \}\}/,
|
||||
);
|
||||
assert.match(
|
||||
releaseWorkflow,
|
||||
/verify_stable:\n\s+if: github\.event_name == 'workflow_dispatch'\n\s+uses: \.\/\.github\/workflows\/release-verify\.yml\n\s+with:\n\s+ref: \$\{\{ inputs\.source_ref \}\}/,
|
||||
);
|
||||
assert.doesNotMatch(releaseWorkflow, /verify_(?:canary|stable):[\s\S]*?pnpm test:run(?:\n|$)/);
|
||||
});
|
||||
|
||||
test("release verify workflow covers the same split test surface as stable PR verification", () => {
|
||||
const verifyWorkflow = readWorkflow("release-verify.yml");
|
||||
|
||||
assert.match(verifyWorkflow, /workflow_call:/);
|
||||
assert.match(verifyWorkflow, /node \.\/scripts\/release-package-map\.mjs check/);
|
||||
assert.match(verifyWorkflow, /pnpm -r typecheck/);
|
||||
assert.match(verifyWorkflow, /pnpm build/);
|
||||
|
||||
for (const group of ["general-server", "general-workspaces-a", "general-workspaces-b"]) {
|
||||
assert.match(verifyWorkflow, new RegExp(`group: ${group}`));
|
||||
}
|
||||
|
||||
for (const shardIndex of [0, 1, 2]) {
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
new RegExp(`group: general-server[\\s\\S]*?shard_index: ${shardIndex}[\\s\\S]*?shard_count: 3`),
|
||||
);
|
||||
}
|
||||
|
||||
for (const shardIndex of [0, 1, 2, 3]) {
|
||||
assert.match(verifyWorkflow, new RegExp(`shard_index: ${shardIndex}[\\s\\S]*?shard_count: 4`));
|
||||
}
|
||||
|
||||
assert.match(verifyWorkflow, /pnpm test:run:general -- --group/);
|
||||
assert.match(verifyWorkflow, /pnpm test:run:serialized -- --shard-index/);
|
||||
});
|
||||
|
|
@ -133,29 +133,35 @@ describeEmbeddedPostgres("heartbeat worktree suppression", () => {
|
|||
return { companyId, agentId, issueId };
|
||||
}
|
||||
|
||||
async function waitForTerminalRun(runId: string) {
|
||||
for (let attempt = 0; attempt < 20; attempt += 1) {
|
||||
async function waitForCompletedRun(runId: string, agentId: string) {
|
||||
let latestStatus: string | null = null;
|
||||
let latestLastRunId: string | null = null;
|
||||
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
const run = await db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (run && run.status !== "queued" && run.status !== "running") return run.status;
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function waitForRuntimeStateLastRun(agentId: string, runId: string) {
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
const state = await db
|
||||
.select({ lastRunId: agentRuntimeState.lastRunId })
|
||||
.from(agentRuntimeState)
|
||||
.where(eq(agentRuntimeState.agentId, agentId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (state?.lastRunId === runId) return;
|
||||
|
||||
latestStatus = run?.status ?? null;
|
||||
latestLastRunId = state?.lastRunId ?? null;
|
||||
|
||||
if (run && run.status !== "queued" && run.status !== "running" && state?.lastRunId === runId) {
|
||||
return run.status;
|
||||
}
|
||||
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`Timed out waiting for heartbeat run ${runId} to finish; latest status=${latestStatus ?? "missing"}, runtime lastRunId=${latestLastRunId ?? "missing"}`,
|
||||
);
|
||||
}
|
||||
|
||||
it("suppresses new assignment wakes in worktree instances without creating heartbeat runs", async () => {
|
||||
|
|
@ -239,6 +245,11 @@ describeEmbeddedPostgres("heartbeat worktree suppression", () => {
|
|||
|
||||
it("still creates live-plane assignment runs when suppression is not active", async () => {
|
||||
const { agentId, issueId } = await insertAgentAndIssue();
|
||||
await db
|
||||
.update(issues)
|
||||
.set({ status: "in_review", updatedAt: new Date() })
|
||||
.where(eq(issues.id, issueId));
|
||||
|
||||
const heartbeat = heartbeatService(db, { runtimeEnv: {} });
|
||||
|
||||
const run = await heartbeat.wakeup(agentId, {
|
||||
|
|
@ -246,27 +257,22 @@ describeEmbeddedPostgres("heartbeat worktree suppression", () => {
|
|||
triggerDetail: "system",
|
||||
reason: "issue_assigned",
|
||||
payload: { issueId },
|
||||
contextSnapshot: { issueId, wakeReason: "issue_assigned" },
|
||||
contextSnapshot: { issueId, wakeReason: "issue_assigned", skipIssueComment: true },
|
||||
requestedByActorType: "system",
|
||||
requestedByActorId: "issue_assignment",
|
||||
});
|
||||
|
||||
expect(run).not.toBeNull();
|
||||
const terminalStatus = await waitForTerminalRun(run!.id);
|
||||
expect(["succeeded", null]).toContain(terminalStatus);
|
||||
const terminalStatus = await waitForCompletedRun(run!.id, agentId);
|
||||
await heartbeat.waitForRunExecutionDrain(run!.id);
|
||||
expect(terminalStatus).toBe("succeeded");
|
||||
|
||||
const runCount = await db
|
||||
.select({ count: sql<number>`count(*)::int` })
|
||||
.from(heartbeatRuns)
|
||||
.then((rows) => rows[0]?.count ?? 0);
|
||||
expect(runCount).toBe(1);
|
||||
|
||||
await db
|
||||
.update(issues)
|
||||
.set({ status: "done", updatedAt: new Date() })
|
||||
.where(eq(issues.id, issueId));
|
||||
await waitForRuntimeStateLastRun(agentId, run!.id);
|
||||
});
|
||||
}, 10_000);
|
||||
|
||||
it("recognizes explicit restore-in-progress suppression", () => {
|
||||
expect(resolveHeartbeatSchedulingSuppression({
|
||||
|
|
|
|||
|
|
@ -2640,6 +2640,8 @@ export function shouldResetTaskSessionForWake(
|
|||
function shouldRequireIssueCommentForWake(
|
||||
contextSnapshot: Record<string, unknown> | null | undefined,
|
||||
) {
|
||||
if (contextSnapshot?.skipIssueComment === true) return false;
|
||||
|
||||
const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason);
|
||||
return (
|
||||
wakeReason === "issue_assigned" ||
|
||||
|
|
@ -15043,6 +15045,21 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
}
|
||||
|
||||
return {
|
||||
waitForRunExecutionDrain: async (
|
||||
runId: string,
|
||||
options: { timeoutMs?: number; intervalMs?: number } = {},
|
||||
) => {
|
||||
const timeoutMs = options.timeoutMs ?? 5_000;
|
||||
const intervalMs = options.intervalMs ?? 25;
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
|
||||
while (liveRunExecutions.has(runId)) {
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`Timed out waiting for heartbeat run ${runId} execution to drain`);
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, intervalMs));
|
||||
}
|
||||
},
|
||||
list: async (
|
||||
companyId: string,
|
||||
agentId?: string,
|
||||
|
|
|
|||
Loading…
Reference in New Issue