From c5c80e1febe903e5e2d45d2664cd4394b5e797ed Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 10 Sep 2026 17:47:00 -0700 Subject: [PATCH] ci(release-verify): split server tests five ways like pr-trusted (#13185) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every master push publishes a canary through release.yml, gated by release-verify.yml — the fleet's staging deploys and the nightly/beta/stable chain all start from those canaries > - release-verify splits the server test suite across three shards with a 20-minute job cap, while pr-trusted splits the same suite across five > - The server suite grew on 2026-09-10 and the three shards moved to 17-19 minutes; that evening every push-triggered canary run was cancelled by the 20-minute cap mid-verify, and no canary published after 18:50 UTC > - This pull request mirrors pr-trusted's five-way server split in release-verify, putting shards back at the 10-15 minute range with real headroom > - The benefit is a canary lane that reports test verdicts instead of dying on an infrastructure cap ## Linked Issues or Issue Description **What happened?** Push-triggered Release runs stopped publishing canaries on 2026-09-10. Runs at 19:34, 22:30, and 22:37 UTC were all cancelled by "The job has exceeded the maximum execution time of 20m0s" on a `verify_canary / General tests (server (N/3))` shard. No canary published after 18:50 UTC, which also starves the staging fleet's continuous deploys. **Expected behavior** release-verify's server shards finish well inside the 20-minute cap and runs conclude with a test verdict, as pr-trusted's five-way split of the same suite does (10-15 minutes per shard). **Steps to reproduce** 1. Compare server shard durations in the `verify_canary` job across 2026-09-10: 11-14 minutes in the morning, 17-19 minutes from 15:06 UTC, over 20 minutes by evening. 2. Observe runs 34521169020, 34537798488, and 34538332689 cancelled at the cap. **Paperclip version or commit** `master` at `d1ba17eec` (current tip; its canary run was one of the cancelled ones). ## What Changed - `release-verify.yml`: the `general-server` matrix goes from three shards to five, byte-for-byte the shape `pr-trusted.yml` already runs, with a comment recording why. ## Verification - The identical five-way split runs green on every pr-trusted run (10-15 minutes per shard today, including on PRs merged this evening). - The suite's own growth (slower chat-connector tests) is being addressed separately; this PR only removes the artificial cliff. ## Risks - Low risk: two more runners per verify run; no test content changes. If shard durations regress further, the cap fires again — which is the correct signal once shards have honest headroom. ## Model Used - Claude (Anthropic), model ID `claude-fable-5` (Claude Fable 5), extended thinking, tool use via Claude Code CLI. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- .github/workflows/release-verify.yml | 24 ++++++++++++++----- .../release-verify-workflow.test.mjs | 4 ++-- 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 7193f38c84..ecf71da950 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -58,18 +58,30 @@ jobs: fail-fast: false matrix: include: + # Five-way server split, matching pr-trusted.yml. Three shards sat + # at 17-19 minutes against this job's 20-minute cap after the + # server suite grew on 2026-09-10, and every canary that evening + # died on the timeout instead of reporting a verdict. - group: general-server - group_label: server (1/3) + group_label: server (1/5) shard_index: 0 - shard_count: 3 + shard_count: 5 - group: general-server - group_label: server (2/3) + group_label: server (2/5) shard_index: 1 - shard_count: 3 + shard_count: 5 - group: general-server - group_label: server (3/3) + group_label: server (3/5) shard_index: 2 - shard_count: 3 + shard_count: 5 + - group: general-server + group_label: server (4/5) + shard_index: 3 + shard_count: 5 + - group: general-server + group_label: server (5/5) + shard_index: 4 + shard_count: 5 # Keep parity with pr.yml: workspaces-a is split with Vitest's # native --shard because the ui project dominates the lane. - group: general-workspaces-a diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 8b18a89039..cc725f15ab 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -214,11 +214,11 @@ test("release verify workflow covers the same split test surface as stable PR ve assert.match(verifyWorkflow, new RegExp(`group: ${group}`)); } - for (const shardIndex of [0, 1, 2]) { + for (const shardIndex of [0, 1, 2, 3, 4]) { assert.match( verifyWorkflow, new RegExp( - `group: general-server[\\s\\S]*?shard_index: ${shardIndex}[\\s\\S]*?shard_count: 3`, + `group: general-server[\\s\\S]*?shard_index: ${shardIndex}[\\s\\S]*?shard_count: 5`, ), ); }