From 37d7dfb0e3bdd962cb55ee6ca0d80c27c49768cf Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 11 Sep 2026 15:56:21 -0700 Subject: [PATCH] ci: allow dependency changes in cloud eval verification (#13286) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud deployment requires source verification for the exact merged commit. > - Contributor PRs leave lockfile updates to a separate bot PR. > - Most release checks can refresh an outdated lockfile while installing dependencies. > - Two Runner checks still require a frozen lockfile and fail after dependency changes. > - This PR gives those checks the same install policy as the other release checks. > - A valid dependency change can become deployable without waiting for another merge. ## Linked Issues or Issue Description Refs #13257. The dependency change in #13256 exposed this gap. The separate lockfile update is #13279. Related #12115 addresses the bot PR check trigger; this PR fixes exact-source cloud verification itself. **What happened?** [Cloud readiness for 2083bf6](https://github.com/paperclipai/paperclip/actions/runs/34651761811) failed in the Runner scorer and chaos jobs with `ERR_PNPM_OUTDATED_LOCKFILE`. The commit added `svix` to server dependencies. The tracked lockfile still describes the previous manifest. The other release checks install with `--no-frozen-lockfile`. **Expected behavior** Every source check installs and tests the same checked-out commit. A pending bot lockfile PR must not block cloud readiness. **Steps to reproduce** 1. Check out master commit 250deab, which retains the manifest/lockfile mismatch. 2. Run `pnpm install --ignore-scripts --frozen-lockfile`. It fails with the same outdated-lockfile error. 3. Run `pnpm install --ignore-scripts --no-frozen-lockfile --resolution-only`. It succeeds. 4. Restore the generated lockfile. This PR does not commit it. ## What Changed - Use `--no-frozen-lockfile` in the release Runner scorer job. - Use the same option in the reusable Runner chaos workflow. - Document why cloud source checks allow a job-local lockfile refresh. - Update the existing Runner scorer workflow assertion to match its install policy. ## Verification - All 457 workflow tests pass across `.github/scripts/tests/*.test.mjs` and `scripts/__tests__/release-verify-workflow.test.mjs`. - `actionlint` passes for both changed workflows. - Reproduced the frozen install failure against the real tracked manifest and lockfile. The refresh command passes in 4.6 seconds. - `git diff --check` passes. No lockfile changes remain. - No application source changes. Full local application typecheck, build, and test commands were not rerun in this dependency-free workflow worktree. Current-head GitHub CI must pass before merge. - After merge, verify both affected jobs pass on the exact master source even if the lockfile bot PR remains pending. ## Risks pnpm can resolve allowed dependency ranges when a manifest outgrows the tracked lockfile. This matches the existing release install policy. The resulting lockfile stays in the job workspace. Verification commands and runner routing are unchanged. The security reviewer explicitly accepted this existing dependency-policy tradeoff for both jobs after reviewing repository policy and the source/authorization checks. A future shared immutable dependency artifact would improve reproducibility across jobs. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] Local verification passes: all 457 workflow tests, actionlint, and the stale-lockfile reproduction described above. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/release-verify.yml | 2 +- .github/workflows/runner-chaos-evals.yml | 2 +- doc/cloud-build-readiness.md | 7 +++++++ scripts/__tests__/release-verify-workflow.test.mjs | 2 +- 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index 38faeff353..442d198f20 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -256,7 +256,7 @@ jobs: cache: pnpm - name: Install dependencies - run: pnpm install --frozen-lockfile + run: pnpm install --no-frozen-lockfile - name: Run deterministic Runner workflow scorer tests run: pnpm test:runner-workflow-evals diff --git a/.github/workflows/runner-chaos-evals.yml b/.github/workflows/runner-chaos-evals.yml index 65a453ee8c..91438bec3a 100644 --- a/.github/workflows/runner-chaos-evals.yml +++ b/.github/workflows/runner-chaos-evals.yml @@ -43,7 +43,7 @@ jobs: cache: pnpm - name: Install dependencies - run: pnpm install --frozen-lockfile + run: pnpm install --no-frozen-lockfile - name: Build eval and Runner contracts run: | diff --git a/doc/cloud-build-readiness.md b/doc/cloud-build-readiness.md index e6197ed595..9bb0bee1d4 100644 --- a/doc/cloud-build-readiness.md +++ b/doc/cloud-build-readiness.md @@ -49,6 +49,13 @@ job still runs one test worker. The partition covers every suite exactly once; normal PR and local test groups keep their existing shape. More jobs increase concurrent runner demand, so compare queue time as well as test duration. +All release verification installs, including the Runner scorer and chaos evals, +allow pnpm to refresh an outdated lockfile. Contributor PRs leave lockfile updates +to the separate refresh bot, so a dependency-changing master commit can arrive +before that bot's PR merges. Verification must install and test that commit +without waiting for another merge. The generated lockfile stays in the job's +workspace; these checks do not commit it back to the repository. + The artifact wait runs for up to 30 minutes and reports what is missing. Only an HTTP 404 means publication is pending; authorization errors, upstream outages, and identity mismatches fail the job. A failed, cancelled, or skipped prerequisite diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 08e88a2e40..702edcab8f 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -233,7 +233,7 @@ test("release verify workflow covers the same split test surface as stable PR ve ); assert.match( verifyWorkflow, - /runner_workflow_evals:[\s\S]*?Install dependencies\n\s+run: pnpm install --frozen-lockfile[\s\S]*?Run deterministic Runner workflow scorer tests/, + /runner_workflow_evals:[\s\S]*?Install dependencies\n\s+run: pnpm install --no-frozen-lockfile[\s\S]*?Run deterministic Runner workflow scorer tests/, ); assert.match(verifyWorkflow, /pnpm test:runner-workflow-evals/);