ci: allow dependency changes in cloud eval verification (#13286)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud deployment requires source verification for the exact merged
commit.
> - Contributor PRs leave lockfile updates to a separate bot PR.
> - Most release checks can refresh an outdated lockfile while
installing dependencies.
> - Two Runner checks still require a frozen lockfile and fail after
dependency changes.
> - This PR gives those checks the same install policy as the other
release checks.
> - A valid dependency change can become deployable without waiting for
another merge.

## Linked Issues or Issue Description

Refs #13257. The dependency change in #13256 exposed this gap. The
separate lockfile update is #13279. Related #12115 addresses the bot PR
check trigger; this PR fixes exact-source cloud verification itself.

**What happened?**

[Cloud readiness for
2083bf6](https://github.com/paperclipai/paperclip/actions/runs/34651761811)
failed in the Runner scorer and chaos jobs with
`ERR_PNPM_OUTDATED_LOCKFILE`. The commit added `svix` to server
dependencies. The tracked lockfile still describes the previous
manifest. The other release checks install with `--no-frozen-lockfile`.

**Expected behavior**

Every source check installs and tests the same checked-out commit. A
pending bot lockfile PR must not block cloud readiness.

**Steps to reproduce**

1. Check out master commit 250deab, which retains the manifest/lockfile
mismatch.
2. Run `pnpm install --ignore-scripts --frozen-lockfile`. It fails with
the same outdated-lockfile error.
3. Run `pnpm install --ignore-scripts --no-frozen-lockfile
--resolution-only`. It succeeds.
4. Restore the generated lockfile. This PR does not commit it.

## What Changed

- Use `--no-frozen-lockfile` in the release Runner scorer job.
- Use the same option in the reusable Runner chaos workflow.
- Document why cloud source checks allow a job-local lockfile refresh.
- Update the existing Runner scorer workflow assertion to match its
install policy.

## Verification

- All 457 workflow tests pass across `.github/scripts/tests/*.test.mjs`
and `scripts/__tests__/release-verify-workflow.test.mjs`.
- `actionlint` passes for both changed workflows.
- Reproduced the frozen install failure against the real tracked
manifest and lockfile. The refresh command passes in 4.6 seconds.
- `git diff --check` passes. No lockfile changes remain.
- No application source changes. Full local application typecheck,
build, and test commands were not rerun in this dependency-free workflow
worktree. Current-head GitHub CI must pass before merge.
- After merge, verify both affected jobs pass on the exact master source
even if the lockfile bot PR remains pending.

## Risks

pnpm can resolve allowed dependency ranges when a manifest outgrows the
tracked lockfile. This matches the existing release install policy. The
resulting lockfile stays in the job workspace. Verification commands and
runner routing are unchanged. The security reviewer explicitly accepted
this existing dependency-policy tradeoff for both jobs after reviewing
repository policy and the source/authorization checks. A future shared
immutable dependency artifact would improve reproducibility across jobs.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] Local verification passes: all 457 workflow tests, actionlint, and
the stale-lockfile reproduction described above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley 2026-09-11 15:56:21 -07:00 committed by GitHub
parent f2e2d38972
commit 37d7dfb0e3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 10 additions and 3 deletions

View File

@ -256,7 +256,7 @@ jobs:
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
run: pnpm install --no-frozen-lockfile
- name: Run deterministic Runner workflow scorer tests
run: pnpm test:runner-workflow-evals

View File

@ -43,7 +43,7 @@ jobs:
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
run: pnpm install --no-frozen-lockfile
- name: Build eval and Runner contracts
run: |

View File

@ -49,6 +49,13 @@ job still runs one test worker. The partition covers every suite exactly once;
normal PR and local test groups keep their existing shape. More jobs increase
concurrent runner demand, so compare queue time as well as test duration.
All release verification installs, including the Runner scorer and chaos evals,
allow pnpm to refresh an outdated lockfile. Contributor PRs leave lockfile updates
to the separate refresh bot, so a dependency-changing master commit can arrive
before that bot's PR merges. Verification must install and test that commit
without waiting for another merge. The generated lockfile stays in the job's
workspace; these checks do not commit it back to the repository.
The artifact wait runs for up to 30 minutes and reports what is missing. Only
an HTTP 404 means publication is pending; authorization errors, upstream outages,
and identity mismatches fail the job. A failed, cancelled, or skipped prerequisite

View File

@ -233,7 +233,7 @@ test("release verify workflow covers the same split test surface as stable PR ve
);
assert.match(
verifyWorkflow,
/runner_workflow_evals:[\s\S]*?Install dependencies\n\s+run: pnpm install --frozen-lockfile[\s\S]*?Run deterministic Runner workflow scorer tests/,
/runner_workflow_evals:[\s\S]*?Install dependencies\n\s+run: pnpm install --no-frozen-lockfile[\s\S]*?Run deterministic Runner workflow scorer tests/,
);
assert.match(verifyWorkflow, /pnpm test:runner-workflow-evals/);