From 2da6a248c3e0cd7d5afbf3590b7bfa00df6bb297 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 11 Aug 2026 08:48:13 -0700 Subject: [PATCH] fix(release): surface recovery commands when a lane tag push is rejected (#11208) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release subsystem's promotion lanes publish to npm, then push a lane tag and dispatch the Docker image build at that tag > - The first nightly of the beta-tooling merge published to npm and then died at the tag push: GITHUB_TOKEN may not create refs pointing at workflow-modifying commits from dispatch or scheduled runs > - The failure was a bare `remote rejected` with no guidance, leaving the release half-finished (npm live, no tag, no images) until an operator reverse-engineered the recovery > - This pull request makes every lane's tag push degrade into exact recovery instructions in the job summary > - The benefit is that a rare platform-permission rejection becomes a two-minute runbook operation instead of a forensic exercise ## Linked Issues or Issue Description Refs #11008 — the incident occurred promoting that change's own merge commit, the first workflow-modifying commit to flow through the lanes it introduced. **Subsystem affected** Release automation: `.github/workflows/release.yml`, `doc/RELEASING.md`, workflow wiring tests. **Problem or motivation** Run 31445344811 published `2026.811.0-nightly.0` to npm, then failed pushing `nightly/v2026.811.0-nightly.0`: `refusing to allow a GitHub App to create or update workflow .github/workflows/release.yml without workflows permission`. The tagged commit modifies workflow files, and GITHUB_TOKEN may not create refs pointing at such commits from dispatch or scheduled runs (push-event runs are exempt, which is why the canary tag on the same commit succeeded). The job failed with no explanation and the Docker dispatch never ran. **Proposed solution** Wrap the nightly, beta, and stable tag pushes: on rejection, write the exact recovery commands into the job summary — create and push the tag with maintainer credentials, dispatch `docker.yml` at the tag, and for stable also run `create-github-release.sh` — then fail the job. Document the cause and recovery in the failure playbooks and pin the three recovery blocks with a wiring test. ## What Changed - `.github/workflows/release.yml`: recovery-summary wrappers on the nightly, beta, and stable tag-push steps - `doc/RELEASING.md`: failure-playbook entry for the workflows-permission rejection - `scripts/__tests__/release-verify-workflow.test.mjs`: wiring test asserting all three lanes carry the recovery summary ## Verification - Wiring tests: 6 pass; YAML parse of the workflow - The recovery commands are exactly the ones used to resolve the real incident (tag push + `docker.yml` dispatch for `nightly/v2026.811.0-nightly.0`) ## Risks - Low. The happy path is unchanged (a successful push skips the wrapper); the failure path trades a bare error for actionable output and still fails the job, since the release state is genuinely incomplete ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with extended thinking and full tool use. All changes model-authored under human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending — will confirm before merge) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending — will confirm before merge) - [x] I will address all Greptile and reviewer comments before requesting merge --- .github/workflows/release.yml | 61 ++++++++++++++++++- doc/RELEASING.md | 11 ++++ .../release-verify-workflow.test.mjs | 10 +++ 3 files changed, 79 insertions(+), 3 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c8e74ea146..3e07b36372 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -307,7 +307,25 @@ jobs: echo "Error: no nightly tag points at HEAD after release." >&2 exit 1 fi - git push origin "refs/tags/${tag}" + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN @@ -485,7 +503,25 @@ jobs: echo "Error: no beta tag points at HEAD after release." >&2 exit 1 fi - git push origin "refs/tags/${tag}" + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN @@ -743,7 +779,26 @@ jobs: echo "Error: no stable tag points at HEAD after release." >&2 exit 1 fi - git push origin "refs/tags/${tag}" + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo "./scripts/create-github-release.sh ${tag#v}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN diff --git a/doc/RELEASING.md b/doc/RELEASING.md index a77ca18729..f6f3e6d41c 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -380,6 +380,17 @@ force one: dispatch `release.yml` with `channel: nightly` (optionally pinning If the nightly published to npm but the tag push or Docker dispatch failed, push the `nightly/v*` tag manually and run `docker.yml` at that tag. +### If a tag push is rejected with a workflows-permission error + +GITHUB_TOKEN may not create refs that point at commits which modify workflow +files when the run was started by dispatch or schedule (push-triggered runs +are exempt, which is why canary tags on the same commit succeed). The npm +publish is already complete and correct when this happens. The failed job's +summary contains the exact recovery commands: create and push the tag with +maintainer credentials, then dispatch `docker.yml` at the tag (and for +stable, run `create-github-release.sh`). This only occurs when a +release-infrastructure commit itself becomes a promotion source. + ### If a beta looks bad during soak Do not promote it to stable. Fix forward: land the fix on `master`, let it diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs index 02de233cb6..df30f9e953 100644 --- a/scripts/__tests__/release-verify-workflow.test.mjs +++ b/scripts/__tests__/release-verify-workflow.test.mjs @@ -47,6 +47,16 @@ test("promotion selection guards against sources that predate their channel tool assert.match(releaseWorkflow, /git show "\$\{sha\}:scripts\/release\.sh" \| grep -qF 'canary\|nightly\|beta\|stable\)'/); }); +test("every lane's tag push degrades to recovery instructions when rejected", () => { + const releaseWorkflow = readWorkflow("release.yml"); + + // GITHUB_TOKEN may not create refs pointing at workflow-modifying commits + // from dispatch or scheduled runs; a rejected tag push after a successful + // npm publish must surface runbook recovery commands, not a bare error. + const occurrences = releaseWorkflow.match(/## Tag push rejected/g) ?? []; + assert.equal(occurrences.length, 3, "nightly, beta, and stable each carry the recovery summary"); +}); + test("release smoke workflow extends the container readiness budget for CI", () => { const smokeWorkflow = readWorkflow("release-smoke.yml"); const harness = readFileSync(path.join(repoRoot, "scripts/docker-onboard-smoke.sh"), "utf8");