fix(release): surface recovery commands when a lane tag push is rejected (#11208)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release subsystem's promotion lanes publish to npm, then push a lane tag and dispatch the Docker image build at that tag > - The first nightly of the beta-tooling merge published to npm and then died at the tag push: GITHUB_TOKEN may not create refs pointing at workflow-modifying commits from dispatch or scheduled runs > - The failure was a bare `remote rejected` with no guidance, leaving the release half-finished (npm live, no tag, no images) until an operator reverse-engineered the recovery > - This pull request makes every lane's tag push degrade into exact recovery instructions in the job summary > - The benefit is that a rare platform-permission rejection becomes a two-minute runbook operation instead of a forensic exercise ## Linked Issues or Issue Description Refs #11008 — the incident occurred promoting that change's own merge commit, the first workflow-modifying commit to flow through the lanes it introduced. **Subsystem affected** Release automation: `.github/workflows/release.yml`, `doc/RELEASING.md`, workflow wiring tests. **Problem or motivation** Run 31445344811 published `2026.811.0-nightly.0` to npm, then failed pushing `nightly/v2026.811.0-nightly.0`: `refusing to allow a GitHub App to create or update workflow .github/workflows/release.yml without workflows permission`. The tagged commit modifies workflow files, and GITHUB_TOKEN may not create refs pointing at such commits from dispatch or scheduled runs (push-event runs are exempt, which is why the canary tag on the same commit succeeded). The job failed with no explanation and the Docker dispatch never ran. **Proposed solution** Wrap the nightly, beta, and stable tag pushes: on rejection, write the exact recovery commands into the job summary — create and push the tag with maintainer credentials, dispatch `docker.yml` at the tag, and for stable also run `create-github-release.sh` — then fail the job. Document the cause and recovery in the failure playbooks and pin the three recovery blocks with a wiring test. ## What Changed - `.github/workflows/release.yml`: recovery-summary wrappers on the nightly, beta, and stable tag-push steps - `doc/RELEASING.md`: failure-playbook entry for the workflows-permission rejection - `scripts/__tests__/release-verify-workflow.test.mjs`: wiring test asserting all three lanes carry the recovery summary ## Verification - Wiring tests: 6 pass; YAML parse of the workflow - The recovery commands are exactly the ones used to resolve the real incident (tag push + `docker.yml` dispatch for `nightly/v2026.811.0-nightly.0`) ## Risks - Low. The happy path is unchanged (a successful push skips the wrapper); the failure path trades a bare error for actionable output and still fails the job, since the release state is genuinely incomplete ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with extended thinking and full tool use. All changes model-authored under human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending — will confirm before merge) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending — will confirm before merge) - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Devin Foley
GitHub
parent
1ea2f0e2d6
commit
2da6a248c3
|
|
@ -307,7 +307,25 @@ jobs:
|
|||
echo "Error: no nightly tag points at HEAD after release." >&2
|
||||
exit 1
|
||||
fi
|
||||
git push origin "refs/tags/${tag}"
|
||||
if ! git push origin "refs/tags/${tag}"; then
|
||||
sha="$(git rev-parse HEAD)"
|
||||
{
|
||||
echo "## Tag push rejected"
|
||||
echo ""
|
||||
echo "The npm publish succeeded, but pushing \`${tag}\` was rejected."
|
||||
echo "This usually means the tagged commit modifies workflow files,"
|
||||
echo "which GITHUB_TOKEN may not reference when creating refs from"
|
||||
echo "dispatch or scheduled runs. Recover with maintainer credentials:"
|
||||
echo ""
|
||||
echo '```'
|
||||
echo "git tag ${tag} ${sha}"
|
||||
echo "git push origin refs/tags/${tag}"
|
||||
echo "gh workflow run docker.yml --ref refs/tags/${tag}"
|
||||
echo '```'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "::error::Tag push rejected; see the job summary for recovery commands." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag
|
||||
# trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN
|
||||
|
|
@ -485,7 +503,25 @@ jobs:
|
|||
echo "Error: no beta tag points at HEAD after release." >&2
|
||||
exit 1
|
||||
fi
|
||||
git push origin "refs/tags/${tag}"
|
||||
if ! git push origin "refs/tags/${tag}"; then
|
||||
sha="$(git rev-parse HEAD)"
|
||||
{
|
||||
echo "## Tag push rejected"
|
||||
echo ""
|
||||
echo "The npm publish succeeded, but pushing \`${tag}\` was rejected."
|
||||
echo "This usually means the tagged commit modifies workflow files,"
|
||||
echo "which GITHUB_TOKEN may not reference when creating refs from"
|
||||
echo "dispatch or scheduled runs. Recover with maintainer credentials:"
|
||||
echo ""
|
||||
echo '```'
|
||||
echo "git tag ${tag} ${sha}"
|
||||
echo "git push origin refs/tags/${tag}"
|
||||
echo "gh workflow run docker.yml --ref refs/tags/${tag}"
|
||||
echo '```'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "::error::Tag push rejected; see the job summary for recovery commands." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag
|
||||
# trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN
|
||||
|
|
@ -743,7 +779,26 @@ jobs:
|
|||
echo "Error: no stable tag points at HEAD after release." >&2
|
||||
exit 1
|
||||
fi
|
||||
git push origin "refs/tags/${tag}"
|
||||
if ! git push origin "refs/tags/${tag}"; then
|
||||
sha="$(git rev-parse HEAD)"
|
||||
{
|
||||
echo "## Tag push rejected"
|
||||
echo ""
|
||||
echo "The npm publish succeeded, but pushing \`${tag}\` was rejected."
|
||||
echo "This usually means the tagged commit modifies workflow files,"
|
||||
echo "which GITHUB_TOKEN may not reference when creating refs from"
|
||||
echo "dispatch or scheduled runs. Recover with maintainer credentials:"
|
||||
echo ""
|
||||
echo '```'
|
||||
echo "git tag ${tag} ${sha}"
|
||||
echo "git push origin refs/tags/${tag}"
|
||||
echo "gh workflow run docker.yml --ref refs/tags/${tag}"
|
||||
echo "./scripts/create-github-release.sh ${tag#v}"
|
||||
echo '```'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "::error::Tag push rejected; see the job summary for recovery commands." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag
|
||||
# trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN
|
||||
|
|
|
|||
|
|
@ -380,6 +380,17 @@ force one: dispatch `release.yml` with `channel: nightly` (optionally pinning
|
|||
If the nightly published to npm but the tag push or Docker dispatch failed,
|
||||
push the `nightly/v*` tag manually and run `docker.yml` at that tag.
|
||||
|
||||
### If a tag push is rejected with a workflows-permission error
|
||||
|
||||
GITHUB_TOKEN may not create refs that point at commits which modify workflow
|
||||
files when the run was started by dispatch or schedule (push-triggered runs
|
||||
are exempt, which is why canary tags on the same commit succeed). The npm
|
||||
publish is already complete and correct when this happens. The failed job's
|
||||
summary contains the exact recovery commands: create and push the tag with
|
||||
maintainer credentials, then dispatch `docker.yml` at the tag (and for
|
||||
stable, run `create-github-release.sh`). This only occurs when a
|
||||
release-infrastructure commit itself becomes a promotion source.
|
||||
|
||||
### If a beta looks bad during soak
|
||||
|
||||
Do not promote it to stable. Fix forward: land the fix on `master`, let it
|
||||
|
|
|
|||
|
|
@ -47,6 +47,16 @@ test("promotion selection guards against sources that predate their channel tool
|
|||
assert.match(releaseWorkflow, /git show "\$\{sha\}:scripts\/release\.sh" \| grep -qF 'canary\|nightly\|beta\|stable\)'/);
|
||||
});
|
||||
|
||||
test("every lane's tag push degrades to recovery instructions when rejected", () => {
|
||||
const releaseWorkflow = readWorkflow("release.yml");
|
||||
|
||||
// GITHUB_TOKEN may not create refs pointing at workflow-modifying commits
|
||||
// from dispatch or scheduled runs; a rejected tag push after a successful
|
||||
// npm publish must surface runbook recovery commands, not a bare error.
|
||||
const occurrences = releaseWorkflow.match(/## Tag push rejected/g) ?? [];
|
||||
assert.equal(occurrences.length, 3, "nightly, beta, and stable each carry the recovery summary");
|
||||
});
|
||||
|
||||
test("release smoke workflow extends the container readiness budget for CI", () => {
|
||||
const smokeWorkflow = readWorkflow("release-smoke.yml");
|
||||
const harness = readFileSync(path.join(repoRoot, "scripts/docker-onboard-smoke.sh"), "utf8");
|
||||
|
|
|
|||
Loading…
Reference in New Issue