ci: dispatch a Docker build for every canary tag (#12950)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Every master merge publishes an npm canary, and downstream managed deployments consume the matching `sha-<short>-cloud` Docker image. > - The canary image relies on the master-push `docker.yml` run, whose single pending concurrency slot is superseded by every newer push. > - On a busy day the image build never survives: five consecutive canaries shipped npm packages with no cloud image on 2026-09-06, and downstream deploys starved for ~18 hours while master kept moving. > - The nightly and beta lanes already solve exactly this by dispatching `docker.yml` at their tag ref after pushing it. > - This pull request wires the canary lane into the same mechanism, so every canary tag gets an image build no master push can supersede. > - The benefit is that a published canary always has its images, and downstream release resolution stops walking back to day-old builds during busy merge windows. ## Linked Issues or Issue Description Refs #12769 / #12855 (the earlier image-publishing incident in the same pipeline, different failure mode). **What happened?** `docker.yml` serialises per ref with one pending slot (`concurrency: docker-${{ github.ref }}`, `cancel-in-progress: false`). Master pushes arriving faster than the ~50-minute build supersede the pending build indefinitely. On 2026-09-06, canaries `2026.906.0-canary.1` through `.3` (and the commits between) published to npm with no `sha-<short>-cloud` image on GHCR — the Docker run list shows `cancelled, cancelled, cancelled` for their commits. Downstream managed rollouts correctly refused to deploy image-less canaries and pinned at `canary.0` for ~18 hours. **Expected behavior** Every published canary has its Docker images. A busy merge window must not be able to prevent image publication for tagged releases. **Steps to reproduce** 1. Merge to master more often than the Docker build takes to complete, for several hours. 2. Observe npm canaries advancing while every `Docker` run for their commits completes `cancelled`. 3. Observe `ghcr.io/paperclipai/paperclip:sha-<short>-cloud` returning 404 for each of those canaries. **Paperclip version or commit** Master at `83987210` (diagnosis time); the starved canaries were `2026.906.0-canary.1`–`.3`. **Deployment mode** GitHub Actions release + image pipeline; consumed by managed cloud deployments. ## What Changed - `.github/workflows/release.yml`, `publish_canary` job: - New step after "Push canary tag": dispatch `docker.yml` at `refs/tags/canary/v<version>` — identical to the nightly and beta lanes' existing step, including the step-summary note. The dispatched run keys its concurrency off the tag ref, so master pushes cannot supersede it, and `docker.yml`'s `type=sha` mapping publishes `sha-<short>` / `sha-<short>-cloud` for any ref. - The job gains `actions: write`, mirroring `publish_nightly`. - No `docker.yml` changes: it already accepts `workflow_dispatch` for exactly this pattern. - No dry-run guard needed: `publish_canary` runs only on `push` events, so the `dry_run` dispatch input cannot reach it. ## Verification - YAML lints clean; the step is a line-for-line mirror of the proven nightly-lane step (tag source swapped for the job's existing `canary_tag` output). - The concurrency claim is docker.yml's own documented behavior: groups are per-ref, and a tag ref is distinct from `refs/heads/master`. - Not run: a live canary publish — the next master merge after this lands exercises it end to end; the worst case (double build for a canary whose master-push run also survives) is benign, since both runs push identical content-addressed tags. ## Risks - Low. Roughly one additional Docker build per canary on busy days (on quiet days the master-push run and the dispatched run both execute — duplicate work, identical images, no conflict since the sha tags are content-equivalent for the same commit). - `actions: write` on `publish_canary` matches the grant `publish_nightly` already carries for the same purpose. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Fable 5 (Anthropic, model id `claude-fable-5`), extended thinking, agentic tool use in Claude Code: GHCR manifest probing, Actions run-list forensics, and workflow-lane comparison. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (workflow-only change; YAML linted; no test harness targets release.yml) - [x] I have added or updated tests where applicable (not applicable — CI wiring mirroring an existing proven lane) - [x] I have updated relevant documentation to reflect my changes (the step's inline comment documents the invariant) - [x] I have considered and documented the risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Devin Foley
GitHub
parent
7ed122911b
commit
8f099c3f83
|
|
@ -20,7 +20,13 @@ permissions:
|
|||
|
||||
# Serialise builds per ref without killing an in-flight one: a newer push
|
||||
# supersedes only the pending slot, so the image build that is already
|
||||
# running always finishes and publishes.
|
||||
# running always finishes and publishes. Canary TAG refs each get their
|
||||
# own group on purpose: their builds run in parallel so every published
|
||||
# canary gets its sha images regardless of merge cadence. The mutable
|
||||
# `:canary` channel tags are NOT written by the build matrix (which
|
||||
# would race across parallel runs) — each canary-tag run retags the
|
||||
# channel afterwards, only if it still matches the npm `canary`
|
||||
# dist-tag, so the channel moves monotonically and always mirrors npm.
|
||||
concurrency:
|
||||
group: docker-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
|
@ -150,16 +156,18 @@ jobs:
|
|||
echo "last=${last}" >> "$GITHUB_OUTPUT"
|
||||
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Lane tag mapping: master pushes publish `:canary`, nightly/v* tags
|
||||
# publish `:nightly`, and only stable v* tags move `:latest` and the
|
||||
# versioned tags. `:sha-<short>` is published on every build.
|
||||
# Lane tag mapping: nightly/v* tags publish `:nightly`, and only
|
||||
# stable v* tags move `:latest` and the versioned tags.
|
||||
# `:sha-<short>` is published on every build. `:canary` is
|
||||
# deliberately absent here — the channel tag is moved by the
|
||||
# dist-tag-checked retag step below, never by the build matrix,
|
||||
# so parallel canary builds cannot race it backwards.
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: ghcr.io/${{ github.repository }}
|
||||
tags: |
|
||||
type=raw,value=canary,enable=${{ github.ref == 'refs/heads/master' }}
|
||||
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
|
||||
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
|
||||
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
||||
|
|
@ -201,6 +209,7 @@ jobs:
|
|||
# build: the step above is multi-arch with `push: true`, so nothing is
|
||||
# loaded into the runner's daemon. The cloud variant is FROM production
|
||||
# and inherits the same ENTRYPOINT, so checking this image covers both.
|
||||
|
||||
- name: Verify PID 1 reaps orphaned processes
|
||||
env:
|
||||
# Through the environment, not interpolated into the script body, so
|
||||
|
|
@ -345,8 +354,9 @@ jobs:
|
|||
echo "count=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Published under the same lane tag set as the self-hosted image, with a
|
||||
# `-cloud` suffix (canary-cloud, nightly-cloud, latest-cloud,
|
||||
# <version>-cloud, sha-<short>-cloud).
|
||||
# `-cloud` suffix (nightly-cloud, latest-cloud, <version>-cloud,
|
||||
# sha-<short>-cloud). `:canary-cloud` follows the same retag-step
|
||||
# ownership rule as `:canary` above.
|
||||
- name: Docker meta (cloud)
|
||||
id: meta-cloud
|
||||
uses: docker/metadata-action@v6
|
||||
|
|
@ -355,7 +365,6 @@ jobs:
|
|||
flavor: |
|
||||
suffix=-cloud,onlatest=true
|
||||
tags: |
|
||||
type=raw,value=canary,enable=${{ github.ref == 'refs/heads/master' }}
|
||||
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
|
||||
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
|
||||
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
|
||||
|
|
@ -401,6 +410,7 @@ jobs:
|
|||
# module resolution walks. Verify the image this job just pushed, not
|
||||
# a local build, so a build-cache or layer-ordering regression is
|
||||
# caught before any tenant runs the image.
|
||||
|
||||
- name: Verify the pushed image resolves the declared Sentry version
|
||||
env:
|
||||
IMAGE_TAGS: ${{ steps.meta-cloud.outputs.tags }}
|
||||
|
|
@ -423,3 +433,58 @@ jobs:
|
|||
exit 1
|
||||
fi
|
||||
echo "The pushed image resolves the declared @sentry/node version."
|
||||
|
||||
# Moves the mutable `:canary` / `:canary-cloud` channel tags. Kept OUT
|
||||
# of the build jobs and serialized in its own lane, and — the load-
|
||||
# bearing property — CONVERGENT rather than self-interested: a
|
||||
# promotion does not promote "its own" canary, it retags the channel
|
||||
# to whatever the npm `canary` dist-tag names at execution time,
|
||||
# provided that version's sha images are published. GitHub's shared
|
||||
# concurrency lane keeps one running and one pending promotion and
|
||||
# REPLACES the pending slot with the latest enqueued — an older build
|
||||
# finishing late can therefore evict the newest canary's pending
|
||||
# promotion. With convergent promotion that eviction is harmless:
|
||||
# whichever promotion survives resolves the current dist-tag fresh
|
||||
# and lands the channel there (the current canary's images always
|
||||
# exist by the time any later promotion runs, because per-tag build
|
||||
# groups mean canary builds are never superseded and each run's
|
||||
# promotion is gated on its own completed pushes). Every interleaving
|
||||
# converges the Docker channel onto the npm channel.
|
||||
promote_canary_channel:
|
||||
if: startsWith(github.ref, 'refs/tags/canary/v')
|
||||
needs: [build-and-push, build-and-push-cloud]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
concurrency:
|
||||
group: docker-canary-channel-promotion
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.repository_owner }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Converge the channel tags onto the current npm canary
|
||||
env:
|
||||
IMAGE: ghcr.io/${{ github.repository }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
current="$(curl -fsS "https://registry.npmjs.org/-/package/@paperclipai%2Fdb/dist-tags" | jq -er .canary)"
|
||||
sha="$(gh api "repos/${GITHUB_REPOSITORY}/commits/$(printf 'canary/v%s' "$current" | jq -sRr @uri)" --jq .sha 2>/dev/null || true)"
|
||||
if [ -z "$sha" ]; then
|
||||
echo "canary/v${current} does not resolve yet; a later promotion converges the channel"
|
||||
exit 0
|
||||
fi
|
||||
short="$(printf '%s' "$sha" | cut -c1-7)"
|
||||
if ! docker buildx imagetools inspect "$IMAGE:sha-${short}-cloud" >/dev/null 2>&1; then
|
||||
echo "images for ${current} (sha-${short}) not published yet; its own promotion converges the channel"
|
||||
exit 0
|
||||
fi
|
||||
docker buildx imagetools create -t "$IMAGE:canary" "$IMAGE:sha-${short}"
|
||||
docker buildx imagetools create -t "$IMAGE:canary-cloud" "$IMAGE:sha-${short}-cloud"
|
||||
echo "channel tags moved to canary ${current} (sha-${short})"
|
||||
|
|
|
|||
|
|
@ -304,6 +304,8 @@ jobs:
|
|||
permissions:
|
||||
contents: write
|
||||
id-token: write
|
||||
# For the explicit docker.yml dispatch below.
|
||||
actions: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
|
|
@ -362,6 +364,29 @@ jobs:
|
|||
git push origin "refs/tags/${tag}"
|
||||
echo "version=${tag#canary/v}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Canary images previously relied on the master-push docker.yml run,
|
||||
# whose single pending concurrency slot gets superseded by every
|
||||
# newer push — on a busy day no canary image publishes at all (five
|
||||
# consecutive canaries shipped npm packages with no cloud image on
|
||||
# 2026-09-06, starving downstream managed deploys for ~18 hours).
|
||||
# Tag pushes made with GITHUB_TOKEN do not fire docker.yml's
|
||||
# triggers, so dispatch the image build at the canary tag
|
||||
# explicitly, exactly like the nightly and beta lanes: the run keys
|
||||
# its concurrency off the tag ref, so no master push can supersede
|
||||
# it, and docker.yml's `type=sha` mapping publishes the
|
||||
# sha-<short> and sha-<short>-cloud images either way.
|
||||
- name: Build Docker images for the canary tag
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
{
|
||||
echo "## Canary published"
|
||||
echo ""
|
||||
echo "- Published canary: \`${{ steps.canary_tag.outputs.version }}\`"
|
||||
echo "- Docker build dispatched at \`canary/v${{ steps.canary_tag.outputs.version }}\`"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
gh workflow run docker.yml --ref "refs/tags/canary/v${{ steps.canary_tag.outputs.version }}" --repo "$GITHUB_REPOSITORY"
|
||||
|
||||
# The package is already public when this gate runs. A red result leaves the
|
||||
# immutable canary in npm, but makes the release workflow visibly fail before
|
||||
# anyone mistakes an installable package for an onboardable one.
|
||||
|
|
|
|||
Loading…
Reference in New Issue