paperclip/tests/release-smoke
Devin Foley 4042eb1c48
test(release-smoke): follow the connect-step source question and the first-task chat (#13166)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release pipeline promotes canary → nightly → beta → stable, and
the nightly lane is gated by the Docker release smoke, a Playwright walk
of first-run onboarding against the exact published artifact
> - Onboarding changed twice since the smoke was last updated: the
connect step now opens as a model-source question (#12796, #12801), and
the seeded first task now opens as a chat with the lead that
deliberately creates no run until the user answers (#13068)
> - The smoke still waited for an immediate "Connect" button and then
polled for an assignment-triggered heartbeat run, so it failed every
scheduled nightly since 2026-09-03 and blocked all nightly and beta
promotions
> - This pull request updates the smoke to follow the current arc: pick
the Claude source tile, press Connect, launch, then assert the seeded
chat greeting, the opening question card, and the absence of heartbeat
runs
> - The benefit is a release pipeline that can promote current master
again, with the smoke asserting the product's current contract instead
of a removed one

## Linked Issues or Issue Description

**What happened?**

The scheduled nightly lane of `release.yml` has failed every night since
2026-09-03. The `smoke_nightly / smoke` job fails in
`tests/release-smoke/docker-auth-onboarding.spec.ts` at
`expect(connectButton).toBeVisible()`. No nightly has published since
`2026.902.0-nightly.0`, so no beta can promote recent master.

**Expected behavior**

The release smoke follows the current onboarding arc and passes against
a healthy published artifact. The nightly lane promotes the newest green
canary each night.

**Steps to reproduce**

1. Run `PAPERCLIPAI_VERSION=2026.910.0-canary.5 SMOKE_DETACH=true
./scripts/docker-onboard-smoke.sh`.
2. Run `pnpm run test:release-smoke` against the container with the
previous spec.
3. The spec times out waiting for a "Connect" button. The step now shows
a model-source tile row first, and after launch the seeded task is a
chat with no heartbeat run.

**Paperclip version or commit**

Reproduced against published `paperclipai@2026.910.0-canary.5`; spec
updated on current `master`.

## What Changed

- The spec answers the connect step's model-source question: it asserts
the "Connect a model" heading, picks the Claude tile from the "Model
source" radiogroup, and only then waits for the "Connect" footer button
(#12796, #12801 rebuilt the step around that question).
- The spec replaces the assignment-run poll with the first-task chat
contract from #13068: it asserts the deterministic greeting ("Welcome to
Paperclip!"), the opening question card ("What would you like to do?"),
and that the lead has zero heartbeat runs, because launch must not wake
the assignee before the user answers.

## Verification

- Launched the CI harness locally: `scripts/docker-onboard-smoke.sh`
with `PAPERCLIPAI_VERSION=2026.910.0-canary.5` (the newest canary, the
one the next nightly would promote).
- `pnpm run test:release-smoke` against that container: 1 passed.
- The previous spec against the same container reproduces the CI failure
mode first (Connect-button wait), and after the connect-step fix, the
run-poll failure — both match the nightly logs.

## Risks

- Low risk: the change touches only the release smoke spec. If
onboarding's copy for the greeting or the question card changes, the
smoke fails loudly at that assertion, which is this suite's job.

## Model Used

- Claude Fable 5 (`claude-fable-5`), via Claude Code CLI, extended
thinking and tool use enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-10 12:33:57 -07:00
..
docker-auth-onboarding.spec.ts test(release-smoke): follow the connect-step source question and the first-task chat (#13166) 2026-09-10 12:33:57 -07:00
playwright.config.ts ci: use runner Chrome for headless workflows (#6967) 2026-05-29 00:18:52 -07:00