test(release-smoke): follow the connect-step source question and the first-task chat (#13166)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release pipeline promotes canary → nightly → beta → stable, and the nightly lane is gated by the Docker release smoke, a Playwright walk of first-run onboarding against the exact published artifact > - Onboarding changed twice since the smoke was last updated: the connect step now opens as a model-source question (#12796, #12801), and the seeded first task now opens as a chat with the lead that deliberately creates no run until the user answers (#13068) > - The smoke still waited for an immediate "Connect" button and then polled for an assignment-triggered heartbeat run, so it failed every scheduled nightly since 2026-09-03 and blocked all nightly and beta promotions > - This pull request updates the smoke to follow the current arc: pick the Claude source tile, press Connect, launch, then assert the seeded chat greeting, the opening question card, and the absence of heartbeat runs > - The benefit is a release pipeline that can promote current master again, with the smoke asserting the product's current contract instead of a removed one ## Linked Issues or Issue Description **What happened?** The scheduled nightly lane of `release.yml` has failed every night since 2026-09-03. The `smoke_nightly / smoke` job fails in `tests/release-smoke/docker-auth-onboarding.spec.ts` at `expect(connectButton).toBeVisible()`. No nightly has published since `2026.902.0-nightly.0`, so no beta can promote recent master. **Expected behavior** The release smoke follows the current onboarding arc and passes against a healthy published artifact. The nightly lane promotes the newest green canary each night. **Steps to reproduce** 1. Run `PAPERCLIPAI_VERSION=2026.910.0-canary.5 SMOKE_DETACH=true ./scripts/docker-onboard-smoke.sh`. 2. Run `pnpm run test:release-smoke` against the container with the previous spec. 3. The spec times out waiting for a "Connect" button. The step now shows a model-source tile row first, and after launch the seeded task is a chat with no heartbeat run. **Paperclip version or commit** Reproduced against published `paperclipai@2026.910.0-canary.5`; spec updated on current `master`. ## What Changed - The spec answers the connect step's model-source question: it asserts the "Connect a model" heading, picks the Claude tile from the "Model source" radiogroup, and only then waits for the "Connect" footer button (#12796, #12801 rebuilt the step around that question). - The spec replaces the assignment-run poll with the first-task chat contract from #13068: it asserts the deterministic greeting ("Welcome to Paperclip!"), the opening question card ("What would you like to do?"), and that the lead has zero heartbeat runs, because launch must not wake the assignee before the user answers. ## Verification - Launched the CI harness locally: `scripts/docker-onboard-smoke.sh` with `PAPERCLIPAI_VERSION=2026.910.0-canary.5` (the newest canary, the one the next nightly would promote). - `pnpm run test:release-smoke` against that container: 1 passed. - The previous spec against the same container reproduces the CI failure mode first (Connect-button wait), and after the connect-step fix, the run-poll failure — both match the nightly logs. ## Risks - Low risk: the change touches only the release smoke spec. If onboarding's copy for the greeting or the question card changes, the smoke fails loudly at that assertion, which is this suite's job. ## Model Used - Claude Fable 5 (`claude-fable-5`), via Claude Code CLI, extended thinking and tool use enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
parent
6728e133f8
commit
4042eb1c48
|
|
@ -109,12 +109,26 @@ test.describe("Docker authenticated onboarding smoke", () => {
|
|||
await expect(nextButton).toBeEnabled({ timeout: 10_000 });
|
||||
await nextButton.click();
|
||||
|
||||
// Step 4: keep the default adapter and connect (hire) the lead. Connect
|
||||
// probes the adapter environment first and blocks the hire on a `fail`. In
|
||||
// the smoke container no agent CLI is installed, which the probe reports as
|
||||
// a warning rather than an error, so the hire proceeds — a genuine failure
|
||||
// here means the published artifact cannot hire on a clean machine. Allow
|
||||
// generous time for the probe + hire + auto-approval.
|
||||
// Step 4: answer the model-source question, then connect (hire) the lead.
|
||||
// The step now opens as a row of source tiles and the footer button has
|
||||
// nothing to do until one is picked (#12796/#12801 rebuilt the step around
|
||||
// that question); picking Claude collapses the row and turns the button
|
||||
// into "Connect". In the smoke container the resolved login environment is
|
||||
// the local host, not a sandbox, so there is no sign-in to run and Connect
|
||||
// goes straight to the hire, with the missing agent CLI reported by the
|
||||
// probe as a warning rather than an error — a genuine failure here means
|
||||
// the published artifact cannot hire on a clean machine. Allow generous
|
||||
// time for the probe + hire + auto-approval.
|
||||
await expect(
|
||||
page.getByRole("heading", { name: "Connect a model" })
|
||||
).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
const claudeSourceTile = page
|
||||
.getByRole("radiogroup", { name: "Model source" })
|
||||
.getByRole("radio", { name: /Claude/ });
|
||||
await expect(claudeSourceTile).toBeVisible({ timeout: 10_000 });
|
||||
await claudeSourceTile.click();
|
||||
|
||||
const connectButton = page.getByRole("button", {
|
||||
name: "Connect",
|
||||
exact: true,
|
||||
|
|
@ -178,31 +192,28 @@ test.describe("Docker authenticated onboarding smoke", () => {
|
|||
true
|
||||
);
|
||||
|
||||
await expect.poll(
|
||||
async () => {
|
||||
const runs = await getJson<
|
||||
Array<{ agentId: string; invocationSource: string; status: string }>
|
||||
>(
|
||||
page,
|
||||
`${baseUrl}/api/companies/${company!.id}/heartbeat-runs?agentId=${leadAgent!.id}`
|
||||
);
|
||||
const latestRun = runs.find((entry) => entry.agentId === leadAgent!.id);
|
||||
return latestRun
|
||||
? {
|
||||
invocationSource: latestRun.invocationSource,
|
||||
status: latestRun.status,
|
||||
}
|
||||
: null;
|
||||
},
|
||||
{
|
||||
timeout: 30_000,
|
||||
intervals: [1_000, 2_000, 5_000],
|
||||
}
|
||||
).toEqual(
|
||||
expect.objectContaining({
|
||||
invocationSource: "assignment",
|
||||
status: expect.stringMatching(/^(queued|running|succeeded|failed)$/),
|
||||
})
|
||||
);
|
||||
// #13068 rebuilt the seeded first task as a chat with the lead: launch
|
||||
// posts a deterministic, server-owned greeting plus an opening question
|
||||
// card, and deliberately does not wake the assignee — "no run until the
|
||||
// user answers". Assert the chat actually opened (the greeting and the
|
||||
// card are seeded without an LLM, so their absence means the launch
|
||||
// half-finished) …
|
||||
await expect(
|
||||
page.getByText("Welcome to Paperclip!").first()
|
||||
).toBeVisible({ timeout: 20_000 });
|
||||
await expect(page.getByText("What would you like to do?")).toBeVisible();
|
||||
|
||||
// … and that the no-run contract holds. This spec used to poll for an
|
||||
// assignment-triggered heartbeat run here; a run appearing before the
|
||||
// user's first answer is now the regression, not the success. Wake
|
||||
// dispatch is asynchronous, so watch the endpoint over a bounded window
|
||||
// rather than sampling it once — a launch-time wake that slips through
|
||||
// lands well within this window.
|
||||
const runsUrl = `${baseUrl}/api/companies/${company!.id}/heartbeat-runs?agentId=${leadAgent!.id}`;
|
||||
const noRunDeadline = Date.now() + 15_000;
|
||||
while (Date.now() < noRunDeadline) {
|
||||
expect(await getJson<Array<{ id: string }>>(page, runsUrl)).toEqual([]);
|
||||
await page.waitForTimeout(1_000);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Reference in New Issue