test(release-smoke): update onboarding spec for the mission-first wizard (#11190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release subsystem's nightly lane gates every nightly publish on the release smoke suite, which drives real onboarding in a browser against the published artifact > - With the harness fixed (#11187, #11189), the gate reached the Playwright suite for the first time in CI — and the spec still walks the old onboarding wizard, so it fails at "Create your first agent" on every current build > - The wizard was redesigned to a mission-first five-step flow, and the spec rotted silently because the suite never ran in CI before > - This pull request rewrites the spec to drive the current wizard end to end > - The benefit is a smoke gate that actually tests today's product, verified against a real published canary ## Linked Issues or Issue Description **Subsystem affected** Release smoke testing: `tests/release-smoke/docker-auth-onboarding.spec.ts`. **Problem or motivation** Nightly run 31431273139 failed in the smoke Playwright suite: the spec expects the old wizard step "Create your first agent", but current builds show the redesigned mission-first flow (front door → company → mission → team lead → connect model → review). The page snapshot in the run artifact shows the "Define your mission" step where the spec expected the agent step. Both retries failed identically — this is deterministic spec drift, not flake. **Proposed solution** Rewrite the spec for the current flow: fill the company name, define the mission directly (confirming creates the company), name the team lead, hire it through the adapter step — the adapter environment probe reports unhealthy in the CLI-less smoke container by design and must not block the hire — then launch to the dashboard. Assert the company, the ceo-role agent, and the company goal through the API. The first-task and assignment-run assertions are removed together with the wizard flow that created them. ## What Changed - `tests/release-smoke/docker-auth-onboarding.spec.ts`: rewritten for the mission-first wizard; sign-in and wizard-opening helpers and the company-name step are unchanged ## Verification - Full local run against the real nightly candidate: launched the smoke container for `paperclipai@2026.810.0-canary.3` via `scripts/docker-onboard-smoke.sh` (with the #11189 bind fix), then ran `pnpm run test:release-smoke` against it — 1 passed (4.5s) - After merge: dispatch `release.yml` with `channel: nightly` to run the full gate in CI ## Risks - Low. Test-only change. The spec now asserts less about first-task creation because the wizard no longer creates a first task; if a first-run trigger returns to onboarding, the spec should grow that assertion back ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with extended thinking and full tool use (CI artifact forensics, UI source tracing, local Docker + Playwright reproduction and verification). All changes model-authored under human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending — will confirm before merge) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending — will confirm before merge) - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Devin Foley
GitHub
parent
f94f6003c6
commit
5a0985f80a
|
|
@ -10,8 +10,11 @@ const ADMIN_PASSWORD =
|
|||
"paperclip-smoke-password";
|
||||
|
||||
const COMPANY_NAME = `Release-Smoke-${Date.now()}`;
|
||||
const MISSION = "Ship a reliable release smoke suite for Paperclip.";
|
||||
const AGENT_NAME = "CEO";
|
||||
const TASK_TITLE = "Release smoke task";
|
||||
// Seeded by the wizard's launch step (DEFAULT_TASK_TITLE in
|
||||
// ui/src/components/OnboardingWizard.tsx).
|
||||
const FIRST_TASK_TITLE = "Hire your first engineer and create a hiring plan";
|
||||
|
||||
async function signIn(page: Page) {
|
||||
await page.goto("/");
|
||||
|
|
@ -38,39 +41,50 @@ async function openOnboarding(page: Page) {
|
|||
}
|
||||
|
||||
test.describe("Docker authenticated onboarding smoke", () => {
|
||||
test("logs in, completes onboarding, and triggers the first CEO run", async ({
|
||||
test("logs in, completes onboarding, and hires the lead agent", async ({
|
||||
page,
|
||||
}) => {
|
||||
await signIn(page);
|
||||
await openOnboarding(page);
|
||||
|
||||
// Step 1: name the company.
|
||||
await page.locator('input[placeholder="Acme Corp"]').fill(COMPANY_NAME);
|
||||
await page.getByRole("button", { name: "Next" }).click();
|
||||
|
||||
// Step 2: define the mission directly; confirming creates the company.
|
||||
await expect(
|
||||
page.locator("h3", { hasText: "Create your first agent" })
|
||||
).toBeVisible({ timeout: 10_000 });
|
||||
|
||||
await expect(page.locator('input[placeholder="CEO"]')).toHaveValue(AGENT_NAME);
|
||||
await page.getByRole("button", { name: "Next" }).click();
|
||||
|
||||
await expect(
|
||||
page.locator("h3", { hasText: "Give it something to do" })
|
||||
page.locator("h3", { hasText: "Define your mission" })
|
||||
).toBeVisible({ timeout: 10_000 });
|
||||
await page.getByRole("button", { name: "I know my mission" }).click();
|
||||
await page
|
||||
.locator('input[placeholder="e.g. Research competitor pricing"]')
|
||||
.fill(TASK_TITLE);
|
||||
.locator('textarea[placeholder="What is your team trying to achieve?"]')
|
||||
.fill(MISSION);
|
||||
await page.getByRole("button", { name: "Confirm mission" }).click();
|
||||
|
||||
// Step 3: name the team lead.
|
||||
const leadNameInput = page.locator('input[placeholder="Chief of staff"]');
|
||||
await expect(leadNameInput).toBeVisible({ timeout: 20_000 });
|
||||
await leadNameInput.fill(AGENT_NAME);
|
||||
await page.getByRole("button", { name: "Next" }).click();
|
||||
|
||||
await expect(
|
||||
page.locator("h3", { hasText: "Ready to launch" })
|
||||
).toBeVisible({ timeout: 10_000 });
|
||||
await expect(page.getByText(COMPANY_NAME)).toBeVisible();
|
||||
await expect(page.getByText(AGENT_NAME)).toBeVisible();
|
||||
await expect(page.getByText(TASK_TITLE)).toBeVisible();
|
||||
// Step 4: keep the default adapter and hire the lead. The adapter
|
||||
// environment test runs inside the smoke container, where no agent CLIs
|
||||
// are installed; an unhealthy report is expected and must not block the
|
||||
// hire. Allow generous time for the env probe + hire + auto-approval.
|
||||
const heartbeatButton = page.getByRole("button", {
|
||||
name: "Give it a heartbeat",
|
||||
});
|
||||
await expect(heartbeatButton).toBeVisible({ timeout: 10_000 });
|
||||
await expect(heartbeatButton).toBeEnabled({ timeout: 30_000 });
|
||||
await heartbeatButton.click();
|
||||
|
||||
await page.getByRole("button", { name: "Create & Open Task" }).click();
|
||||
await expect(page).toHaveURL(/\/issues\//, { timeout: 10_000 });
|
||||
// Step 5: review, then launch. "Get started" provisions the onboarding
|
||||
// goal/project and navigates to the dashboard only on success.
|
||||
const getStartedButton = page.getByRole("button", { name: "Get started" });
|
||||
await expect(getStartedButton).toBeVisible({ timeout: 60_000 });
|
||||
await expect(getStartedButton).toBeEnabled({ timeout: 10_000 });
|
||||
await getStartedButton.click();
|
||||
await expect(page).toHaveURL(/\/dashboard/, { timeout: 30_000 });
|
||||
|
||||
const baseUrl = new URL(page.url()).origin;
|
||||
|
||||
|
|
@ -95,6 +109,26 @@ test.describe("Docker authenticated onboarding smoke", () => {
|
|||
expect(ceoAgent!.role).toBe("ceo");
|
||||
expect(ceoAgent!.adapterType).not.toBe("process");
|
||||
|
||||
const goalsRes = await page.request.get(
|
||||
`${baseUrl}/api/companies/${company!.id}/goals`
|
||||
);
|
||||
expect(goalsRes.ok()).toBe(true);
|
||||
const goals = (await goalsRes.json()) as Array<{
|
||||
id: string;
|
||||
title: string;
|
||||
level: string;
|
||||
status: string;
|
||||
}>;
|
||||
expect(goals).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
title: MISSION,
|
||||
level: "company",
|
||||
status: "active",
|
||||
}),
|
||||
])
|
||||
);
|
||||
|
||||
const issuesRes = await page.request.get(
|
||||
`${baseUrl}/api/companies/${company!.id}/issues`
|
||||
);
|
||||
|
|
@ -104,9 +138,9 @@ test.describe("Docker authenticated onboarding smoke", () => {
|
|||
title: string;
|
||||
assigneeAgentId: string | null;
|
||||
}>;
|
||||
const issue = issues.find((entry) => entry.title === TASK_TITLE);
|
||||
expect(issue).toBeTruthy();
|
||||
expect(issue!.assigneeAgentId).toBe(ceoAgent!.id);
|
||||
const seededIssue = issues.find((entry) => entry.title === FIRST_TASK_TITLE);
|
||||
expect(seededIssue).toBeTruthy();
|
||||
expect(seededIssue!.assigneeAgentId).toBe(ceoAgent!.id);
|
||||
|
||||
await expect.poll(
|
||||
async () => {
|
||||
|
|
|
|||
Loading…
Reference in New Issue