RomanNum3ral
/
paperclip
mirror of https://github.com/paperclipai/paperclip.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
paperclip/doc/connections/SMOKE-LAB-TUTORIAL.md

16 KiB
Raw Permalink Blame History

Smoke Lab — hands-on tutorial

A guided, click-by-click walkthrough of the Smoke Lab for a person sitting at the board. You'll turn on the experimental flag, start the deterministic fixture services, drive every integration path (P1–P7) through its full governed lifecycle, and read the results in the matrix and the dashboard card. Nothing here touches a real vendor or a real credential — the OAuth provider and the MCP servers are local fakes.

Every "You should see" below was checked against the real screens; where a button or label is quoted, that's the exact text in the product.

Companion docs: the automated counterparts live in SMOKE-LAB-BROWSER-RUNNER.md (the agent-driven browser runner), tests/e2e/smoke-lab.shared.ts (the shared headless runner), and the tests/e2e/smoke-lab-p1-p4.spec.ts and tests/e2e/smoke-lab-p5-p7.spec.ts CI halves. The daily recurring routine that runs the browser smoke for you is described in §8.

0. Prerequisite: any private (non-public) instance

The Smoke Lab fail-closes on public deployments. It runs anywhere else — you do not need a special local_trusted box or any extra environment variables. Turning on the flag is all the setup there is.

Requirement Where
Smoke Lab experimental flag ON Instance settings → Experimental
deployment exposure not public (i.e. not internet-facing) how the instance was started

That's it. The everyday dev server works as-is: a local_trusted localhost box, an authenticated instance behind Tailscale + login (e.g. http://paperclip-dev:45439), and a pnpm dev server built with NODE_ENV=production are all fine — those are private, so the Smoke Lab is available. The auth mode and the Node build target no longer matter; only public exposure is disallowed (the fake OAuth provider and fixture sidecars must never be reachable from the open internet).

If the flag is off you'll see the tab say "Smoke Lab is turned off". If you're on a public instance, API calls return 403 "Smoke lab is only available on private (non-public) deployments" — move to a private instance.

Throughout this tutorial, {PREFIX} is your company's short issue prefix (shown in the URL bar, e.g. PAP). Replace it in the example paths.

1. Turn on the flag

  1. Open Instance settings → Experimental (/{PREFIX}/settings/experimental).
  2. Find the Smoke Lab card and toggle it on.
  3. You should see: the toggle stays on after a refresh.

2. Open the Smoke Lab and start the services

  1. In the left sidebar open Apps, then under the Developer section ("Advanced setup for developers.") click Smoke Lab (/{PREFIX}/apps/advanced/smoke-lab). The breadcrumb reads Apps → Advanced setup → Smoke Lab.
  2. You should see: a Developer tools page header, then the Smoke Lab section with an Experimental badge and a Hands-on tutorial link, a Fixture services row with four buttons — Start services, Stop, Install fixture apps, Reset — an Integration matrix (all cells "not run" at first), and a Runs panel ("no runs yet"). A card shows the Fake OAuth demo credentials:
    • email: smoke@paperclip.test
    • password: smoke-password
  3. Click Start services. You should see: two service cards flip to a green running dot:
    • Fake OAuth 2.0 provider — its URL is on the instance's own host (…/api/companies/{companyId}/smoke-lab/oauth/authorize); the provider runs in-process, so there is no separate port.
    • HTTP MCP fixture — a loopback sidecar with a http://127.0.0.1:<port>/mcp URL.
  4. Click Install fixture apps. You should see: a toast — "Fixture apps installed" the first time, "Fixture apps already present" on a re-run (installing again is safe; it's idempotent). Two connections now exist under Apps → Connections:
    • Smoke Lab HTTP MCP fixture — remote HTTP transport, used by P1, P2, P5, P6, P7. This is the one with the OAuth walkthrough.
    • Smoke Lab stdio MCP fixture — local stdio transport, used by P3, P4. No OAuth here — stdio servers are spawned locally and don't sign in to anything.

Which fixture am I in? The Connections list shows both, and the stdio one may be listed first. Open Permissions and check the action names: the HTTP fixture includes List synthetic todos, while the stdio fixture includes Deterministic time. Use Smoke Lab HTTP MCP fixture for the OAuth steps.

If Start services errors with a 403, re-check §0 — you're on a public (internet-facing) instance. Any private instance works, including the everyday authenticated dev server.

3. The lifecycle you'll exercise on every path

Each path P1–P7 walks the same governed lifecycle. You drive it from a fixture connection's pages — a small left-hand menu inside the app with Permissions and Review (plus Services for broker connections) (/{PREFIX}/apps/{connectionId}/{tab}).

Two things to know before you start:

  • Actions are listed by their display title, with the raw tool name behind them — e.g. todo.list renders as List synthetic todos. The table below gives both.
  • "Policies" are the three-way per-action toggles on the Permissions tab. Each action is Off, Ask first, or Allowed. When a step below says "with a require-approval policy in force", set that action to Ask first. "Block policy" means set it to Off. New connections start Allowed; narrow an action before testing when the scenario requires another decision.
Step What you do What you should see
connect Open the fixture connection (for P1, complete the fake OAuth consent). Connection shows Connected, with the action count.
discover-catalog Open Permissions. The action list includes the path's tools (e.g. List synthetic todos).
allowed-read Set the read action to Allowed, then use its Test button on Permissions. Decision badge Allowed; the call returns without error.
ask-first-write Set the write action to Ask first, then use its Test button. Decision Ask first; a pending request appears in Review.
approve Review tab → approve the pending write. The request clears; the call completes.
denied-call Set the blocked action to Off, then use its Test button on Permissions. Decision Off; the call is refused with a reason.
schema-change / quarantine Trigger the fixture schema flip (HTTP paths), then Refresh actions on Permissions. A quarantine pill with the changed entries held back.
revoke From Connectors, choose Remove connection from the connection's management menu. In the classic table, use the trash button labeled Delete app connection. (For P6, revoke the gateway session instead.) Agent access is removed immediately; a revoked token is cut off (401).
audit-evidence Open company Audit and choose Apps & tools in the Action filter. Audit rows for the allowed, approved, denied, quarantine, and revoke decisions.

(The results matrix in §6 folds approve into its Ask-first write column, so the matrix shows 8 columns for these 9 steps.)

The per-path tools are:

read (allowed) write (ask-first) denied schema-flip (quarantine)
HTTP (P1, P2, P5, P6, P7) todo.list — List synthetic todos todo.add — Add synthetic todo email.send — Send outbox email fixture.schemaFlip — Fixture schema mutation
stdio (P3, P4) time.now — Deterministic time slow.ping — Slow stdio fixture crash.now — Crashing stdio fixture malicious.metadata — Malicious metadata fixture

4. Path P1 — Remote HTTP MCP, OAuth (the worked example)

This is the richest path — do it by hand once and the rest are variations.

  1. Connect via the fake OAuth provider.
    • From Apps → Connections (/{PREFIX}/apps), open Smoke Lab HTTP MCP fixture (not the stdio one — see the callout in §2). If its header says Needs attention, use the Reconnect action directly below the header.
    • You should see: the reconnect card explains that the saved connection needs authorization and offers Connect with Smoke OAuth. If the fixture is already healthy, no reconnect card is shown.
    • Click it. The fake provider's real consent page opens: a brown banner "SMOKE TEST - not a real provider", headed "Paperclip Smoke OAuth login + consent".
    • The email is pre-filled (smoke@paperclip.test). Type the password smoke-password and click Authorize smoke test app.
    • You should see: the provider accepts the credentials and returns you to this connection's Permissions page with a Connected status. Wrong credentials are rejected with a 403.
  2. Discover the catalog. Open Permissions and confirm List synthetic todos (todo.list) and Add synthetic todo (todo.add) appear under Actions.
  3. Allowed read. Make sure List synthetic todos is set to Allowed in Permissions. Click its Test button, pick an agent in the Act as picker, and run it. You should see: an Allowed badge and a result with no error.
  4. Ask-first write → approve. In Permissions, set Add synthetic todo to Ask first. Click its Test button and run it. You should see: an Ask first badge and a pending request. Switch to the Review tab (its idle state says "Nothing is waiting for your OK right now") and approve it. You should see: the request clears and the write completes.
  5. Denied call. In Permissions, set Send outbox email (email.send) to Off, then click its Test button. You should see: an Off badge and a refusal carrying a reason code.
  6. Schema change → quarantine. Run Fixture schema mutation (fixture.schemaFlip) — it changes a tool's schema — then click Refresh actions on the Permissions tab. You should see: a quarantine pill (on Review and Permissions) — the changed entries are held back until you explicitly turn them on.
  7. Revoke. Return to Apps → Connections and choose Remove connection from the connection's management menu. In the classic table, use the trash button labeled Delete app connection. You should see: a confirmation explaining that saved credentials are deleted and agent access ends immediately. Reinstall the fixture apps before continuing with another path that uses this connection.
  8. Audit evidence. Open company Audit and choose Apps & tools in the Action filter. You should see: rows for each decision above (allowed, approved, denied, quarantine, revoke).

Prefer not to click all seven by hand? Use the automated browser smoke — §7 — which performs exactly these steps and leaves you screenshots to read, including a shot of the filled OAuth consent page.

5. Paths P2–P7 — what's different

Each path reuses the §3 lifecycle. Only the connect/transport and a couple of tools change.

  • P2 — Remote HTTP MCP, API key. Same HTTP fixture and tools as P1, but the connection is authenticated with a static fixture credential instead of OAuth. You should see: audit rows preserve the decisions without ever exposing the credential value.
  • P3 — Local stdio MCP template. Uses the Smoke Lab stdio MCP fixture connection and its tools (see the stdio row in §3's table). The read is Deterministic time (time.now); the "denied" tool Crashing stdio fixture (crash.now) is blocked by policy. It does not require OAuth. Quarantine evidence is recorded via fixture metadata rather than an HTTP schema flip.
  • P4 — Plugin-provided integration. Exercises the catalog-backed app install path a plugin would use, over the stdio fixture. Same stdio tools as P3. You should see: Audit rows record the install + lifecycle decisions.
  • P5 — Paste-a-config / run-your-own import. Entry via the Developer section of Apps; import the HTTP fixture through the advanced configuration surface, then run the same HTTP lifecycle. You should see: advanced Audit rows show the import and the governed calls.
  • P6 — Token broker / gateway session. Create a run-scoped gateway session for the smoke agent, list tools through the session token, then revoke the session. You should see: the token lists tools before revoke and is cut off (401) after. Entry/evidence via Audit.
  • P7 — Governance surfaces. Entry via Review. This path is about the governance surfaces themselves — profiles, ask-first policies, block policies, and quarantine. You should see: Review and Audit expose the ask-first, block, quarantine, and revoke evidence together.

6. Read the results matrix

  1. Back on Apps → Developer → Smoke Lab, look at the Integration matrix.
  2. You should see: a row per path (P1 Remote HTTP · OAuth … P7 Governance surfaces) and a column per lifecycle stage — Connect, Discover catalog, Allowed read, Ask-first write, Denied call, Schema-change quarantine, Revoke, Audit evidence — with a glyph per cell: ✓ pass (green), ✗ fail (red), – skipped (amber), and a dot for not run. A health dot (green/amber/red) summarizes the selected run, and any failing paths are listed next to it.
  3. Click a run in the Runs list to drill into its steps. Each recorded step shows its status, a one-line detail, its duration, and — when present — a View screenshot link (for P1 this includes the typed OAuth consent page).

7. Run the automated browser smoke (optional but recommended)

Rather than click all seven paths by hand, let the agent-driven runner do it and read the evidence:

  1. In the Runs panel of the Smoke Lab tab, click Run browser smoke now to open a run, or run the reference driver from a shell (it types the demo credentials into the real consent page for you): 
    SMOKE_BASE=http://127.0.0.1:3251 \
  node --experimental-strip-types tests/e2e/smoke-lab-browser-runner.mts
# SMOKE_ONLY=P1,P3 restricts to a subset; omit for the full P1–P7 sweep.
  2. You should see: the matrix fills in green as each step is recorded, every step carrying a viewable screenshot, and a new entry in the Runs list.

8. The daily routine (hands-off)

A recurring Paperclip routine — "Daily Smoke Lab integration smoke (P1-P7)" — runs the browser smoke for you every day and:

  • records each run to the results API (matrix + dashboard);
  • on a real failure, files a high-priority issue with the failing step and a screenshot, assigned to the owning coder, and links it back to the run;
  • when the flag is off or the instance is unreachable, records an amber/skipped run instead of failing silently.

It's driven by tests/e2e/smoke-lab-routine.mts. See that file's header and the routine's own description for the runbook.

9. Read the dashboard card

  1. Open the Dashboard (/{PREFIX}/dashboard).
  2. You should see: an Integration smoke card summarizing the latest run — "All paths passing" when green, the failing paths when not, or "No runs yet — Run one from the Smoke Lab tab" before the first run. It's the at-a-glance health signal; the Smoke Lab tab is the drill-down. Clicking the card takes you to the Smoke Lab.

10. Clean up

  • Click Reset on the Smoke Lab tab to clear runs and fixture state.
  • Click Stop to stop the fixture services.
  • If you booted a throwaway instance for §0, stop it (Ctrl-C) — its embedded database is disposable.

That's the whole loop: flag on → services up → fixtures installed → drive/observe the P1–P7 lifecycle → read the matrix and the dashboard card.