diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml index 3bb61f84b..fa911176d 100644 --- a/.github/workflows/evals.yml +++ b/.github/workflows/evals.yml @@ -183,6 +183,14 @@ jobs: # bin/ + sections/ are committed). $HOME is /github/home here; the spawned # claude inherits it (this runner adds no HOME/CLAUDE_CONFIG_DIR override, # no hermetic mode) and the Seed step already proved claude reads $HOME. + # + # KEEP THIS STEP even though seedSkills/hermeticSkillsConfigDir() now + # registers skills for hermetic PTY children: that registry is SYMLINKS + # into the repo checkout, and this container's cross-mount symlinks + # defeat the TUI skill scanner (see the note inside the step below) — + # the real-file copies here are what the TUI actually reads. HOME is + # also not hermeticized, so the absolute ~/.claude/skills/gstack/... + # preamble paths resolve through the gstack root symlink this step makes. - name: Register gstack skills for PTY smoke if: matrix.suite.name == 'e2e-pty-plan-smoke' run: | diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3dba8f3ba..f6d584c8c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -91,14 +91,15 @@ When a user runs `pair-agent --client`, the daemon starts an ngrok tunnel so a r The fix is **two HTTP listeners**, not one: -- **Local listener** (`127.0.0.1:LOCAL_PORT`) — always bound. Serves bootstrap (`/health` with token delivery), `/cookie-picker`, `/inspector/*`, `/welcome`, `/refs`, the sidebar-agent API, and the full command surface. Never forwarded. +- **Local listener** (`127.0.0.1:LOCAL_PORT`) — always bound. Serves token bootstrap (`POST /extension-token`, released only to the pinned extension identity), `/health` (liveness/status only — never a token), `/cookie-picker`, `/inspector/*`, `/welcome`, `/refs`, the sidebar-agent API, and the full command surface. Never forwarded. - **Tunnel listener** (`127.0.0.1:TUNNEL_PORT`) — bound lazily on `/tunnel/start`, torn down on `/tunnel/stop`. Serves a locked allowlist: `/connect` (pairing ceremony, unauth + rate-limited), `/command` (scoped tokens only, further restricted to a browser-driving command allowlist), and `/sidebar-chat`. Everything else 404s. ngrok forwards only the tunnel port. The security property comes from **physical port separation**: a tunnel caller cannot reach `/health` or `/cookie-picker` because those paths don't exist on that TCP socket. Header inference (check `x-forwarded-for`, check origin) is unreliable (ngrok header behavior changes; local proxies can add these headers); socket separation isn't. | Endpoint | Local listener | Tunnel listener | Notes | |---|---|---|---| -| `GET /health` | public (no token unless headed/extension) | 404 | Token bootstrap for extension happens locally only | +| `GET /health` | public (liveness/status only — never a token) | 404 | Token bootstrap moved to `POST /extension-token` (v1.63) | +| `POST /extension-token` | pinned Origin (`chrome-extension://`) + loopback Host | 404 | The only endpoint that hands out the root token | | `GET /connect` | public (`{alive:true}`) | public (`{alive:true}`) | Probe path for tunnel liveness | | `POST /connect` | public (rate-limited 300/min) | public (rate-limited) | Setup-key exchange for pair-agent | | `POST /command` | auth (Bearer root OR scoped) | auth (scoped only, allowlisted commands) | Root token on tunnel = 403 | @@ -114,6 +115,8 @@ ngrok forwards only the tunnel port. The security property comes from **physical | `GET /inspector/events` | Bearer OR HttpOnly `gstack_sse` cookie | 404 | SSE. Same cookie as /activity/stream | | `POST /sse-session` | auth (Bearer) | 404 | Mints the view-only 30-min SSE session cookie | +**Extension token bootstrap (v1.63.0.0).** `GET /health` never carries a token in any mode — it is liveness/status only. The sidebar extension obtains the root token via `POST /extension-token`, which releases it only when the caller's Origin is exactly `chrome-extension://` (pinned by the `key` field in `extension/manifest.json`; reproduce the derivation with `bun browse/scripts/extension-id.ts`) and the Host header parses to a loopback hostname — parsed with `new URL()`, never compared raw, because Host carries the port. Web pages cannot forge a `chrome-extension://` Origin, and the endpoint is never added to the tunnel allowlist, so the tunnel surface 404s it by default-deny. + **Tunnel surface denial logs.** Every rejection on the tunnel listener (`path_not_on_tunnel`, `root_token_on_tunnel`, `missing_scoped_token`, `disallowed_command:*`) is recorded asynchronously to `~/.gstack/security/attempts.jsonl` with timestamp, source IP (from `x-forwarded-for`), path, and method. Rate-capped at 60 writes/min globally to prevent log-flood DoS. Shares the attempt log with the prompt-injection scanner. **SSE session cookies.** EventSource can't send Authorization headers, so the extension POSTs `/sse-session` once at bootstrap with the root Bearer and receives a 30-minute view-only cookie (`gstack_sse`, HttpOnly, SameSite=Strict). The cookie is valid ONLY for `/activity/stream` and `/inspector/events` — it is NOT a scoped token and cannot be used on `/command`. Scope isolation is enforced by the module boundary: `sse-session-cookie.ts` has no imports from `token-registry.ts`. @@ -144,6 +147,14 @@ Cookies are the most sensitive data gstack handles. The design: The browser registry (Comet, Chrome, Arc, Brave, Edge) is hardcoded. Database paths are constructed from known constants, never from user input. Keychain access uses `Bun.spawn()` with explicit argument arrays, not shell string interpolation. +### Egress receipt ledger (v1.63.0.0) + +Every enumerated gstack-initiated off-machine sink writes a hash-chained, tamper-evident receipt to `~/.gstack/security/egress.jsonl` BEFORE the send — `writeReceipt` in `lib/egress-receipt.ts` for TypeScript callers, `_receipted_curl` / `_receipted_git` from `bin/gstack-egress-lib.sh` for shell scripts. Receipts record a sha256 of the exact bytes sent when the caller owns them (subprocess-owned sends like git pushes record `sha256: null`); they never store the body. + +Failure polarity is per-class and pinned by tests. Sensitive sinks are fail-closed: brain-sync pushes, memory-ingest, gbrain-sync, telemetry, ngrok tunnel starts, mcp-verify, and supabase-provision refuse to send if the receipt can't be written (each refusal prints problem + cause + fix). User-facing sinks fail open with a stderr warning — the design binary's OpenAI calls, update-check, the read-only dashboards, and git-class receipts proceed even when the receipt write failed, so a fail-open send can go unrecorded (warned, by design). The new-sink scanner in `test/egress-receipt-wiring.test.ts` fails CI when an off-machine sink ships unwired; its only exemptions are enumerated with reasons (user-directed page fetches, reachability probes, install-doc strings, skill prose). + +Inspect the ledger with `bin/gstack-egress`: `list` (what gstack attempted to send), `verify` (recompute the chain, exit 3 on tamper), `grants` (the standing consent settings and how to revoke each). Threat model: the ledger is forensic observability of ATTEMPTED egress — it records what gstack tried to send so accidents are auditable; it is not an exfiltration control. + ### Unicode sanitization at server egress (v1.38.0.0) Page content harvested by CDP can contain lone UTF-16 surrogate halves (orphaned high or low surrogates from broken JavaScript string handling on the page). When those reach `JSON.stringify`, Bun emits them as `\uD800`-style escape sequences that the downstream consumer's `JSON.parse` accepts, but the Anthropic API rejects with a 400 — turning a single weird page into a session-killing error. Defense is single-point, applied at every server egress that ships page-derived strings. @@ -414,7 +425,7 @@ The `EvalCollector` accumulates test results and writes them in two ways: 1. **Incremental:** `savePartial()` writes `_partial-e2e.json` after each test (atomic: write `.tmp`, `fs.renameSync`). Survives kills. 2. **Final:** `finalize()` writes a timestamped eval file (e.g. `e2e-20260314-143022.json`). The partial file is never cleaned up — it persists alongside the final file for observability. -`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack-dev/evals/`. +`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack-dev/evals/`. Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. ### Test tiers diff --git a/BROWSER.md b/BROWSER.md index affa0447d..fde16b45e 100644 --- a/BROWSER.md +++ b/BROWSER.md @@ -705,6 +705,10 @@ Or do it manually: `chrome://extensions` → toggle Developer mode → Load unpacked → navigate to `~/.claude/skills/gstack/extension` → pin the extension → enter the port from `$B status`. +v1.63 pinned the extension identity via the manifest `key` field, so existing +unpacked installs get a new extension ID and panel-local state (saved port) +resets once — a one-time in-product notice explains this. + --- ## Pair-agent @@ -758,6 +762,15 @@ remote agent that tries them gets a 403 plus a fresh entry in the denial log. + domain only (no raw IP, no full request body), rotates at 10MB with 5 generations. Per-device salt at `~/.gstack/security/device-salt` (mode 0600). +### Tunnel egress receipts (v1.63+) + +Every tunnel session open writes a hash-chained egress receipt (sink +`browse-tunnel`) to `~/.gstack/security/egress.jsonl` BEFORE ngrok forwards +anything. Fail-closed: if the receipt can't be written, the tunnel listener +is torn down and the start is refused. Inspect the ledger with +`bin/gstack-egress list` and verify chain integrity with +`bin/gstack-egress verify` (exit 3 on tamper). + See [`docs/REMOTE_BROWSER_ACCESS.md`](docs/REMOTE_BROWSER_ACCESS.md) for the full operator guide. @@ -800,6 +813,19 @@ The Terminal pane uses a separate session cookie, `gstack_pty`, minted via PTY, can't dispatch arbitrary `/command` calls. `/health` endpoint MUST NOT surface this token. +### Extension token bootstrap (v1.63+) + +`GET /health` is liveness/status only — it never carries a token, in any +mode. The Side Panel extension bootstraps the root token via +`POST /extension-token` on the local listener. The server releases the +token only when the caller's Origin is exactly +`chrome-extension://` — the `key` field in +`extension/manifest.json` pins the extension ID (`GSTACK_EXTENSION_ID` in +`browse/src/server.ts`; derivation reproducible via +`bun browse/scripts/extension-id.ts`) — AND the parsed Host hostname is +loopback. Anything else gets a detail-free 403. The endpoint is never +added to `TUNNEL_PATHS`, so the tunnel surface 404s it by default-deny. + ### Token registry `browse/src/token-registry.ts` handles mint/validate/revoke for all three diff --git a/CHANGELOG.md b/CHANGELOG.md index b817fc933..9f3efb048 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,113 @@ # Changelog +## [1.63.0.0] - 2026-08-13 + +**Everything gstack sends off your machine now leaves a receipt you can read.** +**And the eval harness stopped grading itself a passing grade.** + +This release ports the parts of the GStack 2 fork that earned their way back into +main. The headline is a hash-chained egress ledger: every place gstack itself +sends data off your machine now writes a local, tamper-evident receipt first, and +`gstack-egress list` / `verify` show you exactly what left and prove the chain is +intact. Two new command-line tools ship with it: `gstack-egress` (the auditor's +view) and `gstack-context-bill` (a token bill-of-materials for any skills tree, so +you can see what a gstack install costs your context window before you invoke +anything). The test harness got three real fixes, one of them a bug that had been +quietly lying to every contributor for months. + +### The numbers that matter + +Source: the assembled branch (`git log 1.62.0.0..HEAD`), the free suite +(`bun test`), and the discovery-surface gate (`test/catalog-budget.test.ts`). + +| Metric | Before | After | Δ | +|---|---|---|---| +| gstack-owned off-machine sinks with a receipt | 0 | every enumerated sink | tripwire-enforced, zero exceptions | +| Eval "no regressions" lines that were self-comparisons | every one | 0 | the harness compared runs against their own in-progress accumulator | +| Paid gate runner isolation | one process, one hung file kills the tier | one process per file, group-SIGKILL on stall | + never-started accounting | +| Discovery catalog budget | unenforced | 1,105 token-equivalents measured, 1,150 ceiling | ratchet-protocol on every skill add | +| Browser `/health` endpoint | served the root auth token to any localhost caller in headed mode | serves no token in any mode | token bootstrap moved to a pinned-origin POST | + +The eval-store line is the one that matters most for anyone hacking on gstack: +`findPreviousRun` picked the newest same-tier file as the baseline, and the +in-progress `_partial` accumulator always won that sort, so the auto-comparison +compared a run against itself and printed "no regressions" no matter what. That is +fixed, with regression tests, and the fix was confirmed against the bug on the +prior release before landing. + +### What this means for you + +If you care what gstack does with your data, you can now audit it: run +`gstack-egress list` after any session and see every off-machine send, or +`gstack-egress verify` to confirm nothing was rewritten. If you contribute to +gstack, your eval comparisons mean something again, the paid gate can't be taken +down by one wedged test, and `gstack-context-bill` tells you what your skill +changes cost before you ship them. Nothing new phones home; the ledger is local +and the receipts record what gstack *attempts* to send, so accidents are auditable. + +Ported from the GStack 2 fork by Sina Matian (time-attack/gstack); the eval-store +bug fix and the port shortlist were selected and hardened for upstream. + +### Itemized changes + +#### Added +- `gstack-egress` — read the hash-chained egress receipt ledger: `list` (what + gstack attempted to send off-machine), `verify` (recompute the chain, exit 3 on + tamper), `grants` (the standing consent settings and how to revoke each). +- `gstack-context-bill` — offline token bill-of-materials for a skills tree: + always-on discovery cost vs per-invocation cost, `--diff` between two trees, + `--budget`, and `--exact` (opt-in, measures against the real tokenizer). +- Hash-chained egress receipts (`lib/egress-receipt.ts`): fail-closed + receipt-before-send for sensitive sinks (brain-sync, memory-ingest, gbrain-sync, + telemetry, tunnels), fail-open with a warning for user-facing sinks (the design + binary's model calls, update-check, dashboards). A tripwire test enforces that + every off-machine sink in the tree is wired, with zero silent exceptions. +- Sharded paid-gate runner (`test:gate:sharded` / `test:periodic:sharded`): one + process per test file, an external wall-clock timeout that group-SIGKILLs a + wedged file's whole process tree, and four-way per-shard status so a crash can't + masquerade as a pass. +- `gstack-context-bill` and the egress tools install through the standard `./setup` + path like every other gstack binary. + +#### Changed +- The browser `/health` endpoint no longer carries the root auth token in any + mode. The sidebar extension bootstraps its token through a new + `POST /extension-token` that requires the pinned extension origin and a loopback + Host; the tunnel listener never exposes it. Upgrading resets the sidebar's + panel-local state once, explained in-product. +- Hermetic PTY test children can register the repo's shipped skills, so + slash-command gate tests actually exercise the skill under test instead of + silently measuring nothing. +- Discovery-surface cost is now gated: `test/catalog-budget.test.ts` pins the + aggregate skill name+description budget with a self-service ratchet protocol. + +#### Fixed +- The eval harness auto-comparison compared every run against its own in-progress + accumulator and reported "no regressions" unconditionally. Fixed with regression + tests; comparisons now find the latest *completed* same-tier run. +- The browser `/health` token leak (a headed-mode carve-out that handed the root + token to any localhost caller). + +#### For contributors +- Shared modules replace duplicated logic: one paid-test-set definition consumed by + both the free-suite filter and the paid runner, one skill-census helper with three + explicit counts (physical files, authored skills, registry entries) consumed by + the seeder, context-bill, and the catalog gate. +- `CLAUDE.md`'s compiled-binaries note corrected: the `browse/dist` binaries have + been untracked since v0.11.16.0, so they no longer appear in `git status`. +- External-service E2E tests (Codex, Gemini, benchmark providers) are declared + periodic-tier with the canonical whole-file guard, so the merge-blocking gate + never waits on a third-party CLI. The Codex runner passes + `--skip-git-repo-check` (now required in non-git working dirs) and the Gemini + runner classifies an unusable CLI (removed flags, retired auth paths) as a + skip instead of a false failure. +- The PTY test runner parses AskUserQuestion prompts that reflow onto a single + logical line and strips DEC cursor-visibility residue, pinned by + `test/pty-askuserquestion-single-line.test.ts` — the failure class that + previously ate a gate test's whole time budget. +- New follow-ups filed in `TODOS.md`: egress ledger rotation (chain-genesis + records), a launch-nonce token bootstrap, and eval-watch shard-awareness. + ## [1.62.0.0] - 2026-08-12 ## **Plan reviews stop asking what to review when you're in plan mode.** diff --git a/CLAUDE.md b/CLAUDE.md index 49515f480..461914a1d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,6 +9,8 @@ bun run test:evals # run paid evals: LLM judge + E2E (diff-based, ~$4/run max) bun run test:evals:all # run ALL paid evals regardless of diff bun run test:gate # run gate-tier tests only (CI default, blocks merge) bun run test:periodic # run periodic-tier tests only (weekly cron / manual) +bun run test:gate:sharded # gate tier via the sharded paid runner (one Bun process per test file) +bun run test:periodic:sharded # periodic tier via the sharded paid runner (implies EVALS_ALL=1) bun run test:e2e # run E2E tests only (diff-based, ~$3.85/run max) bun run test:e2e:all # run ALL E2E tests regardless of diff bun run eval:select # show which tests would run based on current diff @@ -48,13 +50,21 @@ MCP servers / skills), a temp `GSTACK_HOME`, and `--strict-mcp-config`. Local eval signal matches CI. Debug against real operator state with `EVALS_HERMETIC=0` (restores the legacy env AND drops the strict-MCP flag). Per-test `env:` overrides merge last, so deliberate contamination -(`CONDUCTOR_WORKSPACE_PATH`, per-test `GSTACK_HOME`) keeps working. Wiring -is pinned by `test/hermetic-wiring.test.ts` (static tripwire) and two -gate-tier canaries in `test/skill-e2e-hermetic-canary.test.ts`. +(`CONDUCTOR_WORKSPACE_PATH`, per-test `GSTACK_HOME`) keeps working. The +hermetic config dir seeds NO skills by default; a PTY test that types a +`/skill` slash command must pass `seedSkills: true` to the PTY runner, which +points the child's `CLAUDE_CONFIG_DIR` at `hermeticSkillsConfigDir()` — a +seeded registry that symlinks the LIVE working tree's SKILL.md files (by +design: the skills ARE the subject under test; a snapshot would measure stale +copies). Wiring is pinned by `test/hermetic-wiring.test.ts` (static tripwire), +two gate-tier canaries in `test/skill-e2e-hermetic-canary.test.ts`, and the +seeding tripwires in `test/hermetic-skills-seeding.test.ts` / +`test/pty-skill-seeding-wiring.test.ts`. E2E tests stream progress in real-time (tool-by-tool via `--output-format stream-json --verbose`). Results are persisted to `~/.gstack-dev/evals/` with auto-comparison -against the previous run. +against the previous finalized run (in-flight `_partial` files are never used as +a baseline, so a run can't compare against itself). **Diff-based test selection:** `test:evals` and `test:e2e` auto-select tests based on `git diff` against the base branch. Each test declares its file dependencies in @@ -110,6 +120,7 @@ gstack/ │ ├── host-adapters/ # Host-specific adapters (OpenClaw tool mapping) │ ├── resolvers/ # Template resolver modules (preamble, design, review, gbrain, etc.) │ ├── skill-check.ts # Health dashboard +│ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard) │ └── dev-skill.ts # Watch mode ├── test/ # Skill validation + eval tests │ ├── helpers/ # skill-parser.ts, session-runner.ts, llm-judge.ts, eval-store.ts @@ -147,7 +158,7 @@ gstack/ │ ├── test/ # Integration tests │ └── dist/ # Compiled binary ├── extension/ # Chrome extension (side panel + activity feed + CSS inspector) -├── lib/ # Shared libraries (worktree.ts) +├── lib/ # Shared libraries (worktree.ts, egress-receipt.ts, context-bill.ts, redact-engine.ts) ├── docs/designs/ # Design documents ├── setup-deploy/ # /setup-deploy skill (one-time deploy config) ├── .github/ # CI workflows + Docker image @@ -184,6 +195,16 @@ behavior). If you blow past 40K, the right fix is usually: (1) look at WHAT grew or as a reference doc, (3) only compress carefully-tuned prose as a last resort — cuts to the coverage audit, review army, or voice directive have real quality cost. +A second, harder ceiling guards the DISCOVERY surface: `test/catalog-budget.test.ts` +caps the aggregate frontmatter `name` + `description` across all skills at 1,150 +token-equivalents (260-byte per-skill sub-cap), counted through the shared census +in `test/helpers/skill-census.ts`. This one is enforced, not a warning — every +host loads the full catalog every session, so growth here taxes every +conversation. The failure message carries the re-measure + ratchet protocol. +`bin/gstack-context-bill` shows the full token bill-of-materials for a skills +tree (always-on vs per-invocation, `--diff`, `--budget`; `--exact` opts into the +real tokenizer and POSTs file text to api.anthropic.com with an egress receipt). + **Merge conflicts on SKILL.md files:** NEVER resolve conflicts on generated SKILL.md files by accepting either side. Instead: (1) resolve conflicts on the `.tmpl` templates and `scripts/gen-skill-docs.ts` (the sources of truth), (2) run `bun run gen:skill-docs` @@ -282,10 +303,14 @@ PTY via `window.gstackInjectToTerminal(text)`, exposed by `sidepanel-terminal.js`. No `/sidebar-command` POST — the live REPL is the only execution surface in the sidebar now. -**`/health` MUST NOT surface any shell-grant token.** It already leaks -`AUTH_TOKEN` to localhost callers in headed mode (a v1.1+ TODO). Don't -make that worse by adding the PTY session token there. PTY auth flows -through `POST /pty-session` only. +**`/health` MUST NOT surface any token — and it no longer does** (v1.63+). +The historical headed-mode leak of `AUTH_TOKEN` is fixed: `GET /health` is +liveness/status only in every mode. Token bootstrap is `POST /extension-token`, +which validates the caller's Origin against the pinned extension identity +(the `key` field in `extension/manifest.json` pins the extension ID — +`GSTACK_EXTENSION_ID` in `browse/src/server.ts`, derivation reproducible via +`bun browse/scripts/extension-id.ts`) plus a loopback Host. PTY auth still +flows through `POST /pty-session` only. Don't add any token to `/health`. **Transport-layer security** (v1.6.0.0+). When `pair-agent` starts an ngrok tunnel, the daemon binds two HTTP listeners: a local listener (127.0.0.1, full command @@ -315,6 +340,23 @@ response in `server.ts`, read `browse/test/server-sanitize-surrogates.test.ts` pins the wiring with invariant tests, so bypasses fail CI. +**Egress receipts at every off-machine sink** (v1.63.0.0+). Every gstack-initiated +send off the machine MUST write a hash-chained receipt to +`~/.gstack/security/egress.jsonl` BEFORE the send: TypeScript callers use +`writeReceipt` from `lib/egress-receipt.ts`; shell scripts source +`bin/gstack-egress-lib.sh` and use `_receipted_curl` / `_receipted_git`. Failure +polarity is per-class: fail-closed for sensitive sinks (brain-sync, memory-ingest, +gbrain-sync, telemetry, ngrok tunnels, mcp-verify, supabase-provision), fail-open ++ stderr warning for user-facing ones (design OpenAI calls, update-check, +dashboards, git-class ops). The new-sink scanner in +`test/egress-receipt-wiring.test.ts` fails CI on an unreceipted `curl` / +`git push` / `fetch` to a non-loopback host unless the file carries a reasoned +entry in its `SCANNER_EXEMPT` list (user-directed page fetches, reachability +probes, instruction strings, skill prose) — if you add a new off-machine sink, +wire it through the helpers and add it to the enumerated sink list. Inspect with +`bin/gstack-egress` (`list` | `verify`, exit 3 on tamper | `grants`). Threat +model: forensic observability of ATTEMPTED egress, not an exfiltration control. + **SSE endpoint helper** (v1.51.0.0+). New SSE endpoints in `server.ts` MUST route through `createSseEndpoint(req, config)` from `browse/src/sse-helpers.ts`. The helper owns the cleanup contract (abort + enqueue-throw + heartbeat-throw, all @@ -425,19 +467,21 @@ migration script to `gstack-upgrade/migrations/`. Read CONTRIBUTING.md's "Upgrad migrations" section for the format and testing requirements. The upgrade skill runs these automatically after `./setup` during `/gstack-upgrade`. -## Compiled binaries — NEVER commit browse/dist/ or design/dist/ +## Compiled binaries — never commit browse/dist/, design/dist/, or make-pdf/dist/ -The `browse/dist/` and `design/dist/` directories contain compiled Bun binaries -(`browse`, `find-browse`, `design`, ~58MB each). These are Mach-O arm64 only — they -do NOT work on Linux, Windows, or Intel Macs. The `./setup` script already builds -from source for every platform, so the checked-in binaries are redundant. They are -tracked by git due to a historical mistake and should eventually be removed with -`git rm --cached`. +The `browse/dist/`, `design/dist/`, and `make-pdf/dist/` directories contain +compiled Bun binaries (`browse`, `find-browse`, `design`, ~62MB each). These are +Mach-O arm64 only — they do NOT work on Linux, Windows, or Intel Macs. The +`./setup` script builds from source for every platform. -**NEVER stage or commit these files.** They show up as modified in `git status` -because they're tracked despite `.gitignore` — ignore them. When staging files, -always use specific filenames (`git add file1 file2`) — never `git add .` or -`git add -A`, which will accidentally include the binaries. +These directories are **untracked and gitignored** (`.gitignore:3-6`; the +`browse/dist/` binaries were untracked in `64d5a3e4`, v0.11.16.0; the others were +never tracked). They will NOT appear in `git status`. If a dist binary ever does +show up in `git status`, something force-added it (`git add -f`) — do not commit +it; unstage it and find out how it got there. + +When staging files, always use specific filenames (`git add file1 file2`) — never +`git add .` or `git add -A`, which can sweep in build outputs and junk. ## Redaction guard (PII / secrets / legal content) @@ -864,7 +908,17 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session machine-wide `gstack-evals` lock (concurrent worktrees serialize instead of saturating the shared model API), a per-tier watchdog, and a **run-scoped** log under `~/.gstack-dev/eval-runs/` (no shared-`/tmp` collision). Each prints its - log path. Or call `gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- + log path. `eval:bg:gate` / `eval:bg:periodic` run their tier through the + sharded paid runner (`scripts/test-paid-shards.ts`, also exposed as + `test:gate:sharded` / `test:periodic:sharded`): one Bun process per test + file, an external wall-clock timeout that kills the shard's process GROUP + (stray `claude`/`codex` grandchildren included), a per-shard + `GSTACK_EVAL_DIR=/shards//` honored by the `EvalCollector` + constructor, and an aggregate that separates failed vs timed-out vs + never-started shards — the detach timeouts (25200s gate / 28800s periodic) + are sized against worst-case shard wall clock. `eval:list` / `eval:compare` / + `eval:summary` read the shard dirs too. Or call + `gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- ` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never pass keys in argv). - Then **poll the printed logfile** with a death-aware watcher: break on the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bd24c2241..99aeb8673 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -160,6 +160,7 @@ Runs automatically with `bun test`. No API keys needed. - **Skill validation tests** (`test/skill-validation.test.ts`) — Validates that SKILL.md files reference only real commands and flags, and that command descriptions meet quality thresholds. - **Generator tests** (`test/gen-skill-docs.test.ts`) — Tests the template system: verifies placeholders resolve correctly, output includes value hints for flags (e.g. `-d ` not just `-d`), enriched descriptions for key commands (e.g. `is` lists valid states, `press` lists key examples). - **Tier-alignment invariant** (`test/e2e-tier-alignment.test.ts`) — For every self-gated `test/skill-e2e-*.test.ts` named in a touchfiles dep list, the file's `EVALS_TIER` self-gate must match its declared tier in `E2E_TIERS`. Kills the "inert demotion" class where a test is re-tiered in `touchfiles.ts` but the file still gates on the old tier and keeps running in the wrong lane. Unmapped or mixed-tier files are reported, never silently skipped. +- **Catalog budget** (`test/catalog-budget.test.ts`) — Caps the aggregate discovery surface: the sum of every skill's frontmatter `name` + `description` (what every host loads at discovery, every session) must stay under 1,150 token-equivalents, with a 260-byte per-skill cap. Counting goes through the shared census in `test/helpers/skill-census.ts` (physical files vs authored skills vs registry entries — three deliberately different counts). Adding a skill? The failure message carries the re-measure + ratchet protocol. ### Tier 2: E2E via `claude -p` (~$3.85/run) @@ -184,10 +185,16 @@ seeded `CLAUDE_CONFIG_DIR`, a temp `GSTACK_HOME`, and `--strict-mcp-config`. You operator `~/.claude` config, MCP servers (gbrain, Conductor), skills, `~/.gstack` decision logs, and `CONDUCTOR_*` env never leak into the child, so local eval signal matches CI instead of disagreeing for reasons unrelated to the code under -test. Set `EVALS_HERMETIC=0` to debug against your real operator state (this also +test. The hermetic `CLAUDE_CONFIG_DIR` seeds no skills by default; a PTY test +that types a `/skill` slash command passes `seedSkills: true` to the PTY runner, +which swaps in `hermeticSkillsConfigDir()` — a seeded skill registry that +symlinks the LIVE working tree's SKILL.md files (by design: the skills are the +subject under test, so a snapshot would measure stale copies). Set +`EVALS_HERMETIC=0` to debug against your real operator state (this also drops `--strict-mcp-config`). The wiring is pinned by `test/hermetic-wiring.test.ts` -(a free static tripwire) and two gate-tier isolation canaries in -`test/skill-e2e-hermetic-canary.test.ts`. +(a free static tripwire), two gate-tier isolation canaries in +`test/skill-e2e-hermetic-canary.test.ts`, and the skill-seeding tripwires in +`test/hermetic-skills-seeding.test.ts` / `test/pty-skill-seeding-wiring.test.ts`. ### E2E observability @@ -227,8 +234,16 @@ bun run eval:bg:gate # detached gate-tier suite bun run eval:bg:periodic # detached periodic-tier suite ``` -Each prints its log path. Humans running `bun run test:evals` foreground in their -own terminal don't need this — Ctrl-C is intended there. +Each prints its log path. The gate and periodic variants run their tier through +the sharded paid runner (`scripts/test-paid-shards.ts`, also available directly +as `bun run test:gate:sharded` / `bun run test:periodic:sharded`): one Bun +process per test file, an external wall-clock timeout that kills the shard's +whole process group (stray `claude`/`codex` grandchildren included), a per-shard +eval dir (`GSTACK_EVAL_DIR=/shards//`), and an aggregate that +distinguishes failed vs timed-out vs never-started shards. `eval:list`, +`eval:compare`, and `eval:summary` are shard-aware. Humans running +`bun run test:evals` foreground in their own terminal don't need this — Ctrl-C +is intended there. **Eval comparison commentary:** `eval:compare` generates natural-language Takeaway sections interpreting what changed between runs — flagging regressions, noting improvements, calling out efficiency gains (fewer turns, faster, cheaper), and producing an overall summary. This is driven by `generateCommentary()` in `eval-store.ts`. diff --git a/README.md b/README.md index af534d2c5..9ef4b0851 100644 --- a/README.md +++ b/README.md @@ -235,7 +235,7 @@ Each skill feeds into the next. `/office-hours` writes a design doc that `/plan- | `/ios-qa` | **iOS Live-Device QA (v1.43.0.0+)** — drive a real iPhone over USB CoreDevice via an embedded `StateServer` in the app. Read Swift source, codegen typed `@Observable` accessors, run the agent loop. Optional `--tailnet` flag exposes the device to OpenClaw or any HTTP-capable agent on your Tailscale tailnet so remote agents can run iOS QA without ever touching the hardware. Capability-tier allowlist (observe/interact/mutate/restore), per-device session lock, audit log. | | `/ios-fix`, `/ios-design-review`, `/ios-clean`, `/ios-sync` | iOS bug-fix loop, designer's-eye HIG audit, debug-bridge cleanup, and accessor resync. See `docs/skills.md`. End-to-end walkthrough: [docs/howto-ios-testing-with-gstack.md](docs/howto-ios-testing-with-gstack.md). | -### New binaries (v0.19) +### Standalone binaries Beyond the slash-command skills, gstack ships standalone CLIs for workflows that don't belong inside a session: @@ -243,6 +243,8 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that |---------|-------------| | `gstack-model-benchmark` | **Cross-model benchmark** — run the same prompt through Claude, GPT (via Codex CLI), and Gemini; compare latency, tokens, cost, and (optionally) LLM-judge quality score. Auth detected per provider, unavailable providers skip cleanly. Output as table, JSON, or markdown. `--dry-run` validates flags + auth without spending API calls. | | `gstack-taste-update` | **Design taste learning** — writes approvals and rejections from `/design-shotgun` into a persistent per-project taste profile. Decays 5%/week. Feeds back into future variant generation so the system learns what you actually pick. | +| `gstack-egress` | **Egress receipt auditor** — every gstack-initiated off-machine send writes a tamper-evident, hash-chained receipt to `~/.gstack/security/egress.jsonl` before the send. `list` shows what gstack attempted to send and to which host, `grants` shows the standing consent settings plus the exact command that revokes each, `verify` recomputes the hash chain and exits 3 on tamper. | +| `gstack-context-bill` | **Token bill-of-materials** — read-only, offline audit of what an installed skills tree costs in tokens: always-on frontmatter every session pays vs per-invocation SKILL.md + forced references. `--diff` compares two trees, `--budget` enforces a ceiling, `--exact` opts into Anthropic `count_tokens` (sends file text off-machine; writes an egress receipt first, degrades to the offline estimate if the receipt can't be written). | | `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](docs/howto-ios-testing-with-gstack.md). | | `gstack-ios-qa-mint` | **iOS allowlist manager** — owner-grant CLI for the tailnet allowlist. `grant`/`revoke`/`list` against `~/.gstack/ios-qa-allowlist.json` (mode 0600). Remote agents never auto-allowlist; this is the explicit-intent path. | | `gstack-ios-qa-regen` | **iOS bridge regenerator** — deterministically installs the canonical DebugBridge package, generates typed state accessors, and records the installed gstack version. Safe to rerun after source changes or upgrades. | @@ -418,7 +420,7 @@ The skill asks once per repo. The decision is sticky across worktrees and branch **GStack memory sync (different feature, same private-repo infra).** Optionally pushes your gstack state (learnings, CEO plans, design docs, retros, developer profile) to a private git repo so your memory follows you across machines, with a one-time privacy prompt (everything allowlisted / artifacts only / off) and a defense-in-depth secret scanner that blocks AWS keys, tokens, PEM blocks, and JWTs before they leave your machine. ```bash -gstack-brain-init +gstack-artifacts-init ``` **Running gstack in Conductor?** Conductor explicitly strips `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` from every workspace's process env, so paid evals and gbrain embeddings won't work out of the box. Set `GSTACK_ANTHROPIC_API_KEY` and `GSTACK_OPENAI_API_KEY` in Conductor's workspace env config instead — gstack's TS entry points promote them to canonical names at runtime. Full details and the contributor checklist for adding the import to new entry points: [Conductor + GSTACK_* env vars](USING_GBRAIN_WITH_GSTACK.md#conductor--gstack_-env-vars). @@ -450,6 +452,7 @@ gstack includes **opt-in** usage telemetry to help improve the project. Here's e - **What's sent (if you opt in):** skill name, duration, success/fail, gstack version, OS. That's it. - **What's never sent:** code, file paths, repo names, branch names, prompts, or any user-generated content. - **Change anytime:** `gstack-config set telemetry off` disables everything instantly. +- **Every off-machine send is receipted.** Any gstack-initiated network send — telemetry included — writes a hash-chained, tamper-evident receipt to `~/.gstack/security/egress.jsonl` before the send; sensitive sinks refuse to send at all if the receipt can't be written. Audit with `gstack-egress list`, verify the chain with `gstack-egress verify` (exit 3 on tamper), see the standing consent settings with `gstack-egress grants`. The ledger records attempted sends so accidents are auditable — it's an audit trail, not a network firewall. Data is stored in [Supabase](https://supabase.com) (open source Firebase alternative). The schema is in [`supabase/migrations/`](supabase/migrations/) — you can verify exactly what's collected. The Supabase publishable key in the repo is a public key (like a Firebase API key) — row-level security policies deny all direct access. Telemetry flows through validated edge functions that enforce schema checks, event type allowlists, and field length limits. diff --git a/TODOS.md b/TODOS.md index d62f0ad29..d375009fe 100644 --- a/TODOS.md +++ b/TODOS.md @@ -875,32 +875,6 @@ plus a TTL so abandoned PTYs eventually exit. --- -### v1.1+: Audit `/health` token distribution - -**What:** Codex's outside-voice review on cc-pty-import flagged that -`/health` already surfaces `AUTH_TOKEN` to any localhost caller in headed -mode (`server.ts:1657`). That's a pre-existing soft leak — anything -running on localhost gets the root token by hitting `/health`. - -**Why:** cc-pty-import sidesteps it by NOT putting the PTY token there -(uses an HttpOnly cookie path instead). But the underlying leak is still -shippable surface. A second extension or a localhost web app could -currently scrape `AUTH_TOKEN` and hit any browse-server endpoint. - -**Pros:** Closes a real privilege-escalation path on multi-extension -machines. **Cons:** Either we tighten the gate (Origin must be OUR -extension id, not just any chrome-extension://) or we move bootstrap -discovery off `/health` entirely. Either has migration cost for tests -and the existing extension. - -**Context:** codex finding #2 on cc-pty-import plan-eng review. Not in -scope of that PR; deliberately deferred to keep PTY-import small. - -**Priority:** P2. -**Effort:** M. - ---- - ## Testing ## P2: Per-finding AskUserQuestion count assertion for /plan-ceo-review @@ -2654,3 +2628,125 @@ CI-hard-fail contract has to land five times. five green files at the tail of a release. Zero user-facing value; pure DRY. **Effort:** S (human ~3h, CC ~20min). **Depends on:** None. + +## Egress-receipt follow-ups (filed via /plan-eng-review + /codex on the v1.63 port wave) + +### P2: egress ledger rotation with chain-genesis records + +**What:** Rotate `~/.gstack/security/egress.jsonl` at a size threshold (match +`attempts.jsonl`'s 10MB/5-generation pattern in `browse/src/security.ts`), where +each new generation's FIRST record embeds the prior file's tail hash so +`gstack-egress verify` can walk across generations. + +**Why:** v1.63 ships WARN-at-25MB (visible growth) but nothing bounds the file. +Rotation was deliberately deferred: it changes the verify contract, and a wrong +implementation makes healthy ledgers verify as "broken". + +**Pros:** Bounded disk forever; verify stays meaningful across generations. +**Cons:** Chain-genesis semantics are subtle; needs its own focused tests +(cross-generation verify, mid-rotation crash). + +**Context:** `lib/egress-receipt.ts` (`appendChained`/`verifyLedger`) carries the +design sketch in its rotation TODO comment. Start from the `attempts.jsonl` +rotation precedent. + +**Effort:** S (human ~4h, CC ~25min). **Depends on:** v1.63 port wave landed. + +### P3: launch-nonce token bootstrap (local-process impersonation) + +**What:** Add a launch-time nonce to the `/extension-token` bootstrap: `browse` +mints a nonce at headed launch, seeds it into the extension (CDP +`chrome.storage` injection or a launcher-written sidecar), and the endpoint +requires it alongside the pinned origin. + +**Why:** v1.63's pinned-origin check authenticates browser contexts; any local +PROCESS can still forge an Origin header with curl. That threat is explicitly +outside the current model (any local process can hit the port anyway) — this +TODO documents the deliberate boundary and the designed path across it. + +**Pros:** Closes the local-process impersonation path (strongest of the three +options evaluated in the v1.63 plan review). +**Cons:** Largest bootstrap change; CDP seeding is fiddly across the three +launch paths (`--load-extension`, baked-in Browser.app, real-Chrome fallback); +low present-day value. + +**Context:** `browse/src/server.ts` `/extension-token` handler + +`GSTACK_EXTENSION_ID`; launch paths in `browse/src/browser-manager.ts` (~358, +~455, ~1562); `extension/background.js` bootstrap. + +**Effort:** M (human ~2 days, CC ~1h). **Depends on:** none. + +### P3: eval-watch shard-awareness + +**What:** Teach `scripts/eval-watch.ts` (hardcoded `_partial-e2e.json` path at +~line 17) about the sharded layout: watch `/shards/*/_partial-e2e.json` +and aggregate live progress across shard subdirs. + +**Why:** v1.63's sharded runner gives each shard its own eval subdir (so shards +baseline against their own priors); `findPreviousRun`, `eval-compare`, +`eval-list`, and `eval-summary` were all made shard-aware, but the live watcher +intentionally stayed flat — it shows nothing during sharded runs. + +**Pros:** Live progress during `eval:bg:gate` sharded runs again. +**Cons:** Multi-file watch + aggregation UI; low stakes (the run-scoped detach +log already streams per-shard results). + +**Context:** `scripts/eval-watch.ts`; shard layout defined in +`scripts/test-paid-shards.ts` (slug = test filename); `listEvalJsonFiles` in +`test/helpers/eval-store.ts` already enumerates the layout — reuse it. + +**Effort:** S (human ~2h, CC ~15min). **Depends on:** v1.63 port wave landed. + +## v1.63 port-wave review follow-ups (deferred from /ship review army — non-blocking polish) + +Genuine review findings deferred from the v1.63 ship because they are +informational/polish, not correctness-blocking, and several want their own +tests. Filed so they are tracked, not dropped. + +- **P2 — telemetry-sync HTTP-status outcome is dead code.** `_GSTACK_EGRESS_LAST_RECEIPT` + is set inside a command-substitution subshell in `bin/gstack-telemetry-sync`, so the + parent-shell guard that would append the HTTP status to the receipt never fires. The + generic `exit:N` outcome is still recorded, so the ledger is correct, just less + precise. Fix: have `_receipted_curl` persist the receipt id to a caller-readable temp + file, or restructure the call out of the subshell. (Confirmed by 3 review specialists.) +- **P2 — context-bill "TOTAL on disk" double-counts child skills** in a root-as-container + tree (this repo's own layout): `buildBill` sums the root skill's whole-tree walk plus + each child's subtree again (~2x the TOTAL line). ALWAYS-ON / EAGER / --diff / --budget + are all unaffected — only the informational TOTAL is wrong. Fix: compute the tree total + from a single deduplicated `walkMd(root)` pass, or exclude child dirs from the root + skill's `totalMd`. Needs a fixture test. (`lib/context-bill.ts`.) +- **P3 — DRY/robustness polish:** one shared `_gstack_egress_host_of` helper for the + ~11 hand-rolled URL-to-host extractions across the egress shell sinks; extract the + duplicated tunnel-open `writeReceipt` block in `browse/src/server.ts` (two sites); + hoist the per-iteration `SharedArrayBuffer` alloc out of the egress-receipt lock spin; + replace context-bill's exact-mode `errorPct === 0` sentinel with an explicit flag; + reuse `frontmatterName()` from `skill-census.ts` in `catalog-budget.test.ts`. +- **P3 — test-coverage gaps the audit named:** `PAID_TEST_GLOBS` ↔ `package.json` + `test:gate` parity test; `GSTACK_EXTENSION_ID` ↔ `manifest.json` key derivation parity + test (`browse/scripts/extension-id.ts`); a runner test asserting each shard child gets + its own `GSTACK_EVAL_DIR` under `shards/`; receipt-refusal branch tests for + supabase-provision / gbrain-sync / memory-ingest. + +## P2: harden or re-tier skill-e2e-plan-design-with-ui PTY detection + +**What:** The gate-tier `test/skill-e2e-plan-design-with-ui.test.ts` began executing +for the first time once v1.63's `seedSkills` registered skills in hermetic PTY +children (the fork had deleted this file; it measured nothing before). It now +reliably TIMES OUT even though the skill runs correctly: the transcript shows +`/plan-design-review` reaching its scope-gate AskUserQuestion (5 options, the +`` marker present), but the test's +`isNumberedOptionListVisible`/`parseNumberedOptions` scraping can't classify it out +of the PTY buffer because spinner frames (`[?25l✻Sprouting… still thinking`) are +interleaved character-by-character with the option text. + +**Why:** Shipped behavior is correct — this is a test-harness detection limitation, +not a product bug. But a gate test that always times out is worse than no test. + +**Fix options:** (a) harden the tail-scraping (drop DEC private-mode + spinner +residue before matching; widen/clean the window); (b) add an LLM-judge fallback +classifier (the file's own comments note the regex detectors are "brittle to PTY +rendering quirks"); or (c) move this test to periodic until (a)/(b) lands. + +**Context:** `test/skill-e2e-plan-design-with-ui.test.ts`, +`test/helpers/claude-pty-runner.ts:308` (`isNumberedOptionListVisible`). Evidence: +`~/.gstack-dev/eval-runs/pdwu-verify-*.log`. **Effort:** M (human ~half day / CC ~30min). diff --git a/USING_GBRAIN_WITH_GSTACK.md b/USING_GBRAIN_WITH_GSTACK.md index ec1144c9a..06e50faed 100644 --- a/USING_GBRAIN_WITH_GSTACK.md +++ b/USING_GBRAIN_WITH_GSTACK.md @@ -132,6 +132,8 @@ Storage: `~/.gstack/gbrain-repo-policy.json`, mode 0600, schema-versioned so fut The skill runs three stages — code, memory, brain-sync — independently. A failure in one doesn't block the others. State persists to `~/.gstack/.gbrain-sync-state.json` so re-running picks up cleanly. +Stages that can send data off-machine (code sync into a possibly-remote gbrain DB, memory ingest, the brain-sync push) each write a tamper-evident receipt to the egress ledger (`~/.gstack/security/egress.jsonl`) before sending, fail-closed: if the receipt can't be written, the stage refuses with `EGRESS_RECEIPT_FAILED` instead of syncing unrecorded. Fix is usually `mkdir -p ~/.gstack/security && chmod -R u+w ~/.gstack/security`, then re-run. Inspect receipts with `gstack-egress list`. + **What it does on a fresh worktree:** 1. **Pre-flight.** Checks `gbrain_local_status` (the local engine's health). If the engine is `broken-db` or `broken-config`, the skill STOPs with a remediation menu — it refuses to silently degrade. If the local engine is missing and you're in remote-MCP mode (Path 4), the code stage SKIPs cleanly and only brain-sync runs. @@ -167,14 +169,16 @@ This is different from gbrain itself. Your gstack state (`~/.gstack/` — learni Turn it on with: ```bash -gstack-brain-init +gstack-artifacts-init ``` You'll get a one-time privacy prompt: **everything allowlisted** / **artifacts only** (plans, designs, retros, learnings — skip behavioral data like timelines) / **off**. Every skill run syncs the queue at start and end — no daemon, no background process. Secret-shaped content (AWS keys, GitHub tokens, PEM blocks, JWTs, bearer tokens) is blocked from sync before it leaves your machine. -**On a new machine:** Copy `~/.gstack-brain-remote.txt` over, run `gstack-brain-restore`, and yesterday's learnings surface on today's laptop. +**On a new machine:** Copy `~/.gstack-artifacts-remote.txt` over (the legacy +`~/.gstack-brain-remote.txt` name still works), run `gstack-brain-restore`, and +yesterday's learnings surface on today's laptop. Full guide: [docs/gbrain-sync.md](docs/gbrain-sync.md). Error index: [docs/gbrain-sync-errors.md](docs/gbrain-sync-errors.md). @@ -239,7 +243,7 @@ Gbrain itself ships with these that gstack wraps: | `~/.gstack/.setup-gbrain.lock.d` | Concurrent-run lock (atomic mkdir). Released on normal exit + SIGINT. | | `~/.gstack/.brain-queue.jsonl` | Pending sync entries for gstack memory sync | | `~/.gstack/.brain-last-push` | Timestamp of last sync push (for `/health` scoring) | -| `~/.gstack-brain-remote.txt` | URL of your gstack memory sync remote (safe to copy between machines) | +| `~/.gstack-artifacts-remote.txt` | URL of your gstack memory sync remote (safe to copy between machines; legacy name `~/.gstack-brain-remote.txt` still read) | | `~/.gstack/.setup-gbrain-inflight.json` | Reserved for future `--resume-provision` persisted state | ### Environment variables diff --git a/VERSION b/VERSION index 1042f0adf..232d60188 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.62.0.0 +1.63.0.0 diff --git a/bin/gstack-artifacts-init b/bin/gstack-artifacts-init index b8bfe830c..8c8d32847 100755 --- a/bin/gstack-artifacts-init +++ b/bin/gstack-artifacts-init @@ -40,6 +40,16 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" URL_BIN="$SCRIPT_DIR/gstack-artifacts-url" REMOTE_FILE="$HOME/.gstack-artifacts-remote.txt" +# Egress receipt helpers (_receipted_git): fail-open for user-directed +# git ops against the user's own artifacts remote. +. "$SCRIPT_DIR/gstack-egress-lib.sh" + +# remote host for receipt records (github.com etc). Set once PUSH_URL exists. +_artifacts_host() { + local h="${PUSH_URL#*://}"; h="${h#*@}"; h="${h%%[/:]*}" + echo "${h:-unknown}" +} + REMOTE_URL="" HOST_PREF="" URL_FORM_SUPPORTED="false" @@ -185,7 +195,8 @@ PUSH_URL=$("$URL_BIN" --to ssh "$CANONICAL_HTTPS" 2>/dev/null || echo "$CANONICA # ---- verify push URL is reachable ---- echo "Verifying remote connectivity: $PUSH_URL" -if ! git ls-remote "$PUSH_URL" >/dev/null 2>&1; then +if ! _receipted_git open artifacts-init "$(_artifacts_host)" artifacts-remote-ls-remote "user ran gstack-artifacts-init" \ + bash -c 'git ls-remote "$1" >/dev/null 2>&1' _ "$PUSH_URL"; then cat >&2 </dev/null; then +if ! _receipted_git open artifacts-init "$(_artifacts_host)" artifacts-initial-push "user ran gstack-artifacts-init" \ + bash -c 'git push -q -u origin main 2>/dev/null'; then CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD) - if git fetch origin 2>/dev/null && git pull --ff-only origin "$CURRENT_BRANCH" 2>/dev/null; then - git push -q -u origin "$CURRENT_BRANCH" || { + if _receipted_git open artifacts-init "$(_artifacts_host)" artifacts-fetch "user ran gstack-artifacts-init" \ + bash -c 'git fetch origin 2>/dev/null' \ + && _receipted_git open artifacts-init "$(_artifacts_host)" artifacts-pull "user ran gstack-artifacts-init" \ + bash -c 'git pull --ff-only origin "$1" 2>/dev/null' _ "$CURRENT_BRANCH"; then + _receipted_git open artifacts-init "$(_artifacts_host)" artifacts-initial-push "user ran gstack-artifacts-init" \ + git push -q -u origin "$CURRENT_BRANCH" || { echo "Push to $PUSH_URL failed. The remote may have divergent content." >&2 echo "Try: cd ~/.gstack && git pull --rebase origin $CURRENT_BRANCH && git push origin $CURRENT_BRANCH" >&2 exit 1 diff --git a/bin/gstack-brain-consumer b/bin/gstack-brain-consumer deleted file mode 100755 index 12403ae58..000000000 --- a/bin/gstack-brain-consumer +++ /dev/null @@ -1,201 +0,0 @@ -#!/usr/bin/env bash -# gstack-brain-consumer — manage the consumer (reader) registry. -# -# DEPRECATED in v1.17.0.0. This binary targets a gbrain HTTP /ingest-repo -# endpoint that never shipped on the gbrain side. Live federation now uses -# `gbrain sources` directly via bin/gstack-gbrain-source-wireup. This file -# stays for one cycle to avoid breaking external scripts; removal in v1.18.0.0. -# -# Consumer = a reader that ingests the gstack-brain git repo as a source of -# session memory. v1 primary consumer is GBrain; later versions can register -# Codex, OpenClaw, or third-party readers. -# -# NOTE ON NAMING: internally this helper uses "consumer" (correct data-model -# term). User-facing copy and the alias `gstack-brain-reader` use "reader" -# (matches user mental model: "what's reading my brain?"). -# -# Usage: -# gstack-brain-consumer add --ingest-url --token -# gstack-brain-consumer list -# gstack-brain-consumer remove -# gstack-brain-consumer test -# -# Env: -# GSTACK_HOME — override ~/.gstack - -set -euo pipefail - -GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}" -CONSUMERS_FILE="$GSTACK_HOME/consumers.json" -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -CONFIG_BIN="$SCRIPT_DIR/gstack-config" - -ensure_file() { - mkdir -p "$GSTACK_HOME" - if [ ! -f "$CONSUMERS_FILE" ]; then - echo '{"consumers": []}' > "$CONSUMERS_FILE" - fi -} - -get_remote_url() { - git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "" -} - -sub_add() { - local name="" url="" token="" - local positional="" - while [ $# -gt 0 ]; do - case "$1" in - --ingest-url) url="$2"; shift 2 ;; - --token) token="$2"; shift 2 ;; - --) shift; break ;; - -*) echo "Unknown flag: $1" >&2; exit 1 ;; - *) positional="$1"; shift ;; - esac - done - name="$positional" - if [ -z "$name" ] || [ -z "$url" ]; then - echo "Usage: gstack-brain-consumer add --ingest-url [--token ]" >&2 - exit 1 - fi - ensure_file - # Upsert in consumers.json, store token in gstack-config under `_token`. - python3 - "$CONSUMERS_FILE" "$name" "$url" <<'PYEOF' -import sys, json -path, name, url = sys.argv[1:4] -try: - with open(path) as f: - data = json.load(f) -except Exception: - data = {"consumers": []} -entry = {"name": name, "ingest_url": url, "status": "unknown", "token_ref": f"{name}_token"} -cs = data.setdefault("consumers", []) -for i, c in enumerate(cs): - if c.get("name") == name: - cs[i] = entry - break -else: - cs.append(entry) -with open(path, "w") as f: - json.dump(data, f, indent=2) - f.write("\n") -print(f"registered consumer: {name}") -PYEOF - if [ -n "$token" ]; then - "$CONFIG_BIN" set "${name}_token" "$token" - echo "token stored: gstack-config get ${name}_token to retrieve" - fi - # Attempt registration with remote (HTTP POST). - sub_test "$name" -} - -sub_list() { - if [ ! -f "$CONSUMERS_FILE" ]; then - echo '{"consumers": []}' - return 0 - fi - cat "$CONSUMERS_FILE" -} - -sub_remove() { - local name="${1:-}" - if [ -z "$name" ]; then - echo "Usage: gstack-brain-consumer remove " >&2 - exit 1 - fi - ensure_file - python3 - "$CONSUMERS_FILE" "$name" <<'PYEOF' -import sys, json -path, name = sys.argv[1:3] -try: - with open(path) as f: - data = json.load(f) -except Exception: - data = {"consumers": []} -before = len(data.get("consumers", [])) -data["consumers"] = [c for c in data.get("consumers", []) if c.get("name") != name] -after = len(data["consumers"]) -with open(path, "w") as f: - json.dump(data, f, indent=2) - f.write("\n") -print(f"removed: {before - after} entry(ies)") -PYEOF -} - -sub_test() { - local name="${1:-}" - if [ -z "$name" ]; then - echo "Usage: gstack-brain-consumer test " >&2 - exit 1 - fi - ensure_file - # Look up the consumer by name. - local info - info=$(python3 - "$CONSUMERS_FILE" "$name" <<'PYEOF' -import sys, json -path, name = sys.argv[1:3] -try: - with open(path) as f: - data = json.load(f) -except Exception: - data = {"consumers": []} -for c in data.get("consumers", []): - if c.get("name") == name: - print(c.get("ingest_url", "")) - sys.exit(0) -sys.exit(1) -PYEOF - ) || { echo "No such consumer: $name" >&2; exit 1; } - - local url="$info" - local token - token=$("$CONFIG_BIN" get "${name}_token" 2>/dev/null || echo "") - if [ -z "$url" ] || [ -z "$token" ]; then - echo "consumer '$name': url or token missing; cannot test" - return 0 - fi - local repo_url - repo_url=$(get_remote_url) - echo "Testing $name at ${url%/}/ingest-repo ..." - local resp - resp=$(curl -sS -X POST "${url%/}/ingest-repo" \ - -H "Authorization: Bearer $token" \ - -H "Content-Type: application/json" \ - --data "{\"repo_url\":\"$repo_url\"}" \ - -w "\n%{http_code}" 2>&1 || echo -e "\ncurl-error") - local code - code=$(echo "$resp" | tail -1) - if [ "$code" = "200" ] || [ "$code" = "201" ] || [ "$code" = "204" ]; then - echo "ok (HTTP $code)" - # Update status in consumers.json. - python3 - "$CONSUMERS_FILE" "$name" "ok" <<'PYEOF' -import sys, json -path, name, status = sys.argv[1:4] -with open(path) as f: data = json.load(f) -for c in data.get("consumers", []): - if c.get("name") == name: - c["status"] = status -with open(path, "w") as f: json.dump(data, f, indent=2); f.write("\n") -PYEOF - else - echo "failed (HTTP $code)" - python3 - "$CONSUMERS_FILE" "$name" "error" <<'PYEOF' -import sys, json -path, name, status = sys.argv[1:4] -with open(path) as f: data = json.load(f) -for c in data.get("consumers", []): - if c.get("name") == name: - c["status"] = status -with open(path, "w") as f: json.dump(data, f, indent=2); f.write("\n") -PYEOF - fi -} - -case "${1:-}" in - add) shift; sub_add "$@" ;; - list) sub_list ;; - remove) shift; sub_remove "$@" ;; - test) shift; sub_test "$@" ;; - --help|-h|"") sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//' ;; - *) echo "Unknown subcommand: $1" >&2; exit 1 ;; -esac diff --git a/bin/gstack-brain-reader b/bin/gstack-brain-reader deleted file mode 120000 index 712ce87e6..000000000 --- a/bin/gstack-brain-reader +++ /dev/null @@ -1 +0,0 @@ -gstack-brain-consumer \ No newline at end of file diff --git a/bin/gstack-brain-restore b/bin/gstack-brain-restore index 21f7c1134..bab38f55f 100755 --- a/bin/gstack-brain-restore +++ b/bin/gstack-brain-restore @@ -30,6 +30,10 @@ set -euo pipefail GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" CONFIG_BIN="$SCRIPT_DIR/gstack-config" + +# Egress receipt helpers (_receipted_git): fail-open for user-directed +# git ops against the user's own artifacts remote. +. "$SCRIPT_DIR/gstack-egress-lib.sh" # v1.27.0.0+ canonical name; brain-remote is the legacy fallback during the # migration window. The migration script renames the file in place. if [ -f "$HOME/.gstack-artifacts-remote.txt" ]; then @@ -78,7 +82,9 @@ STAGING=$(mktemp -d "${TMPDIR:-/tmp}/gstack-brain-restore.XXXXXX") trap 'rm -rf "$STAGING" 2>/dev/null' EXIT echo "Cloning $REMOTE_URL to staging..." -if ! git clone --quiet "$REMOTE_URL" "$STAGING/repo" 2>/dev/null; then +RESTORE_HOST="${REMOTE_URL#*://}"; RESTORE_HOST="${RESTORE_HOST#*@}"; RESTORE_HOST="${RESTORE_HOST%%[/:]*}" +if ! _receipted_git open brain-restore "${RESTORE_HOST:-unknown}" brain-restore-clone "user ran gstack-brain-restore" \ + bash -c 'git clone --quiet "$1" "$2" 2>/dev/null' _ "$REMOTE_URL" "$STAGING/repo"; then echo "Clone failed. Check:" >&2 echo " - URL is correct: $REMOTE_URL" >&2 echo " - Auth: gh auth status (github) / glab auth status (gitlab)" >&2 @@ -160,7 +166,8 @@ done # ---- move .git into place ---- if [ -d "$GSTACK_HOME/.git" ]; then # Existing .git with matching remote — just fetch + fast-forward. - git -C "$GSTACK_HOME" fetch origin >/dev/null 2>&1 || true + _receipted_git open brain-restore "${RESTORE_HOST:-unknown}" brain-restore-fetch "user ran gstack-brain-restore" \ + bash -c 'git -C "$1" fetch origin >/dev/null 2>&1' _ "$GSTACK_HOME" || true else mv "$STAGING/repo/.git" "$GSTACK_HOME/.git" fi diff --git a/bin/gstack-brain-sync b/bin/gstack-brain-sync index 2ad0fe773..2fa696869 100755 --- a/bin/gstack-brain-sync +++ b/bin/gstack-brain-sync @@ -32,6 +32,17 @@ DISCOVER_CURSOR="$GSTACK_HOME/.brain-discover-cursor" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" CONFIG_BIN="$SCRIPT_DIR/gstack-config" +# Egress receipt helpers (_receipted_git): receipt-before-send, fail-closed. +. "$SCRIPT_DIR/gstack-egress-lib.sh" + +# origin host for receipt records (github.com etc). +remote_host() { + local url host + url=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "") + host="${url#*://}"; host="${host#*@}"; host="${host%%[/:]*}" + echo "${host:-unknown}" +} + # Remote-specific hint for auth errors (branch on origin URL). remote_auth_hint() { local url @@ -276,6 +287,21 @@ subcmd_once() { exit 0 fi + # Egress receipt for the push, written BEFORE the commit consumes the + # queue (amendment C7 ordering): a refused receipt exits HERE, before any + # queue mutation or local commit, so the queue stays intact and the next + # run retries the whole drain. Content-free: git owns the bytes + # (sha256:null). Fail-closed. + local push_host receipt_err + push_host=$(remote_host) + if ! receipt_err=$(GSTACK_HOME="$GSTACK_HOME" "$SCRIPT_DIR/gstack-egress-receipt" write \ + --sink brain-sync --host "$push_host" --class curated-memory-git-push \ + --no-payload --consent "artifacts_sync_mode!=off" 2>&1 >/dev/null); then + write_status "push_failed" "EGRESS_RECEIPT_FAILED: receipt not writable; push refused (queue preserved)" + _gstack_egress_refusal "brain-sync push" "$(printf '%s' "$receipt_err" | head -c 300)" + exit 1 + fi + # Commit with template message. local n ts n=$(wc -l < "$paths_file" | tr -d ' ') @@ -303,12 +329,16 @@ subcmd_once() { exit 0 fi - # Try a fetch-and-merge + retry. - if git -C "$GSTACK_HOME" fetch origin 2>/dev/null; then + # Try a fetch-and-merge + retry. The fetch and the retry push are their + # own attempted-egress ops, each receipted fail-closed (a refusal falls + # through to the push_failed path below). + if GSTACK_HOME="$GSTACK_HOME" _receipted_git closed brain-sync "$push_host" curated-memory-git-fetch "artifacts_sync_mode!=off" \ + bash -c 'git -C "$1" fetch origin 2>/dev/null' _ "$GSTACK_HOME"; then local branch branch=$(git -C "$GSTACK_HOME" rev-parse --abbrev-ref HEAD 2>/dev/null || echo main) if git -C "$GSTACK_HOME" merge --no-edit "origin/$branch" >/dev/null 2>&1; then - if git -C "$GSTACK_HOME" push origin HEAD 2>/dev/null; then + if GSTACK_HOME="$GSTACK_HOME" _receipted_git closed brain-sync "$push_host" curated-memory-git-push "artifacts_sync_mode!=off" \ + bash -c 'git -C "$1" push origin HEAD 2>/dev/null' _ "$GSTACK_HOME"; then : > "$QUEUE" date -u +%Y-%m-%dT%H:%M:%SZ > "$LAST_PUSH_FILE" write_status "ok" "pushed $n file(s) after rebase" diff --git a/bin/gstack-community-dashboard b/bin/gstack-community-dashboard index adb94cb33..2b40b25b7 100755 --- a/bin/gstack-community-dashboard +++ b/bin/gstack-community-dashboard @@ -12,6 +12,9 @@ set -uo pipefail GSTACK_DIR="${GSTACK_DIR:-$(cd "$(dirname "$0")/.." && pwd)}" +# Egress receipt helpers (_receipted_curl): fail-open for read-only stats. +. "$GSTACK_DIR/bin/gstack-egress-lib.sh" + # Source Supabase config if not overridden by env if [ -z "${GSTACK_SUPABASE_URL:-}" ] && [ -f "$GSTACK_DIR/supabase/config.sh" ]; then . "$GSTACK_DIR/supabase/config.sh" @@ -35,10 +38,11 @@ fi # never as a healthy "Weekly active installs: 0". TMPBODY="$(mktemp)" trap 'rm -f "$TMPBODY"' EXIT -HTTP_CODE="$(curl -s --max-time 15 -w '%{http_code}' -o "$TMPBODY" \ +SUPA_HOST="${SUPABASE_URL#*://}"; SUPA_HOST="${SUPA_HOST%%/*}" +HTTP_CODE="$(_receipted_curl open community-dashboard "$SUPA_HOST" community-pulse-fetch "user-invoked dashboard" --no-payload \ + curl -s --max-time 15 -w '%{http_code}' -o "$TMPBODY" \ "${SUPABASE_URL}/functions/v1/community-pulse" \ - -H "apikey: ${ANON_KEY}" \ - 2>/dev/null || true)" + -H "apikey: ${ANON_KEY}" || true)" # curl prints its own 000 before a non-zero exit — a `|| echo` here would # double it to "000000" in user-facing output. Normalize to the last 3 chars. HTTP_CODE="$(printf '%s' "$HTTP_CODE" | tr -d '[:space:]' | tail -c 3)" diff --git a/bin/gstack-context-bill b/bin/gstack-context-bill new file mode 100755 index 000000000..50b085700 --- /dev/null +++ b/bin/gstack-context-bill @@ -0,0 +1,7 @@ +#!/usr/bin/env bun +// gstack-context-bill — token bill-of-materials for an installed skills tree. +// All behavior lives in lib/context-bill.ts; this is the CLI shim. + +import { contextBillMain } from '../lib/context-bill'; + +process.exit(await contextBillMain(process.argv.slice(2))); diff --git a/bin/gstack-egress b/bin/gstack-egress new file mode 100755 index 000000000..246448ef0 --- /dev/null +++ b/bin/gstack-egress @@ -0,0 +1,207 @@ +#!/usr/bin/env bun +/** + * gstack-egress — the auditor's view of the receipts ledger. + * + * list what gstack ATTEMPTED to send off this machine (one row per receipt) + * grants what CAN leave: every consent grant in force, where it lives, + * and the exact command that revokes it (pure config reads) + * verify recompute the hash chain; exit 3 on tamper + * + * THREAT MODEL: the ledger is forensic observability — it records ATTEMPTED + * egress so accidents are auditable; it is not an exfiltration control. + * + * The ledger is written by lib/egress-receipt.ts at every enumerated sink + * (see test/egress-receipt-wiring.test.ts for the pinned list). + * + * Home: GSTACK_HOME, legacy GSTACK_STATE_DIR, else ~/.gstack. + */ + +import path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { + egressLedgerPath, + listReceipts, + resolveEgressHome, + verifyLedger, +} from '../lib/egress-receipt'; + +// import.meta.dir is Windows-safe; new URL(import.meta.url).pathname yields +// '/C:/...' with percent-encoded spaces there, which would make the +// gstack-config spawn silently fail and grants report every default. Matches +// the sibling bin/*.ts convention. +const BIN_DIR = import.meta.dir; + +/** + * Strip control characters (incl. ANSI escapes) from ledger-derived strings + * before printing. A receipt's host/payloadClass can derive from semi-trusted + * input (a URL argument, a git remote); JSON.parse restores \u001b escapes to + * live ESC bytes, so an attacker-shaped receipt could hide or spoof rows in + * the very output an auditor reads. The hash chain is unaffected; this only + * sanitizes the human render. + */ +function sanitizeForDisplay(value: unknown): string { + return String(value).replace(/[\u0000-\u001F\u007F]/g, ''); +} + +function usage(message: string): never { + process.stderr.write(`gstack-egress: ${message}\n`); + process.stderr.write( + 'Usage: gstack-egress list [--since ] [--host ] [--sink ] [--json]\n' + + ' gstack-egress verify [--json]\n' + + ' gstack-egress grants [--json]\n', + ); + process.exit(2); +} + +function configGet(key: string): string { + const result = spawnSync(path.join(BIN_DIR, 'gstack-config'), ['get', key], { + encoding: 'utf-8', + }); + return (result.stdout || '').trim(); +} + +function egressList(args: string[], home: string): number { + const values = new Map(); + let json = false; + for (let index = 0; index < args.length; index += 1) { + const arg = args[index]; + if (['--since', '--host', '--sink'].includes(arg)) { + const value = args[++index]; + if (value == null || value.startsWith('--')) usage(`${arg} requires a value`); + values.set(arg, value); + } else if (arg === '--json') { + json = true; + } else { + usage(`unknown option: ${arg}`); + } + } + const since = values.get('--since'); + if (since != null && !Number.isFinite(Date.parse(since))) usage('--since must be an ISO timestamp'); + + let receipts = listReceipts(home); + if (since) receipts = receipts.filter((r) => Date.parse(r.ts) >= Date.parse(since)); + if (values.has('--host')) receipts = receipts.filter((r) => r.host === values.get('--host')); + if (values.has('--sink')) receipts = receipts.filter((r) => r.sink === values.get('--sink')); + + if (json) { + process.stdout.write(`${JSON.stringify(receipts, null, 2)}\n`); + return 0; + } + if (!receipts.length) { + process.stdout.write(`no receipts\nledger: ${egressLedgerPath(home)}\n`); + return 0; + } + for (const r of receipts) { + // sink/host/payload_class/consent can carry semi-trusted content — strip + // control chars so a crafted receipt can't spoof the auditor's view. ts, + // bytes, and sha256 are format-constrained at write time. + process.stdout.write( + `${r.ts} ${sanitizeForDisplay(r.sink)} -> ${sanitizeForDisplay(r.host)} ` + + `${sanitizeForDisplay(r.payload_class)} ${r.bytes}B ` + + `sha256=${r.sha256 ?? '(subprocess-owned)'} consent=${sanitizeForDisplay(r.consent)} ` + + `status=${r.status ? sanitizeForDisplay(r.status) : '-'}\n`, + ); + } + process.stdout.write(`${receipts.length} receipt(s) ledger: ${egressLedgerPath(home)}\n`); + return 0; +} + +function egressVerify(args: string[], home: string): number { + for (const arg of args) if (arg !== '--json') usage(`unknown option: ${arg}`); + const result = verifyLedger(home); + if (args.includes('--json')) { + process.stdout.write(`${JSON.stringify({ ...result, ledger: egressLedgerPath(home) }, null, 2)}\n`); + } else { + if (result.ok) { + process.stdout.write(`chain intact: ${result.count} line(s) verified\n`); + } else { + process.stdout.write(`TAMPER: chain broken at line ${result.brokenLine} (${result.reason})\n`); + } + if (result.sizeWarning) process.stdout.write(`${result.sizeWarning}\n`); + } + return result.ok ? 0 : 3; +} + +interface Grant { + grant: string; + value: string; + granted: boolean; + detail: string; + file: string; + key: string; + revoke: string; +} + +function egressGrants(args: string[], home: string): number { + for (const arg of args) if (arg !== '--json') usage(`unknown option: ${arg}`); + const configFile = path.join(home, 'config.yaml'); + + const telemetry = configGet('telemetry') || 'off'; + const syncMode = configGet('artifacts_sync_mode') || 'off'; + const repoVisibility = configGet('redact_repo_visibility') || 'unknown'; + const prepushHook = configGet('redact_prepush_hook') || 'false'; + + const grants: Grant[] = [ + { + grant: 'telemetry', + value: telemetry, + granted: telemetry !== 'off', + detail: 'anonymous and community tiers upload usage events to Supabase; off stays local-only', + file: configFile, + key: 'telemetry', + revoke: 'gstack-config set telemetry off', + }, + { + grant: 'brain-sync', + value: syncMode, + granted: syncMode !== 'off' && syncMode !== '', + detail: 'git push of curated allowlisted memory to the user-configured artifacts remote', + file: configFile, + key: 'artifacts_sync_mode', + revoke: 'gstack-config set artifacts_sync_mode off', + }, + { + grant: 'redact_repo_visibility', + value: repoVisibility, + granted: repoVisibility === 'public', + detail: 'redaction strictness assumption for external sinks; public gets per-finding confirmation', + file: configFile, + key: 'redact_repo_visibility', + revoke: 'gstack-config set redact_repo_visibility unknown (unknown = public-strict)', + }, + { + grant: 'redact_prepush_hook', + value: prepushHook, + granted: prepushHook === 'true', + detail: 'opt-in git pre-push redaction scan; granted here means the guard is ON', + file: configFile, + key: 'redact_prepush_hook', + revoke: 'gstack-config set redact_prepush_hook false (disables the guard)', + }, + ]; + + if (args.includes('--json')) { + process.stdout.write(`${JSON.stringify(grants, null, 2)}\n`); + return 0; + } + for (const grant of grants) { + process.stdout.write( + `${grant.granted ? '[GRANTED]' : '[off] '} ${grant.grant}: ${grant.value}\n` + + ` ${grant.detail}\n` + + ` lives in: ${grant.file} (${grant.key})\n` + + ` revoke: ${grant.revoke}\n`, + ); + } + process.stdout.write("What was ATTEMPTED: 'gstack-egress list'. Chain check: 'gstack-egress verify'.\n"); + return 0; +} + +const [action, ...rest] = process.argv.slice(2); +const home = resolveEgressHome(); + +let code: number; +if (action === 'list') code = egressList(rest, home); +else if (action === 'verify') code = egressVerify(rest, home); +else if (action === 'grants') code = egressGrants(rest, home); +else usage(`unknown subcommand: ${action ?? '(none)'}`); +process.exit(code); diff --git a/bin/gstack-egress-lib.sh b/bin/gstack-egress-lib.sh new file mode 100644 index 000000000..37fe045a7 --- /dev/null +++ b/bin/gstack-egress-lib.sh @@ -0,0 +1,118 @@ +# gstack-egress-lib.sh — shared egress-receipt helpers for bash sinks. +# +# This file is NOT executable; source it: +# +# . "$(dirname "$0")/gstack-egress-lib.sh" +# +# THREAT MODEL: the egress ledger is forensic observability — it records +# ATTEMPTED egress so accidents are auditable; it is not an exfiltration +# control. Receipts are written before send, outcomes are best-effort, and +# fail-open callers can send unrecorded with a warning. +# +# Provides: +# _receipted_curl \ +# (|--no-payload) [args...] +# — Writes an egress receipt BEFORE running the wrapped curl command. +# When a payload file is given, the receipt hashes that file and curl +# receives THE SAME FILE via an appended `--data-binary @file`, so the +# recorded sha256 matches the exact wire bytes (scan-at-sink precedent). +# The payload file is CONSUMED: the helper deletes it after the send +# (and on refusal) — callers need no cleanup trap for it. +# stdout is the wrapped command's stdout; returns its exit code. +# A best-effort outcome record (`exit:N`) is appended after the send; +# the receipt id is exported as _GSTACK_EGRESS_LAST_RECEIPT so callers +# can append a more specific outcome (e.g. the HTTP status). +# +# Fail policy (first argument, per call): +# closed — receipt failure REFUSES the send: nothing hits the network, +# return 3, problem/cause/fix message on stderr. +# open — receipt failure warns on stderr and proceeds unrecorded. +# +# _receipted_git [args...] +# — Same contract for git-class ops: sha256:null receipt (a subprocess +# owns the bytes), no payload file, command runs unmodified. +# +# NO EXIT traps in this file, ever: callers (gstack-telemetry-sync) own +# their own EXIT traps and a trap set by a sourced library would clobber +# the caller's. All temp handling is immediate, per call. + +# Self-locate without dirname (builtins only), so the lib works even under +# a stripped test PATH. +case "${BASH_SOURCE[0]}" in + */*) _gstack_egress_lib_dir="$(cd "${BASH_SOURCE[0]%/*}" && pwd)" ;; + *) _gstack_egress_lib_dir="$(pwd)" ;; +esac + +_gstack_egress_home() { + if [ -n "${GSTACK_HOME:-}" ]; then + printf '%s' "$GSTACK_HOME" + elif [ -n "${GSTACK_STATE_DIR:-}" ]; then + printf '%s' "$GSTACK_STATE_DIR" + else + printf '%s' "$HOME/.gstack" + fi +} + +# Problem + cause + fix, in plain language (DX contract for every +# fail-closed refusal). $1 = sink, $2 = cause text from the receipt bridge. +_gstack_egress_refusal() { + local home + home="$(_gstack_egress_home)" + echo "gstack: $1 NOT sent — the egress receipt could not be written (${2:-unknown cause}). Fix: chmod -R u+w $home/security (or check GSTACK_HOME). What this is: gstack records everything it ATTEMPTS to send off-machine; see gstack-egress." >&2 +} + +# Shared core. $6 is a payload file path or --no-payload; the rest is the +# command to run. Payload files are deleted here (immediate, no traps). +_gstack_egress_run() { + local policy="$1" sink="$2" host="$3" class="$4" consent="$5" payload="$6" + shift 6 + local bin="$_gstack_egress_lib_dir/gstack-egress-receipt" + + local payload_flag=(--no-payload) + [ "$payload" != "--no-payload" ] && payload_flag=(--payload-file "$payload") + + local receipt_id="" receipt_err="" err_file="" + err_file="$(mktemp "${TMPDIR:-/tmp}/gstack-egress-err-XXXXXX" 2>/dev/null)" || err_file="" + if [ -n "$err_file" ]; then + receipt_id="$("$bin" write --sink "$sink" --host "$host" --class "$class" \ + "${payload_flag[@]}" --consent "$consent" 2>"$err_file")" || receipt_id="" + receipt_err="$(head -c 500 "$err_file" 2>/dev/null | tr '\n' ' ')" + rm -f "$err_file" + else + receipt_id="$("$bin" write --sink "$sink" --host "$host" --class "$class" \ + "${payload_flag[@]}" --consent "$consent" 2>/dev/null)" || receipt_id="" + fi + _GSTACK_EGRESS_LAST_RECEIPT="$receipt_id" + + if [ -z "$receipt_id" ]; then + if [ "$policy" = "closed" ]; then + _gstack_egress_refusal "$sink" "$receipt_err" + [ "$payload" != "--no-payload" ] && rm -f "$payload" + return 3 + fi + echo "gstack: egress receipt could not be written for $sink (${receipt_err:-unknown cause}) — sending anyway (fail-open). gstack normally records everything it ATTEMPTS to send off-machine; see gstack-egress." >&2 + fi + + local status=0 + if [ "$payload" = "--no-payload" ]; then + "$@" || status=$? + else + "$@" --data-binary @"$payload" || status=$? + rm -f "$payload" + fi + + if [ -n "$receipt_id" ]; then + "$bin" outcome "$receipt_id" "exit:$status" 2>/dev/null || true + fi + return $status +} + +_receipted_curl() { + _gstack_egress_run "$@" +} + +_receipted_git() { + local policy="$1" sink="$2" host="$3" class="$4" consent="$5" + shift 5 + _gstack_egress_run "$policy" "$sink" "$host" "$class" "$consent" --no-payload "$@" +} diff --git a/bin/gstack-egress-receipt b/bin/gstack-egress-receipt new file mode 100755 index 000000000..b2d903d53 --- /dev/null +++ b/bin/gstack-egress-receipt @@ -0,0 +1,84 @@ +#!/usr/bin/env bun +// gstack-egress-receipt — bun script that BRIDGES shell callers (the bash +// egress sinks: gstack-telemetry-sync, gstack-update-check, gstack-brain-sync, +// and the sourced helpers in gstack-egress-lib.sh) into lib/egress-receipt.ts. +// +// Usage: +// gstack-egress-receipt write --sink S --host H --class C \ +// (--payload-file F | --no-payload) [--consent "key=value"] +// → prints the receipt id on stdout, exit 0. +// → exit 3 + "EGRESS_RECEIPT_FAILED: ..." on stderr when the receipt +// cannot be written. Fail-closed callers MUST then refuse the send. +// +// gstack-egress-receipt outcome +// → best-effort response-status record; never blocks anything. +// +// Home: GSTACK_HOME, legacy GSTACK_STATE_DIR, else ~/.gstack. +// The payload file is hashed as-is: pass the SAME file to curl (`-d @file`) +// so the receipt hash matches the exact bytes sent (scan-at-sink precedent). + +import fs from 'node:fs'; +import { EGRESS_RECEIPT_FAILED, sha256Hex, writeOutcome, writeReceipt } from '../lib/egress-receipt'; + +function parseArgs(args: string[], valueFlags: string[], boolFlags: string[]) { + const values = new Map(); + const flags = new Set(); + for (let index = 0; index < args.length; index += 1) { + const arg = args[index]; + if (valueFlags.includes(arg)) { + const value = args[++index]; + if (value == null) usage(`${arg} requires a value`); + values.set(arg, value); + } else if (boolFlags.includes(arg)) { + flags.add(arg); + } else { + usage(`unknown option: ${arg}`); + } + } + return { values, flags }; +} + +function usage(message: string): never { + process.stderr.write(`gstack-egress-receipt: ${message}\n`); + process.exit(2); +} + +const [command, ...rest] = process.argv.slice(2); + +if (command === 'write') { + const { values, flags } = parseArgs(rest, + ['--sink', '--host', '--class', '--payload-file', '--consent'], ['--no-payload']); + const payloadFile = values.get('--payload-file'); + if (!payloadFile && !flags.has('--no-payload')) usage('write requires --payload-file or --no-payload'); + try { + let bytes = 0; + let sha256: string | null = null; // --no-payload = git-class op, subprocess owns the bytes + if (payloadFile) { + const payload = fs.readFileSync(payloadFile); + bytes = payload.byteLength; + sha256 = sha256Hex(payload); + } + const { id } = writeReceipt({ + sink: values.get('--sink') as string, + host: values.get('--host') as string, + payloadClass: values.get('--class') as string, + bytes, + sha256, + consent: values.get('--consent') ?? 'unspecified', + }); + process.stdout.write(`${id}\n`); + } catch (error) { + process.stderr.write(`${EGRESS_RECEIPT_FAILED}: ${(error as Error)?.message ?? error}\n`); + process.exit(3); + } +} else if (command === 'outcome') { + const [receipt, status, ...extra] = rest; + if (!receipt || !status || extra.length) usage('Usage: gstack-egress-receipt outcome '); + try { + writeOutcome({ receipt, status }); + } catch { + // Best-effort: the pre-send receipt is the invariant, the outcome is bookkeeping. + } +} else { + usage('Usage: gstack-egress-receipt write|outcome ...'); +} diff --git a/bin/gstack-gbrain-mcp-verify b/bin/gstack-gbrain-mcp-verify index 72129a866..b3459ca8f 100755 --- a/bin/gstack-gbrain-mcp-verify +++ b/bin/gstack-gbrain-mcp-verify @@ -43,6 +43,10 @@ URL="$1" command -v curl >/dev/null 2>&1 || { echo "gstack-gbrain-mcp-verify: curl is required" >&2; exit 2; } command -v jq >/dev/null 2>&1 || { echo "gstack-gbrain-mcp-verify: jq is required (brew install jq)" >&2; exit 2; } +# Egress receipt helpers (_receipted_curl): receipt-before-send, fail-closed. +. "$(cd "$(dirname "$0")" && pwd)/gstack-egress-lib.sh" +MCP_HOST=$(echo "$URL" | sed -E 's|^[a-z]+://([^/]+).*|\1|') + emit() { # emit jq -n \ @@ -73,15 +77,20 @@ INIT_BODY='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVers TMPBODY=$(mktemp -t gstack-mcp-verify.XXXXXX) trap 'rm -f "$TMPBODY"' EXIT +# Receipted fail-closed: the payload file is hashed and handed to curl as +# the exact wire bytes. A refused receipt never hits the network — it lands +# in the NETWORK class below (curl never ran, no HTTP code). +INIT_PAYLOAD=$(mktemp -t gstack-mcp-init.XXXXXX) +printf '%s' "$INIT_BODY" > "$INIT_PAYLOAD" set +e -HTTP_CODE=$(curl -s -o "$TMPBODY" -w '%{http_code}' \ +HTTP_CODE=$(_receipted_curl closed gbrain-mcp-verify "$MCP_HOST" mcp-initialize-probe "user-invoked mcp verify" "$INIT_PAYLOAD" \ + curl -s -o "$TMPBODY" -w '%{http_code}' \ --max-time 10 \ -X POST \ -H 'Content-Type: application/json' \ -H 'Accept: application/json, text/event-stream' \ -H "Authorization: Bearer $GBRAIN_MCP_TOKEN" \ - -d "$INIT_BODY" \ - "$URL" 2>/dev/null) + "$URL") CURL_EXIT=$? set -e @@ -150,14 +159,18 @@ URL_SUPPORTED=false TOOLS_BODY_FILE=$(mktemp -t gstack-mcp-tools.XXXXXX) TOOLS_REQ='{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' +# Receipted fail-closed like the initialize probe. A refused receipt skips +# the probe (nonzero TOOLS_EXIT) and the field stays false — best-effort. +TOOLS_PAYLOAD=$(mktemp -t gstack-mcp-tools-req.XXXXXX) +printf '%s' "$TOOLS_REQ" > "$TOOLS_PAYLOAD" set +e -curl -s -o "$TOOLS_BODY_FILE" \ +_receipted_curl closed gbrain-mcp-verify "$MCP_HOST" mcp-tools-list-probe "user-invoked mcp verify" "$TOOLS_PAYLOAD" \ + curl -s -o "$TOOLS_BODY_FILE" \ --max-time 10 \ -X POST \ -H 'Content-Type: application/json' \ -H 'Accept: application/json, text/event-stream' \ -H "Authorization: Bearer $GBRAIN_MCP_TOKEN" \ - -d "$TOOLS_REQ" \ "$URL" >/dev/null 2>&1 TOOLS_EXIT=$? set -e diff --git a/bin/gstack-gbrain-supabase-provision b/bin/gstack-gbrain-supabase-provision index 2bec5384c..8498b6a3c 100755 --- a/bin/gstack-gbrain-supabase-provision +++ b/bin/gstack-gbrain-supabase-provision @@ -69,6 +69,12 @@ set -euo pipefail SUPABASE_API_BASE="${SUPABASE_API_BASE:-https://api.supabase.com}" API_VERSION="v1" + +# Egress receipt helpers (_receipted_curl): receipt-before-send, fail-closed. +# The receipt hashes the request body only — the PAT (Authorization header) +# is never receipted or logged. +. "$(cd "$(dirname "$0")" && pwd)/gstack-egress-lib.sh" +SUPABASE_API_HOST="${SUPABASE_API_BASE#*://}"; SUPABASE_API_HOST="${SUPABASE_API_HOST%%/*}" DEFAULT_WAIT_TIMEOUT=180 POLL_INTERVAL=5 CURL_TIMEOUT=30 @@ -135,11 +141,24 @@ api_call() { -H "Content-Type: application/json" -H "User-Agent: gstack-gbrain-supabase-provision" ) + # Receipted fail-closed. The retry loop reuses $body_file across + # attempts, but the helper consumes its payload file — so each attempt + # hands it a fresh copy (hash still equals the exact wire bytes; the + # helper appends --data-binary @copy). Bodyless calls use --no-payload. + local payload_arg="--no-payload" if [ -n "$body_file" ]; then - curl_args+=(--data-binary "@$body_file") + payload_arg=$(mktemp) + cp "$body_file" "$payload_arg" fi - local status - if ! status=$(curl "${curl_args[@]}" "$url" 2>/dev/null); then + local status rc=0 + status=$(_receipted_curl closed supabase-provision "$SUPABASE_API_HOST" "provision-api-call ($method $apipath)" "user ran gstack-gbrain-supabase-provision" "$payload_arg" \ + curl "${curl_args[@]}" "$url") || rc=$? + if [ "$rc" -eq 3 ] && [ -z "$status" ]; then + # Egress receipt refused — the send never happened (the helper's + # problem/cause/fix message is already on stderr). Don't retry. + exit 8 + fi + if [ "$rc" -ne 0 ]; then # curl itself failed (network, timeout, etc.). Retry. if [ "$attempt" -ge "$max_attempts" ]; then die_net "network failure calling $method $apipath after $attempt attempts" diff --git a/bin/gstack-gbrain-sync.ts b/bin/gstack-gbrain-sync.ts index 786c50255..8aa703e4b 100644 --- a/bin/gstack-gbrain-sync.ts +++ b/bin/gstack-gbrain-sync.ts @@ -39,6 +39,7 @@ import "../lib/conductor-env-shim"; import { detectEngineTier, withErrorContext, canonicalizeRemote } from "../lib/gstack-memory-helpers"; import { ensureSourceRegistered, sourcePageCount, parseSourcesList, cycleCompleted, type CycleStatus } from "../lib/gbrain-sources"; import { detectAutopilot, decideSourceRemove, decideCodeSync } from "../lib/gbrain-guards"; +import { writeReceipt } from "../lib/egress-receipt"; import { localEngineStatus, type LocalEngineStatus } from "../lib/gbrain-local-status"; import { buildGbrainEnv, spawnGbrain, execGbrainJson, NEEDS_SHELL_ON_WINDOWS } from "../lib/gbrain-exec"; import { checkOwnedStagingDir } from "../lib/staging-guard"; @@ -67,7 +68,13 @@ interface CodeStageDetail { source_path?: string; page_count?: number | null; last_imported?: string; - status?: "ok" | "skipped" | "failed" | "refused-autopilot" | "refused-reclone"; + status?: + | "ok" + | "skipped" + | "failed" + | "refused-autopilot" + | "refused-reclone" + | "refused-egress-receipt"; } interface StageResult { @@ -902,6 +909,27 @@ async function runCodeImport(args: CliArgs): Promise { }; } + // Egress receipt BEFORE the code walk (fail-closed): the walk ships repo + // content to the user's gbrain DB, which may be a remote Postgres. The + // gbrain subprocess owns the wire bytes, so the receipt is content-free + // (destination + payload class only; sha256 null). + try { + writeReceipt({ + sink: "gbrain-sync", + host: "gbrain-db (user-configured DATABASE_URL)", + payloadClass: `repo-code-index source=${sourceId} (sent by gbrain subprocess)`, + bytes: 0, + sha256: null, + consent: "gbrain setup consent + per-repo policy chokepoint (repoPolicyTier)", + }); + } catch (err) { + return { + name: "code", ran: true, ok: false, duration_ms: Date.now() - t0, + summary: `EGRESS_RECEIPT_FAILED: ${(err as Error).message} — code sync refused`, + detail: { source_id: sourceId, source_path: root, status: "refused-egress-receipt" }, + }; + } + const walkResult = spawnGbrain(["sync", "--strategy", "code", "--source", sourceId], { stdio: args.quiet ? ["ignore", "ignore", "ignore"] : ["ignore", "inherit", "inherit"], timeout: codeTimeoutMs, diff --git a/bin/gstack-memory-ingest.ts b/bin/gstack-memory-ingest.ts index 653d4069a..532aee4a9 100644 --- a/bin/gstack-memory-ingest.ts +++ b/bin/gstack-memory-ingest.ts @@ -65,6 +65,7 @@ import { withErrorContext, } from "../lib/gstack-memory-helpers"; import { execGbrainText, spawnGbrainAsync } from "../lib/gbrain-exec"; +import { writeReceipt } from "../lib/egress-receipt"; import { checkOwnedStagingDir, STAGING_MARKER } from "../lib/staging-guard"; // ── Types ────────────────────────────────────────────────────────────────── @@ -1690,6 +1691,36 @@ async function ingestPass(args: CliArgs): Promise { // spawn, parent termination orphans the gbrain process (observed // during 2026-05-10 cold-run testing — gbrain kept running 15 min // after the orchestrator timed out). + // + // Egress receipt BEFORE the import (fail-closed): the gbrain DB may be a + // remote Postgres, so the ingest is a potential off-machine send. The + // gbrain subprocess owns the wire bytes (content-free receipt, sha256 + // null). The remote-http branch above stages locally only — its egress + // happens in gstack-brain-sync, which writes its own receipt at the push. + try { + writeReceipt({ + sink: "memory-ingest", + host: "gbrain-db (user-configured DATABASE_URL)", + payloadClass: `transcript-pages count=${staging.written} (sent by gbrain subprocess)`, + bytes: 0, + sha256: null, + consent: "gbrain setup consent (/setup-gbrain)", + }); + } catch (err) { + const msg = `EGRESS_RECEIPT_FAILED: ${(err as Error).message} — ingest refused`; + console.error(`[memory-ingest] ERR: ${msg}`); + failed += prep.prepared.length; + return { + written: 0, + skipped_secret: prep.skippedSecret, + skipped_dedup: prep.skippedDedup, + skipped_unattributed: prep.skippedUnattributed, + failed, + duration_ms: Date.now() - t0, + partial_pages: prep.partialPages, + system_error: msg, + }; + } const importResult = await runGbrainImport(stagingDir, resolveImportTimeoutMs()); const stdout = importResult.stdout || ""; diff --git a/bin/gstack-security-dashboard b/bin/gstack-security-dashboard index 9cf524e37..864fd8f94 100755 --- a/bin/gstack-security-dashboard +++ b/bin/gstack-security-dashboard @@ -17,6 +17,9 @@ set -uo pipefail GSTACK_DIR="${GSTACK_DIR:-$(cd "$(dirname "$0")/.." && pwd)}" +# Egress receipt helpers (_receipted_curl): fail-open for read-only stats. +. "$GSTACK_DIR/bin/gstack-egress-lib.sh" + # Source Supabase config if [ -z "${GSTACK_SUPABASE_URL:-}" ] && [ -f "$GSTACK_DIR/supabase/config.sh" ]; then . "$GSTACK_DIR/supabase/config.sh" @@ -46,10 +49,11 @@ fi # surface are indistinguishable from good news. TMPBODY="$(mktemp)" trap 'rm -f "$TMPBODY"' EXIT -HTTP_CODE="$(curl -s --max-time 15 -w '%{http_code}' -o "$TMPBODY" \ +SUPA_HOST="${SUPABASE_URL#*://}"; SUPA_HOST="${SUPA_HOST%%/*}" +HTTP_CODE="$(_receipted_curl open security-dashboard "$SUPA_HOST" community-pulse-fetch "user-invoked dashboard" --no-payload \ + curl -s --max-time 15 -w '%{http_code}' -o "$TMPBODY" \ "${SUPABASE_URL}/functions/v1/community-pulse" \ - -H "apikey: ${ANON_KEY}" \ - 2>/dev/null || true)" + -H "apikey: ${ANON_KEY}" || true)" # curl prints its own 000 before a non-zero exit — a `|| echo` here would # double it to "000000" in user-facing output. Normalize to the last 3 chars. HTTP_CODE="$(printf '%s' "$HTTP_CODE" | tr -d '[:space:]' | tail -c 3)" diff --git a/bin/gstack-session-update b/bin/gstack-session-update index 66bd44028..7e21e5df0 100755 --- a/bin/gstack-session-update +++ b/bin/gstack-session-update @@ -11,6 +11,10 @@ set +e GSTACK_DIR="${GSTACK_DIR:-$HOME/.claude/skills/gstack}" STATE_DIR="${GSTACK_STATE_DIR:-$HOME/.gstack}" + +# Egress receipt helpers (_receipted_git): fail-open — an update pull must +# never block a session over a receipt hiccup. +. "$(cd "$(dirname "$0")" && pwd)/gstack-egress-lib.sh" THROTTLE_FILE="$STATE_DIR/.last-session-update" LOCK_DIR="$STATE_DIR/.setup-lock" LOG_FILE="$STATE_DIR/analytics/session-update.log" @@ -76,7 +80,10 @@ fi # ── Pull latest ── OLD_HEAD=$(git -C "$GSTACK_DIR" rev-parse HEAD 2>/dev/null) - git -C "$GSTACK_DIR" pull --ff-only -q 2>/dev/null + UPDATE_URL=$(git -C "$GSTACK_DIR" remote get-url origin 2>/dev/null || echo "") + UPDATE_HOST="${UPDATE_URL#*://}"; UPDATE_HOST="${UPDATE_HOST#*@}"; UPDATE_HOST="${UPDATE_HOST%%[/:]*}" + GSTACK_HOME="$STATE_DIR" _receipted_git open session-update "${UPDATE_HOST:-unknown}" gstack-self-update-pull "auto_upgrade=true" \ + bash -c 'git -C "$1" pull --ff-only -q 2>/dev/null' _ "$GSTACK_DIR" PULL_EXIT=$? NEW_HEAD=$(git -C "$GSTACK_DIR" rev-parse HEAD 2>/dev/null) diff --git a/bin/gstack-telemetry-sync b/bin/gstack-telemetry-sync index 20f322043..172fed623 100755 --- a/bin/gstack-telemetry-sync +++ b/bin/gstack-telemetry-sync @@ -13,6 +13,9 @@ set -uo pipefail GSTACK_DIR="${GSTACK_DIR:-$(cd "$(dirname "$0")/.." && pwd)}" STATE_DIR="${GSTACK_STATE_DIR:-$HOME/.gstack}" + +# Egress receipt helpers (_receipted_curl): receipt-before-send, fail-closed. +. "$GSTACK_DIR/bin/gstack-egress-lib.sh" ANALYTICS_DIR="$STATE_DIR/analytics" JSONL_FILE="$ANALYTICS_DIR/skill-usage.jsonl" CURSOR_FILE="$ANALYTICS_DIR/.last-sync-line" @@ -114,12 +117,31 @@ RESP_FILE="$(mktemp "${TMPDIR:-/tmp}/gstack-sync-XXXXXX")" || { exit 0 } trap 'rm -f "$RESP_FILE"' EXIT -HTTP_CODE="$(curl -s -w '%{http_code}' --max-time 10 \ + +# Egress receipt BEFORE the send (fail-closed): the batch is written to a +# temp file, the receipt hashes those exact bytes, and curl sends the SAME +# file. On refusal nothing is sent and the cursor does not advance, so the +# batch stays buffered and the next run retries. The helper consumes the +# payload file. +PAYLOAD_FILE="$(mktemp "${TMPDIR:-/tmp}/gstack-sync-payload-XXXXXX")" || { + echo "gstack-telemetry-sync: mktemp failed — skipping this run" >&2 + exit 0 +} +printf '%s' "$BATCH" > "$PAYLOAD_FILE" +DEST_HOST="${SUPABASE_URL#*://}" +DEST_HOST="${DEST_HOST%%/*}" +HTTP_CODE="$(GSTACK_HOME="$STATE_DIR" _receipted_curl closed telemetry-sync "$DEST_HOST" telemetry-events "telemetry=$TIER" "$PAYLOAD_FILE" \ + curl -s -w '%{http_code}' --max-time 10 \ -X POST "${SUPABASE_URL}/functions/v1/telemetry-ingest" \ -H "Content-Type: application/json" \ -H "apikey: ${ANON_KEY}" \ - -o "$RESP_FILE" \ - -d "$BATCH" 2>/dev/null || echo "000")" + -o "$RESP_FILE" || echo "000")" + +# Best-effort response-status record for the receipt above (overrides the +# helper's generic exit-code outcome with the HTTP status). +if [ -n "${_GSTACK_EGRESS_LAST_RECEIPT:-}" ]; then + GSTACK_HOME="$STATE_DIR" "$GSTACK_DIR/bin/gstack-egress-receipt" outcome "$_GSTACK_EGRESS_LAST_RECEIPT" "$HTTP_CODE" 2>/dev/null || true +fi # ─── Update cursor on success (2xx) ───────────────────────── case "$HTTP_CODE" in diff --git a/bin/gstack-update-check b/bin/gstack-update-check index d0486cb4c..2d6d8af44 100755 --- a/bin/gstack-update-check +++ b/bin/gstack-update-check @@ -15,6 +15,29 @@ set -euo pipefail GSTACK_DIR="${GSTACK_DIR:-$(cd "$(dirname "$0")/.." && pwd)}" STATE_DIR="${GSTACK_STATE_DIR:-$HOME/.gstack}" + +# Egress receipt helpers (_receipted_curl / _receipted_git). Update checks +# are fail-OPEN: a receipt hiccup warns but never blocks the version check. +. "$GSTACK_DIR/bin/gstack-egress-lib.sh" + +# _receipted_version_fetch +# Receipted bodyless GET for a VERSION file (fail-open). Local reads +# (file://, or a URL with no network host) never leave the machine — they +# are not egress, so they get no receipt. +_receipted_version_fetch() { + local url="$1" host scheme + scheme="${url%%://*}" + host="${url#*://}"; host="${host%%/*}" + case "$scheme" in + http|https|git|ssh|ftp|ftps) + GSTACK_HOME="$STATE_DIR" _receipted_curl open update-check "$host" version-fetch "update_check!=false" --no-payload \ + curl -sf --max-time 5 "$url" || true + ;; + *) + curl -sf --max-time 5 "$url" 2>/dev/null || true + ;; + esac +} CACHE_FILE="$STATE_DIR/last-update-check" MARKER_FILE="$STATE_DIR/just-upgraded-from" SNOOZE_FILE="$STATE_DIR/update-snoozed" @@ -172,12 +195,19 @@ _SUPA_KEY="${GSTACK_SUPABASE_ANON_KEY:-}" _TEL_TIER="$("$GSTACK_DIR/bin/gstack-config" get telemetry 2>/dev/null || true)" if [ -n "$_SUPA_URL" ] && [ -n "$_SUPA_KEY" ] && [ "${_TEL_TIER:-off}" != "off" ]; then _OS="$(uname -s | tr '[:upper:]' '[:lower:]')" - curl -sf --max-time 5 \ - -X POST "${_SUPA_URL}/functions/v1/update-check" \ - -H "Content-Type: application/json" \ - -H "apikey: ${_SUPA_KEY}" \ - -d "{\"version\":\"$LOCAL\",\"os\":\"$_OS\"}" \ - >/dev/null 2>&1 & + # Receipted (fail-open) and fully backgrounded: the receipt write happens + # inside the background subshell, so it can never block this script's exit. + _PING_FILE="$(mktemp "${TMPDIR:-/tmp}/gstack-update-ping-XXXXXX" 2>/dev/null)" || _PING_FILE="" + if [ -n "$_PING_FILE" ]; then + printf '{"version":"%s","os":"%s"}' "$LOCAL" "$_OS" > "$_PING_FILE" + _SUPA_HOST="${_SUPA_URL#*://}"; _SUPA_HOST="${_SUPA_HOST%%/*}" + GSTACK_HOME="$STATE_DIR" _receipted_curl open update-check "$_SUPA_HOST" update-check-ping "telemetry=$_TEL_TIER" "$_PING_FILE" \ + curl -sf --max-time 5 \ + -X POST "${_SUPA_URL}/functions/v1/update-check" \ + -H "Content-Type: application/json" \ + -H "apikey: ${_SUPA_KEY}" \ + >/dev/null 2>&1 & + fi fi # Resolve VERSION via a SHA-pinned raw URL. GitHub's branch-raw CDN @@ -193,12 +223,14 @@ REMOTE="" if [ -z "${GSTACK_REMOTE_URL:-}" ]; then # Disable credential prompts and apply a 5-second low-speed timeout so a # flaky network or captive portal can't hang every skill preamble. - _LSR_LINE="$(GIT_TERMINAL_PROMPT=0 GIT_HTTP_LOW_SPEED_LIMIT=1000 GIT_HTTP_LOW_SPEED_TIME=5 \ - git ls-remote "$REMOTE_REPO" refs/heads/main 2>/dev/null || true)" + _REPO_HOST="${REMOTE_REPO#*://}"; _REPO_HOST="${_REPO_HOST#*@}"; _REPO_HOST="${_REPO_HOST%%[/:]*}" + _LSR_LINE="$(GSTACK_HOME="$STATE_DIR" _receipted_git open update-check "${_REPO_HOST:-unknown}" version-ls-remote "update_check!=false" \ + bash -c 'GIT_TERMINAL_PROMPT=0 GIT_HTTP_LOW_SPEED_LIMIT=1000 GIT_HTTP_LOW_SPEED_TIME=5 \ + git ls-remote "$1" refs/heads/main 2>/dev/null' _ "$REMOTE_REPO" || true)" _REMOTE_SHA="$(echo "$_LSR_LINE" | awk '{print $1}')" if echo "$_REMOTE_SHA" | grep -qE '^[0-9a-f]{40}$'; then _SHA_URL="https://raw.githubusercontent.com/garrytan/gstack/${_REMOTE_SHA}/VERSION" - REMOTE="$(curl -sf --max-time 5 "$_SHA_URL" 2>/dev/null || true)" + REMOTE="$(_receipted_version_fetch "$_SHA_URL")" fi fi @@ -206,7 +238,7 @@ fi # network, mirror without refs/heads/main) or when GSTACK_REMOTE_URL was # explicitly overridden. if [ -z "$REMOTE" ]; then - REMOTE="$(curl -sf --max-time 5 "$REMOTE_URL" 2>/dev/null || true)" + REMOTE="$(_receipted_version_fetch "$REMOTE_URL")" fi REMOTE="$(echo "$REMOTE" | tr -d '[:space:]')" diff --git a/browse/scripts/extension-id.ts b/browse/scripts/extension-id.ts new file mode 100755 index 000000000..cc9a7ba7f --- /dev/null +++ b/browse/scripts/extension-id.ts @@ -0,0 +1,45 @@ +#!/usr/bin/env bun +/** + * Derive the Chrome extension ID from the "key" field in + * extension/manifest.json. + * + * Chrome computes an extension's ID as the first 16 bytes of the SHA-256 + * hash of the DER-encoded public key, with each hex nibble mapped from + * 0-9a-f to a-p (the "mpdecimal" alphabet). Pinning the public key in the + * manifest pins the ID, which lets the browse server verify the Origin + * header on POST /extension-token against a single known extension + * identity (GSTACK_EXTENSION_ID in browse/src/server.ts). + * + * The private half of the keypair is intentionally NOT in the repo — the + * extension is loaded unpacked (or baked into Browser.app), so only the + * public key is needed to pin the ID. Regenerating the keypair changes + * the ID and requires updating both the manifest "key" and the + * GSTACK_EXTENSION_ID constant. + * + * Usage: bun browse/scripts/extension-id.ts [path/to/manifest.json] + */ + +import { createHash } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const manifestPath = process.argv[2] + ?? path.join(import.meta.dir, '../../extension/manifest.json'); + +const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf-8')); +if (typeof manifest.key !== 'string' || manifest.key.length === 0) { + console.error(`No "key" field in ${manifestPath}`); + process.exit(1); +} + +export function extensionIdFromPublicKey(publicKeyBase64: string): string { + const der = Buffer.from(publicKeyBase64, 'base64'); + const hex = createHash('sha256').update(der).digest('hex').slice(0, 32); + let id = ''; + for (const c of hex) { + id += String.fromCharCode('a'.charCodeAt(0) + parseInt(c, 16)); + } + return id; +} + +console.log(extensionIdFromPublicKey(manifest.key)); diff --git a/browse/src/browser-manager.ts b/browse/src/browser-manager.ts index f9f3317b5..4b378cc4f 100644 --- a/browse/src/browser-manager.ts +++ b/browse/src/browser-manager.ts @@ -1561,8 +1561,8 @@ export class BrowserManager { if (extensionPath) { launchArgs.push(`--disable-extensions-except=${extensionPath}`); launchArgs.push(`--load-extension=${extensionPath}`); - // Auth token is served via /health endpoint now (no file write needed). - // Extension reads token from /health on connect. + // Auth token is served via POST /extension-token (pinned-origin + // bootstrap, no file write needed). /health is liveness-only. console.log(`[browse] Handoff: loading extension from ${extensionPath}`); } else { console.log('[browse] Handoff: extension not found — headed mode without side panel'); diff --git a/browse/src/server.ts b/browse/src/server.ts index 301781acc..fdbe15e78 100644 --- a/browse/src/server.ts +++ b/browse/src/server.ts @@ -49,6 +49,7 @@ import { isProcessAlive } from './error-handling'; import { sanitizeBody, stripLoneSurrogateEscapes } from './sanitize'; import { startSocksBridge, testUpstream, type BridgeHandle } from './socks-bridge'; import { parseProxyConfig, toUpstreamConfig, ProxyConfigError } from './proxy-config'; +import { writeReceipt } from '../../lib/egress-receipt'; import { redactProxyUrl } from './proxy-redact'; import { shouldSpawnXvfb, pickFreeDisplay, spawnXvfb, xvfbInstallHint, type XvfbHandle } from './xvfb'; import { logTunnelDenial } from './tunnel-denial-log'; @@ -305,6 +306,17 @@ const TUNNEL_PATHS = new Set([ '/sidebar-chat', ]); +/** + * The gstack sidebar extension's pinned Chrome extension ID. Derived from + * the "key" field in extension/manifest.json (first 16 bytes of SHA-256 of + * the DER public key, hex nibbles mapped 0-9a-f → a-p). Reproduce with: + * bun browse/scripts/extension-id.ts + * POST /extension-token releases AUTH_TOKEN only to an Origin of exactly + * `chrome-extension://`. If the manifest keypair is ever rotated, + * this constant must be updated in the same commit. + */ +export const GSTACK_EXTENSION_ID = 'dgbkdbjebeiblbajiilljmhjdpmiglep'; + /** * Commands reachable via POST /command over the tunnel surface. A paired * remote agent can drive the browser (goto, click, text, etc.) but cannot @@ -1769,7 +1781,51 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle { ); } - // Health check — no auth required, does NOT reset idle timer + // ─── POST /extension-token — pinned-origin token bootstrap ────── + // + // The ONLY endpoint that hands out AUTH_TOKEN. GET /health used to + // carry the token (headed mode + any chrome-extension:// Origin), + // which meant ANY extension — or any localhost caller in headed + // mode — could read the root token. Now the token is released only + // to the one extension identity we ship: the Origin header must be + // exactly `chrome-extension://`, where the ID + // is pinned by the "key" field in extension/manifest.json (derive + // it with `bun browse/scripts/extension-id.ts`). Chrome sets Origin + // on cross-origin POSTs from extension contexts and web pages + // cannot forge a chrome-extension:// Origin. + // + // Local listener only: NEVER added to TUNNEL_PATHS, so the tunnel + // surface 404s it by default-deny. + if (url.pathname === '/extension-token' && req.method === 'POST') { + // Defense-in-depth alongside the 127.0.0.1 bind: a DNS-rebinding + // page can't present a localhost Host header. Host arrives as + // '127.0.0.1:34567', so parse out the hostname — never compare + // the raw header (which carries the port) against a literal. + let hostname: string | null = null; + try { + hostname = new URL(`http://${req.headers.get('host') ?? ''}`).hostname; + } catch (err) { + if (!(err instanceof TypeError)) throw err; // TypeError = malformed Host + } + const originOk = + req.headers.get('origin') === `chrome-extension://${GSTACK_EXTENSION_ID}`; + const hostOk = hostname === '127.0.0.1' || hostname === 'localhost'; + if (!originOk || !hostOk) { + // No detail in the body — don't teach a probing caller which + // check failed. + return new Response(JSON.stringify({ error: 'Forbidden' }), { + status: 403, headers: { 'Content-Type': 'application/json' }, + }); + } + return new Response(JSON.stringify({ token: authToken }), { + status: 200, headers: { 'Content-Type': 'application/json' }, + }); + } + + // Health check — no auth required, does NOT reset idle timer. + // NEVER carries a token in any mode: token bootstrap is + // POST /extension-token (pinned extension Origin) and shell auth + // is POST /pty-session. Liveness/status only. if (url.pathname === '/health') { const healthy = await browserManager.isHealthy(); return new Response(JSON.stringify({ @@ -1777,14 +1833,6 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle { mode: browserManager.getConnectionMode(), uptime: Math.floor((Date.now() - startTime) / 1000), tabs: browserManager.getTabCount(), - // Auth token for extension bootstrap. Safe: /health is localhost-only. - // Previously served unconditionally, but that leaks the token if the - // server is tunneled to the internet (ngrok, SSH tunnel). - // In headed mode the server is always local, so return token unconditionally - // (fixes Playwright Chromium extensions that don't send Origin header). - ...(browserManager.getConnectionMode() === 'headed' || - req.headers.get('origin')?.startsWith('chrome-extension://') - ? { token: authToken } : {}), // The chat queue is gone — Terminal pane is the sole sidebar // surface. Keep `chatEnabled: false` so any older extension // build still treats the chat input as disabled. @@ -2309,7 +2357,7 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle { // Dual-listener model: binds a SECOND Bun.serve listener on an // ephemeral 127.0.0.1 port dedicated to tunnel traffic, then points // ngrok.forward() at THAT port. The existing local listener (which - // serves /health+token, /cookie-picker, /inspector/*, welcome, etc.) + // serves /extension-token, /cookie-picker, /inspector/*, welcome, etc.) // is never exposed to ngrok. // // Hard fail if the tunnel listener bind fails — NEVER fall back to @@ -2375,6 +2423,19 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle { const forwardOpts: any = { addr: tunnelPort, authtoken }; if (domain) forwardOpts.domain = domain; + // Egress receipt BEFORE the tunnel session opens, fail-closed: a + // writeReceipt failure lands in this catch, which tears the tunnel + // listener back down and refuses the start. One receipt per session + // open; browse command behavior over the tunnel is unchanged. + writeReceipt({ + sink: 'browse-tunnel', + host: domain || 'connect.ngrok-agent.com', + payloadClass: 'tunnel-session-open (scoped-token browser-command surface)', + bytes: 0, + sha256: null, + consent: 'pair_agent=on', + }); + tunnelListener = await ngrok.forward(forwardOpts); tunnelUrl = tunnelListener.url(); tunnelServer = boundTunnel; @@ -2794,11 +2855,10 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle { // GET /memory — diagnostic snapshot (auth required, does NOT reset idle). // Same auth model as /activity/stream and /inspector/events: Bearer header - // OR view-only SSE-session cookie. Does NOT extend /health (which already - // leaks AUTH_TOKEN to any localhost caller in headed mode — see TODOS.md - // "Audit /health token distribution"); a separate endpoint with the - // standard SSE auth keeps the future /health fix from cascading into the - // sidebar footer poll. + // OR view-only SSE-session cookie. Does NOT extend /health (which is + // unauthenticated liveness-only — token bootstrap moved to the pinned + // POST /extension-token); a separate endpoint with the standard SSE auth + // keeps /health free of anything worth stealing. if (url.pathname === '/memory' && req.method === 'GET') { const cookieToken = extractSseCookie(req); if (!validateAuth(req) && !validateSseSessionToken(cookieToken)) { @@ -3084,6 +3144,18 @@ export async function start() { const forwardOpts: any = { addr: tunnelPort, authtoken }; if (domain) forwardOpts.domain = domain; + // Egress receipt BEFORE the tunnel session opens, fail-closed: a + // writeReceipt failure lands in this catch, which cleans up the + // listener and skips the tunnel (same as any other startup failure). + writeReceipt({ + sink: 'browse-tunnel', + host: domain || 'connect.ngrok-agent.com', + payloadClass: 'tunnel-session-open (scoped-token browser-command surface)', + bytes: 0, + sha256: null, + consent: 'pair_agent=on (BROWSE_TUNNEL=1)', + }); + tunnelListener = await ngrok.forward(forwardOpts); tunnelUrl = tunnelListener.url(); tunnelServer = boundTunnel; diff --git a/browse/test/dual-listener.test.ts b/browse/test/dual-listener.test.ts index 3ce04c1b7..9520fb13f 100644 --- a/browse/test/dual-listener.test.ts +++ b/browse/test/dual-listener.test.ts @@ -57,7 +57,7 @@ describe('Tunnel path allowlist', () => { const paths = extractSetContents(SERVER_SRC, 'TUNNEL_PATHS'); // These must never be on the tunnel surface const forbidden = [ - '/health', '/welcome', '/cookie-picker', + '/health', '/extension-token', '/welcome', '/cookie-picker', '/inspector', '/inspector/pick', '/inspector/events', '/inspector/style', '/tunnel/start', '/tunnel/stop', '/pair', '/token', '/refs', diff --git a/browse/test/extension-token.test.ts b/browse/test/extension-token.test.ts new file mode 100644 index 000000000..950c166bf --- /dev/null +++ b/browse/test/extension-token.test.ts @@ -0,0 +1,178 @@ +/** + * Live behavioral tests for the v1.62 token-bootstrap contract: + * + * - GET /health NEVER carries a token — not in headed mode, not for a + * chrome-extension:// Origin (the two pre-v1.62 carve-outs). IRON-RULE + * regression tests. + * - POST /extension-token releases the token ONLY to the pinned extension + * Origin (chrome-extension://GSTACK_EXTENSION_ID) with a loopback Host. + * - Host arrives with a port ('127.0.0.1:34567') and must be parsed to a + * hostname, not compared literally (amendment C9). 'localhost:34567' + * is accepted too. + * - The tunnel surface 404s /extension-token (not in TUNNEL_PATHS). + * + * Uses the buildFetchHandler factory (same pattern as server-factory.test.ts) + * so no listener/browser is needed. Real-HTTP coverage (Host header set by + * the network stack) lives in pair-agent-e2e.test.ts. + */ + +import { describe, test, expect, beforeEach } from 'bun:test'; +import * as crypto from 'crypto'; +import { + buildFetchHandler, + GSTACK_EXTENSION_ID, + type ServerConfig, +} from '../src/server'; +import { __resetRegistry } from '../src/token-registry'; +import { BrowserManager } from '../src/browser-manager'; +import { resolveConfig } from '../src/config'; + +const PINNED_ORIGIN = `chrome-extension://${GSTACK_EXTENSION_ID}`; + +function makeConfig(overrides: Partial = {}): ServerConfig { + const token = 'ext-token-test-' + crypto.randomBytes(16).toString('hex'); + return { + authToken: token, + browsePort: 34567, + idleTimeoutMs: 1_800_000, + config: resolveConfig(), + browserManager: new BrowserManager(), + startTime: Date.now(), + ...overrides, + }; +} + +function headedBrowserManager(): BrowserManager { + const bm = new BrowserManager(); + // connectionMode is private; force the headed value the old /health + // carve-out keyed on. + (bm as any).connectionMode = 'headed'; + return bm; +} + +function tokenRequest(headers: Record): Request { + // Direct handler invocation — no network stack to synthesize Host, so + // every test sets it explicitly (Bun.serve always delivers one). + return new Request('http://127.0.0.1:34567/extension-token', { + method: 'POST', + headers, + }); +} + +describe('GET /health never carries a token (IRON RULE)', () => { + beforeEach(() => __resetRegistry()); + + test('headed mode: no token field in the body', async () => { + const handle = buildFetchHandler(makeConfig({ browserManager: headedBrowserManager() })); + const resp = await handle.fetchLocal(new Request('http://127.0.0.1:34567/health'), null); + expect(resp.status).toBe(200); + const body = await resp.json() as any; + expect(body.token).toBeUndefined(); + expect(body.mode).toBe('headed'); + }); + + test('chrome-extension Origin (even the pinned one): no token field', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchLocal(new Request('http://127.0.0.1:34567/health', { + headers: { Origin: PINNED_ORIGIN }, + }), null); + expect(resp.status).toBe(200); + const body = await resp.json() as any; + expect(body.token).toBeUndefined(); + }); + + test('headed mode AND pinned chrome-extension Origin together: still no token', async () => { + const handle = buildFetchHandler(makeConfig({ browserManager: headedBrowserManager() })); + const resp = await handle.fetchLocal(new Request('http://127.0.0.1:34567/health', { + headers: { Origin: PINNED_ORIGIN }, + }), null); + const body = await resp.json() as any; + expect(body.token).toBeUndefined(); + }); +}); + +describe('POST /extension-token pinned-origin bootstrap', () => { + beforeEach(() => __resetRegistry()); + + test('pinned Origin + Host with port → 200 with the token', async () => { + const cfg = makeConfig(); + const handle = buildFetchHandler(cfg); + const resp = await handle.fetchLocal(tokenRequest({ + Origin: PINNED_ORIGIN, + Host: '127.0.0.1:34567', + }), null); + expect(resp.status).toBe(200); + const body = await resp.json() as any; + expect(body.token).toBe(cfg.authToken); + }); + + test("Host 'localhost:34567' is accepted too (C9 hostname parse)", async () => { + const cfg = makeConfig(); + const handle = buildFetchHandler(cfg); + const resp = await handle.fetchLocal(tokenRequest({ + Origin: PINNED_ORIGIN, + Host: 'localhost:34567', + }), null); + expect(resp.status).toBe(200); + const body = await resp.json() as any; + expect(body.token).toBe(cfg.authToken); + }); + + test('wrong extension Origin → 403, no token, no detail', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchLocal(tokenRequest({ + Origin: 'chrome-extension://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + Host: '127.0.0.1:34567', + }), null); + expect(resp.status).toBe(403); + const body = await resp.json() as any; + expect(body.token).toBeUndefined(); + // No detail about WHICH check failed + expect(JSON.stringify(body)).not.toContain('origin'); + expect(JSON.stringify(body)).not.toContain('host'); + }); + + test('missing Origin → 403', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchLocal(tokenRequest({ + Host: '127.0.0.1:34567', + }), null); + expect(resp.status).toBe(403); + }); + + test('web-page Origin → 403 (DNS-rebinding page cannot mint a token)', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchLocal(tokenRequest({ + Origin: 'http://evil.example.com', + Host: '127.0.0.1:34567', + }), null); + expect(resp.status).toBe(403); + }); + + test('non-loopback Host → 403 even with the pinned Origin', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchLocal(tokenRequest({ + Origin: PINNED_ORIGIN, + Host: 'evil.example.com:34567', + }), null); + expect(resp.status).toBe(403); + }); + + test('malformed Host → 403, not a crash', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchLocal(tokenRequest({ + Origin: PINNED_ORIGIN, + Host: ':::not a host:::', + }), null); + expect(resp.status).toBe(403); + }); + + test('tunnel surface 404s /extension-token (not in TUNNEL_PATHS)', async () => { + const handle = buildFetchHandler(makeConfig()); + const resp = await handle.fetchTunnel(tokenRequest({ + Origin: PINNED_ORIGIN, + Host: '127.0.0.1:34567', + }), null); + expect(resp.status).toBe(404); + }); +}); diff --git a/browse/test/pair-agent-e2e.test.ts b/browse/test/pair-agent-e2e.test.ts index 921ae4816..2f5c9169e 100644 --- a/browse/test/pair-agent-e2e.test.ts +++ b/browse/test/pair-agent-e2e.test.ts @@ -22,6 +22,7 @@ import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; +import { GSTACK_EXTENSION_ID } from '../src/server'; const ROOT = path.resolve(import.meta.dir, '../..'); const SERVER_ENTRY = path.join(ROOT, 'browse/src/server.ts'); @@ -94,22 +95,44 @@ describe('pair-agent flow end-to-end (HTTP only, no ngrok)', () => { if (daemon) killDaemon(daemon); }); - test('GET /health returns daemon status and includes token for chrome-extension origin', async () => { + test('GET /health returns daemon status and NEVER includes a token (even for chrome-extension origins)', async () => { const resp = await fetch(`${daemon.baseUrl}/health`, { - headers: { Origin: 'chrome-extension://test-extension-id' }, + headers: { Origin: `chrome-extension://${GSTACK_EXTENSION_ID}` }, }); expect(resp.status).toBe(200); const body = await resp.json() as any; expect(body.status).toBeDefined(); - // Extension bootstrap — local listener delivers the token - expect(body.token).toBe(daemon.token); + // v1.62: token bootstrap moved to POST /extension-token. /health is + // liveness-only in every mode. + expect(body.token).toBeUndefined(); }); - test('GET /health without chrome-extension origin does NOT include token', async () => { + test('GET /health without origin does NOT include token', async () => { const resp = await fetch(`${daemon.baseUrl}/health`); expect(resp.status).toBe(200); const body = await resp.json() as any; - // Headless mode + no chrome-extension origin → token withheld + expect(body.token).toBeUndefined(); + }); + + test('POST /extension-token with pinned Origin over real HTTP (Host carries port) returns the token', async () => { + // Real fetch → Host arrives as '127.0.0.1:'; the server must parse + // the hostname out rather than compare the raw header (amendment C9). + const resp = await fetch(`${daemon.baseUrl}/extension-token`, { + method: 'POST', + headers: { Origin: `chrome-extension://${GSTACK_EXTENSION_ID}` }, + }); + expect(resp.status).toBe(200); + const body = await resp.json() as any; + expect(body.token).toBe(daemon.token); + }); + + test('POST /extension-token with a non-pinned extension Origin returns 403 without the token', async () => { + const resp = await fetch(`${daemon.baseUrl}/extension-token`, { + method: 'POST', + headers: { Origin: 'chrome-extension://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }, + }); + expect(resp.status).toBe(403); + const body = await resp.json() as any; expect(body.token).toBeUndefined(); }); diff --git a/browse/test/server-auth.test.ts b/browse/test/server-auth.test.ts index 2469a121b..45d6d4c77 100644 --- a/browse/test/server-auth.test.ts +++ b/browse/test/server-auth.test.ts @@ -22,14 +22,29 @@ function sliceBetween(source: string, startMarker: string, endMarker: string): s } describe('Server auth security', () => { - // Test 1: /health serves token conditionally (headed mode or chrome extension only) - test('/health serves token only in headed mode or to chrome extensions', () => { + // Test 1 (IRON RULE, inverted in v1.62): /health NEVER serves a token in + // ANY mode. Both carve-outs (headed-mode disjunct + chrome-extension:// + // Origin disjunct) are gone. Token bootstrap moved to POST /extension-token + // with a pinned extension Origin. + test('/health never serves a token — no headed-mode or chrome-extension carve-out', () => { const healthBlock = sliceBetween(SERVER_SRC, "url.pathname === '/health'", "url.pathname === '/connect'"); - // v1.35.0.0: AUTH_TOKEN const was deleted; factory uses cfg-derived authToken. - // Token must be conditional, not unconditional - expect(healthBlock).toContain('token: authToken'); - expect(healthBlock).toContain('headed'); - expect(healthBlock).toContain('chrome-extension://'); + expect(healthBlock).not.toContain('token: authToken'); + expect(healthBlock).not.toContain("getConnectionMode() === 'headed'"); + expect(healthBlock).not.toContain("startsWith('chrome-extension://')"); + }); + + // Test 1a: the pinned-origin bootstrap endpoint exists and gates on both + // the exact extension Origin and a loopback Host. + test('POST /extension-token gates on pinned Origin and loopback Host', () => { + const tokenBlock = sliceBetween(SERVER_SRC, "url.pathname === '/extension-token'", "url.pathname === '/health'"); + expect(tokenBlock).toContain('GSTACK_EXTENSION_ID'); + expect(tokenBlock).toContain('token: authToken'); + // Host is parsed to a hostname (arrives as '127.0.0.1:34567'), never + // compared literally against the raw header. + expect(tokenBlock).toContain('.hostname'); + expect(tokenBlock).toContain("'127.0.0.1'"); + expect(tokenBlock).toContain("'localhost'"); + expect(tokenBlock).toContain('403'); }); // Test 1b: /health does not expose sensitive browsing state diff --git a/design/src/check.ts b/design/src/check.ts index 8f4aee9ae..992d606ca 100644 --- a/design/src/check.ts +++ b/design/src/check.ts @@ -5,6 +5,7 @@ import fs from "fs"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; export interface CheckResult { pass: boolean; @@ -22,7 +23,7 @@ export async function checkMockup(imagePath: string, brief: string): Promise controller.abort(), 60_000); try { - const response = await fetch("https://api.openai.com/v1/chat/completions", { + const response = await receiptedFetch("check-screenshot-request", "https://api.openai.com/v1/chat/completions", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/design-to-code.ts b/design/src/design-to-code.ts index 358a6b4e9..67ba95c2a 100644 --- a/design/src/design-to-code.ts +++ b/design/src/design-to-code.ts @@ -6,6 +6,7 @@ import fs from "fs"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; import { readDesignConstraints } from "./memory"; export interface DesignToCodeResult { @@ -37,7 +38,7 @@ export async function generateDesignToCodePrompt( ? `\n\nExisting DESIGN.md (use these as constraints):\n${designConstraints}` : ""; - const response = await fetch("https://api.openai.com/v1/chat/completions", { + const response = await receiptedFetch("design-to-code-request", "https://api.openai.com/v1/chat/completions", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/diff.ts b/design/src/diff.ts index 2d2e1ca19..b96dad652 100644 --- a/design/src/diff.ts +++ b/design/src/diff.ts @@ -6,6 +6,7 @@ import fs from "fs"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; export interface DiffResult { differences: { area: string; description: string; severity: string }[]; @@ -28,7 +29,7 @@ export async function diffMockups( const timeout = setTimeout(() => controller.abort(), 60_000); try { - const response = await fetch("https://api.openai.com/v1/chat/completions", { + const response = await receiptedFetch("diff-screenshots-request", "https://api.openai.com/v1/chat/completions", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/evolve.ts b/design/src/evolve.ts index 58e88ce16..3ecba39ad 100644 --- a/design/src/evolve.ts +++ b/design/src/evolve.ts @@ -8,6 +8,7 @@ import fs from "fs"; import path from "path"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; export interface EvolveOptions { screenshot: string; // Path to current site screenshot @@ -55,7 +56,7 @@ export async function evolve(options: EvolveOptions): Promise { const timeout = setTimeout(() => controller.abort(), 240_000); try { - const response = await fetch("https://api.openai.com/v1/responses", { + const response = await receiptedFetch("evolve-image-request", "https://api.openai.com/v1/responses", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, @@ -113,7 +114,7 @@ async function analyzeScreenshot(apiKey: string, imageBase64: string): Promise controller.abort(), 30_000); try { - const response = await fetch("https://api.openai.com/v1/chat/completions", { + const response = await receiptedFetch("evolve-screenshot-analysis-request", "https://api.openai.com/v1/chat/completions", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/generate.ts b/design/src/generate.ts index 3689aa710..e88f888aa 100644 --- a/design/src/generate.ts +++ b/design/src/generate.ts @@ -5,6 +5,7 @@ import fs from "fs"; import path from "path"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; import { parseBrief } from "./brief"; import { createSession, sessionPath } from "./session"; import { checkMockup } from "./check"; @@ -40,7 +41,7 @@ async function callImageGeneration( const timeout = setTimeout(() => controller.abort(), 240_000); try { - const response = await fetch("https://api.openai.com/v1/responses", { + const response = await receiptedFetch("generate-image-request", "https://api.openai.com/v1/responses", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/iterate.ts b/design/src/iterate.ts index 485944dd0..a2247d042 100644 --- a/design/src/iterate.ts +++ b/design/src/iterate.ts @@ -9,6 +9,7 @@ import fs from "fs"; import path from "path"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; import { readSession, updateSession } from "./session"; export interface IterateOptions { @@ -85,7 +86,7 @@ async function callWithThreading( const timeout = setTimeout(() => controller.abort(), 240_000); try { - const response = await fetch("https://api.openai.com/v1/responses", { + const response = await receiptedFetch("iterate-threaded-image-request", "https://api.openai.com/v1/responses", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, @@ -133,7 +134,7 @@ async function callFresh( const timeout = setTimeout(() => controller.abort(), 240_000); try { - const response = await fetch("https://api.openai.com/v1/responses", { + const response = await receiptedFetch("iterate-fresh-image-request", "https://api.openai.com/v1/responses", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/memory.ts b/design/src/memory.ts index 2fa7c5e8c..513e25d5b 100644 --- a/design/src/memory.ts +++ b/design/src/memory.ts @@ -14,6 +14,7 @@ import fs from "fs"; import path from "path"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; export interface ExtractedDesign { colors: { name: string; hex: string; usage: string }[]; @@ -34,7 +35,7 @@ export async function extractDesignLanguage(imagePath: string): Promise controller.abort(), 60_000); try { - const response = await fetch("https://api.openai.com/v1/chat/completions", { + const response = await receiptedFetch("memory-distill-request", "https://api.openai.com/v1/chat/completions", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, diff --git a/design/src/receipted-fetch.ts b/design/src/receipted-fetch.ts new file mode 100644 index 000000000..39c4fa036 --- /dev/null +++ b/design/src/receipted-fetch.ts @@ -0,0 +1,57 @@ +/** + * receipted-fetch — egress-receipted wrapper for the design binary's OpenAI + * calls (sink 'design-openai'). + * + * Writes a content-free receipt BEFORE the send: sha256 of the JSON body + * plus a byte count — the hash only, never the body itself. FAIL-OPEN: a + * receipt hiccup warns on stderr and the call proceeds. User-facing image + * generation must not die because an audit log could not be written (the + * ledger records ATTEMPTED egress for auditing; it is not a send gate here). + * + * Streams pass through untouched: the response is returned as-is, and a + * non-string request body (e.g. a ReadableStream) is receipted as + * sha256:null rather than being consumed to hash it. + */ + +import { sha256Hex, writeReceipt } from "../../lib/egress-receipt"; + +export type FetchLike = typeof globalThis.fetch; + +/** + * Drop-in fetch replacement for api.openai.com calls. + * + * @param payloadClass content-free description of what is being sent + * (e.g. 'generate-image-request', 'check-screenshot-request') + * @param fetchImpl injectable fetch for tests / callers with their own + * fetch (variants.ts passes its stubbed fetchFn through) + */ +export async function receiptedFetch( + payloadClass: string, + url: string, + init?: RequestInit, + fetchImpl: FetchLike = globalThis.fetch, +): Promise { + try { + const body = init?.body; + let bytes = 0; + let sha256: string | null = null; + if (typeof body === "string") { + bytes = Buffer.byteLength(body); + sha256 = sha256Hex(body); + } + writeReceipt({ + sink: "design-openai", + host: new URL(url).host, + payloadClass, + bytes, + sha256, + consent: "user ran design command (OPENAI_API_KEY configured)", + }); + } catch (err) { + process.stderr.write( + `[design] egress receipt could not be written (${(err as Error).message}) — proceeding (fail-open). ` + + `gstack records what it ATTEMPTS to send off-machine; see gstack-egress.\n`, + ); + } + return fetchImpl(url, init); +} diff --git a/design/src/variants.ts b/design/src/variants.ts index 15be75e5c..ca1c37bfc 100644 --- a/design/src/variants.ts +++ b/design/src/variants.ts @@ -7,6 +7,7 @@ import fs from "fs"; import path from "path"; import { requireApiKey } from "./auth"; +import { receiptedFetch } from "./receipted-fetch"; import { parseBrief } from "./brief"; import { normalizeIntFlag } from "./flag-utils"; @@ -67,7 +68,7 @@ export async function generateVariant( const timeout = setTimeout(() => controller.abort(), 240_000); try { - const response = await fetchFn("https://api.openai.com/v1/responses", { + const response = await receiptedFetch("variants-image-request", "https://api.openai.com/v1/responses", { method: "POST", headers: { "Authorization": `Bearer ${apiKey}`, @@ -79,7 +80,7 @@ export async function generateVariant( tools: [{ type: "image_generation", model: "gpt-image-2", size, quality }], }), signal: controller.signal, - }); + }, fetchFn); clearTimeout(timeout); diff --git a/design/test/receipted-fetch.test.ts b/design/test/receipted-fetch.test.ts new file mode 100644 index 000000000..177f760ce --- /dev/null +++ b/design/test/receipted-fetch.test.ts @@ -0,0 +1,136 @@ +/** + * receipted-fetch — egress receipts for the design binary's OpenAI calls. + * + * Pins the sink contract (fail-OPEN polarity, amendment T3/C8): + * - receipt is written BEFORE the send (ordering observable via the + * ledger's existence at fetch time) + * - the sha256 recorded is the hash of the JSON body; the body itself is + * never stored + * - streaming response bodies pass through the wrapper intact + * - an unwritable ledger warns on stderr and the call still proceeds + */ + +import { describe, test, expect, beforeEach, afterEach } from "bun:test"; +import fs from "fs"; +import os from "os"; +import path from "path"; +import { receiptedFetch } from "../src/receipted-fetch"; +import { egressLedgerPath, listReceipts, sha256Hex } from "../../lib/egress-receipt"; + +let home: string; +let savedHome: string | undefined; + +beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), "design-receipt-")); + savedHome = process.env.GSTACK_HOME; + process.env.GSTACK_HOME = home; +}); + +afterEach(() => { + if (savedHome === undefined) delete process.env.GSTACK_HOME; + else process.env.GSTACK_HOME = savedHome; + try { fs.chmodSync(path.join(home, "security"), 0o700); } catch {} + fs.rmSync(home, { recursive: true, force: true }); +}); + +describe("receiptedFetch", () => { + test("writes the receipt BEFORE the send; sha256 is the hash of the JSON body", async () => { + const body = JSON.stringify({ model: "gpt-4o", input: "a prompt" }); + let receiptsAtFetchTime = -1; + const stub = (async (_url: any, init?: any) => { + // Receipt-before-send: by the time fetch runs, the receipt exists. + receiptsAtFetchTime = listReceipts(home).length; + expect(init.body).toBe(body); // body passes through untouched + return new Response("{}", { status: 200 }); + }) as typeof globalThis.fetch; + + const response = await receiptedFetch("generate-image-request", "https://api.openai.com/v1/responses", { + method: "POST", + body, + }, stub); + + expect(response.status).toBe(200); + expect(receiptsAtFetchTime).toBe(1); + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].sink).toBe("design-openai"); + expect(receipts[0].host).toBe("api.openai.com"); + expect(receipts[0].payload_class).toBe("generate-image-request"); + expect(receipts[0].sha256).toBe(sha256Hex(body)); + expect(receipts[0].bytes).toBe(Buffer.byteLength(body)); + // Hash only — the ledger never contains the body text. + const raw = fs.readFileSync(egressLedgerPath(home), "utf-8"); + expect(raw).not.toContain("a prompt"); + }); + + test("streaming response body arrives intact through the wrapper", async () => { + const chunks = ["data: one\n", "data: two\n", "data: [DONE]\n"]; + const stream = new ReadableStream({ + start(controller) { + for (const c of chunks) controller.enqueue(new TextEncoder().encode(c)); + controller.close(); + }, + }); + const stub = (async () => new Response(stream, { status: 200 })) as typeof globalThis.fetch; + + const response = await receiptedFetch("evolve-image-request", "https://api.openai.com/v1/responses", { + method: "POST", + body: JSON.stringify({ stream: true }), + }, stub); + + expect(response.body).toBeInstanceOf(ReadableStream); + expect(await response.text()).toBe(chunks.join("")); + }); + + test("non-string request body (ReadableStream) is receipted as sha256:null, not consumed", async () => { + const requestStream = new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode("streamed-bytes")); + controller.close(); + }, + }); + let receivedBody: any = null; + const stub = (async (_url: any, init?: any) => { + receivedBody = init.body; + return new Response("{}", { status: 200 }); + }) as typeof globalThis.fetch; + + await receiptedFetch("stream-upload", "https://api.openai.com/v1/responses", { + method: "POST", + body: requestStream, + }, stub); + + expect(receivedBody).toBe(requestStream); // same stream object, untouched + const receipts = listReceipts(home); + expect(receipts[0].sha256).toBeNull(); + // The stream is still readable by the consumer (was not drained to hash). + expect(await new Response(receivedBody).text()).toBe("streamed-bytes"); + }); + + test("fail-open: unwritable ledger warns on stderr and the call proceeds", async () => { + if (process.platform === "win32" || process.getuid?.() === 0) return; + fs.mkdirSync(path.join(home, "security"), { recursive: true, mode: 0o500 }); + let fetched = false; + const stub = (async () => { fetched = true; return new Response("{}", { status: 200 }); }) as typeof globalThis.fetch; + + const captured: string[] = []; + const originalWrite = process.stderr.write.bind(process.stderr); + (process.stderr as any).write = (chunk: string) => { captured.push(String(chunk)); return true; }; + let response: Response; + try { + response = await receiptedFetch("check-screenshot-request", "https://api.openai.com/v1/chat/completions", { + method: "POST", + body: "{}", + }, stub); + } finally { + (process.stderr as any).write = originalWrite; + } + + expect(fetched).toBe(true); // the call proceeded + expect(response.status).toBe(200); + const warning = captured.join(""); + expect(warning).toContain("egress receipt could not be written"); + expect(warning).toContain("fail-open"); + expect(warning).toContain("gstack-egress"); + }); +}); diff --git a/design/test/variants-retry-after.test.ts b/design/test/variants-retry-after.test.ts index 2060791d5..8d84557b7 100644 --- a/design/test/variants-retry-after.test.ts +++ b/design/test/variants-retry-after.test.ts @@ -44,13 +44,19 @@ function makeStubFetch( describe("generateVariant Retry-After handling", () => { let tmpDir: string; let outputPath: string; + let savedHome: string | undefined; beforeEach(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "variants-retry-after-")); outputPath = path.join(tmpDir, "variant.png"); + // The fetch path now writes egress receipts — keep them in the temp home. + savedHome = process.env.GSTACK_HOME; + process.env.GSTACK_HOME = tmpDir; }); afterEach(() => { + if (savedHome === undefined) delete process.env.GSTACK_HOME; + else process.env.GSTACK_HOME = savedHome; fs.rmSync(tmpDir, { recursive: true, force: true }); }); diff --git a/docs/REMOTE_BROWSER_ACCESS.md b/docs/REMOTE_BROWSER_ACCESS.md index 88dc30bb2..373e10690 100644 --- a/docs/REMOTE_BROWSER_ACCESS.md +++ b/docs/REMOTE_BROWSER_ACCESS.md @@ -56,7 +56,7 @@ All command endpoints require a Bearer token: Authorization: Bearer gsk_sess_... ``` -`/connect` is unauthenticated (rate-limited) — it's how a remote agent exchanges a setup key for a scoped session token. `/health` is unauthenticated on the local listener (bootstrap) but does NOT exist on the tunnel listener (404). +`/connect` is unauthenticated (rate-limited) — it's how a remote agent exchanges a setup key for a scoped session token. `/health` is unauthenticated on the local listener (liveness/status only — never a token) but does NOT exist on the tunnel listener (404). Extension token bootstrap is `POST /extension-token` on the local listener, gated by the pinned `chrome-extension://` Origin; it is not on the tunnel surface either. SSE endpoints (`/activity/stream`, `/inspector/events`) accept either a Bearer token or the HttpOnly `gstack_sse` cookie (minted via `POST /sse-session`, 30-minute TTL, stream-scope only — cannot be used against `/command`). As of v1.6.0.0 the `?token=` query-string auth is no longer accepted. @@ -80,6 +80,9 @@ Response: (plain text result of the command) #### GET /health Server status. No auth required. Returns status, tabs, mode, uptime. +Never carries a token — extension token bootstrap is `POST /extension-token` +(local listener only, validates the pinned `chrome-extension://` Origin and a +loopback Host; 403 otherwise). Not reachable over the tunnel (404). ### Commands @@ -176,6 +179,7 @@ Each agent owns the tabs it creates. Rules: - **Path traversal guarded** on `/welcome` — `GSTACK_SLUG` must match `^[a-z0-9_-]+$` or falls back to the built-in template. - **SSRF guards** on `goto`, `download`, and scrape paths — validates URL target against a localhost/private-range blocklist. - **Tunnel surface denial logging.** Every rejection on the tunnel listener (`path_not_on_tunnel`, `root_token_on_tunnel`, `missing_scoped_token`, `disallowed_command:*`) is appended to `~/.gstack/security/attempts.jsonl` with timestamp, source IP, path, method. Rate-capped at 60 writes/min. +- **Egress receipt on tunnel start (v1.63+).** Every tunnel session open writes a hash-chained receipt (sink `browse-tunnel`) to `~/.gstack/security/egress.jsonl` BEFORE ngrok forwards anything. Fail-closed: if the receipt can't be written, the tunnel refuses to start. Audit with `bin/gstack-egress list` / `bin/gstack-egress verify`. - All agent activity is logged with attribution (clientId). **Known non-goal (tracked as #1136):** on Windows, the cookie-import-browser path launches Chrome with `--remote-debugging-port=`. With App-Bound Encryption v20, a same-user local process can connect to that port and exfiltrate decrypted v20 cookies — an elevation path relative to reading the SQLite DB directly. Fix direction is `--remote-debugging-pipe` instead of TCP. diff --git a/docs/designs/SIDEBAR_MESSAGE_FLOW.md b/docs/designs/SIDEBAR_MESSAGE_FLOW.md index 4c8fc8c7f..ae46df16f 100644 --- a/docs/designs/SIDEBAR_MESSAGE_FLOW.md +++ b/docs/designs/SIDEBAR_MESSAGE_FLOW.md @@ -62,13 +62,18 @@ T+500ms terminal-agent.ts boots └── Probes claude → writes claude-available.json T+1-3s Extension loads, sidebar opens + ├── background.js: GET /health (liveness only — no token) then + │ POST /extension-token → AUTH_TOKEN. The server releases the + │ token only to Origin chrome-extension://; the + │ manifest "key" pins the ID (browse/scripts/extension-id.ts) ├── sidepanel-terminal.js: setState(IDLE), shows "Starting Claude Code..." └── tryAutoConnect() polls until window.gstackServerPort + token are set T+ready tryAutoConnect calls connect() ├── POST /pty-session (Authorization: Bearer AUTH_TOKEN) - │ └── server mints session token, posts /internal/grant to agent - │ └── responds with {terminalPort, ptySessionToken} + │ └── server mints attach token, posts /internal/grant to agent + │ └── responds with {terminalPort, sessionId, attachToken, + │ leaseExpiresAt} ├── GET /claude-available (preflight) ├── new WebSocket(`ws://127.0.0.1:/ws`, │ [`gstack-pty.`]) @@ -105,7 +110,7 @@ The protocol-token path is what the browser actually uses. | Token | Lives in | Used for | Lifetime | |-------|----------|----------|----------| -| `AUTH_TOKEN` | `/browse.json`; in-memory in server.ts | `/pty-session` POST (mint cookie + token) | server lifetime | +| `AUTH_TOKEN` | `/browse.json`; in-memory in server.ts; extension memory via pinned-origin `POST /extension-token` (never `GET /health`) | `/pty-session` POST (mint cookie + token) | server lifetime | | `gstack-pty.<...>` (Sec-WebSocket-Protocol) | Browser memory only; agent `validTokens` Set | `/ws` upgrade auth | 30 min, auto-revoked on WS close | | `INTERNAL_TOKEN` | `/terminal-internal-token`; in agent memory | server → agent loopback `/internal/grant` | agent lifetime | diff --git a/docs/gbrain-sync-errors.md b/docs/gbrain-sync-errors.md index 7ab50cdfe..1f5f7bed6 100644 --- a/docs/gbrain-sync-errors.md +++ b/docs/gbrain-sync-errors.md @@ -9,8 +9,9 @@ the command output. ## `BRAIN_SYNC: brain repo detected: ` -**Problem.** You're on a machine that has `~/.gstack-brain-remote.txt` (copied -from another machine) but no local git repo at `~/.gstack/.git`. +**Problem.** You're on a machine that has `~/.gstack-artifacts-remote.txt` +(or the legacy `~/.gstack-brain-remote.txt`, copied from another machine) but +no local git repo at `~/.gstack/.git`. **Cause.** You've set up GBrain sync elsewhere and your gstack hasn't been restored on this machine yet. @@ -92,23 +93,48 @@ your local commit still exists — the next skill run will retry the push. --- -## `gstack-brain-init: ~/.gstack/.git is already a git repo pointing at ` +## `gstack: brain-sync push NOT sent — the egress receipt could not be written` -**Problem.** You tried to init with a remote URL that doesn't match the -existing one. +**Problem.** The push was refused before anything left your machine. Every +brain-sync push writes a tamper-evident receipt to the egress ledger +(`~/.gstack/security/egress.jsonl`) before sending, fail-closed. The +receipt could not be written, so nothing was sent, no local commit was +made, and the queue is preserved — the next run retries the whole drain. +`gstack-brain-sync --status` shows `EGRESS_RECEIPT_FAILED` as the failure +detail. -**Cause.** You already ran `gstack-brain-init` with a different remote. +**Cause.** `~/.gstack/security/` is not writable (the receipt writer creates +it when missing, so absence alone is not the cause), the disk is full, or +`GSTACK_HOME` points at a read-only location. -**Fix.** Either: - -- Use the existing remote: run `gstack-brain-init` without `--remote`, or - with the matching URL. -- Switch remotes: `gstack-brain-uninstall` first, then re-init with the new - URL. This does not delete your data. +**Fix.** +```bash +mkdir -p ~/.gstack/security && chmod -R u+w ~/.gstack/security +``` +Then run any skill (or `gstack-brain-sync --once`) to retry. Inspect the +ledger with `gstack-egress list`; verify its hash chain with +`gstack-egress verify`. --- -## `Remote not reachable: ` +## `gstack-artifacts-init: ~/.gstack/ is already a git repo pointing at: ` + +**Problem.** You tried to init with a remote URL that doesn't match the +existing one. The command refuses to overwrite. + +**Cause.** You already ran `gstack-artifacts-init` with a different remote. + +**Fix.** Either: + +- Use the existing remote: run `gstack-artifacts-init` without `--remote`, or + with the matching URL. +- Switch remotes: `git -C ~/.gstack remote set-url origin ` (the + command's own suggestion), or `gstack-brain-uninstall` first, then re-init + with the new URL. Neither deletes your data. + +--- + +## `Remote not reachable via SSH: ` **Problem.** Init couldn't reach the git remote to verify connectivity. @@ -126,7 +152,7 @@ If that fails, check: --- -## `gstack-brain-init: failed to create or find ''` +## `Failed to create or find ''. Try --remote .` **Problem.** Auto-repo-creation via `gh repo create` failed and the repo isn't discoverable via `gh repo view` either. @@ -141,7 +167,7 @@ gh auth status If unauth'd, run `gh auth login`. If the repo name collides, pass a different name: ```bash -gstack-brain-init --remote git@github.com:YOURUSER/custom-name.git +gstack-artifacts-init --remote git@github.com:YOURUSER/custom-name.git ``` --- @@ -168,7 +194,7 @@ gstack session, or (b) a previous failed restore left partial state. **Fix (three options).** 1. **If this machine's state should become the new truth**: run - `gstack-brain-init` instead of restore — this creates a brand-new brain + `gstack-artifacts-init` instead of restore — this creates a brand-new brain repo from this machine's state. 2. **If you want to adopt the remote and discard this machine's state**: @@ -189,7 +215,7 @@ and `.gitattributes`. **Cause.** You pointed restore at a random git repo, or someone deleted the canonical config files from the brain repo. -**Fix.** Verify the URL. If it's correct, run `gstack-brain-init --remote +**Fix.** Verify the URL. If it's correct, run `gstack-artifacts-init --remote ` to re-seed the canonical config. --- diff --git a/docs/gbrain-sync.md b/docs/gbrain-sync.md index 62a12b56a..e6c0466e4 100644 --- a/docs/gbrain-sync.md +++ b/docs/gbrain-sync.md @@ -18,7 +18,7 @@ GBrain. By design, these stay local even when sync is on: - Credentials: `.auth.json`, `auth-token.json`, `sidebar-sessions/`, - `security/device-salt`, consumer tokens in `config.yaml` + `security/device-salt` - Machine-specific state: Chromium profiles, ONNX model weights, caches, eval-cache, CDP-profile, one-time prompt markers (`.welcome-seen`, `.telemetry-prompted`, `.vendoring-warned-*`, etc.) @@ -31,25 +31,25 @@ it; you can append your own entries below the marker line. ## First-run setup (30–90 seconds) ```bash -gstack-brain-init +gstack-artifacts-init ``` The command: 1. Turns `~/.gstack/` into a git repo. 2. Asks for a remote URL (default: `gh repo create --private - gstack-brain-$USER`). Any git remote works — GitHub, GitLab, Gitea, + gstack-artifacts-$USER`). Any git remote works — GitHub, GitLab, Gitea, self-hosted. 3. Pushes an initial commit with just the config. -4. Writes `~/.gstack-brain-remote.txt` (URL-only, no secrets — +4. Writes `~/.gstack-artifacts-remote.txt` (URL-only, no secrets — safe to copy to another machine). -5. Wires the gstack-brain repo into your local gbrain as a federated - source (via `gbrain sources add` + `git worktree`) so `gbrain search` - can index your synced learnings, plans, and designs. Implementation - lives in `bin/gstack-gbrain-source-wireup`. The old - `gstack-brain-reader add --ingest-url ...` HTTP path was removed in - v1.15.1.0 — it depended on a `/ingest-repo` endpoint gbrain never - shipped. +5. Prints the `gbrain sources add` hookup command for the brain host + (never auto-executed — run it yourself, or on your own machine + `bin/gstack-gbrain-source-wireup` does the same wiring) so + `gbrain search` can index your synced learnings, plans, and designs. + The old `gstack-brain-reader add --ingest-url ...` HTTP path was + removed in v1.15.1.0 — it depended on a `/ingest-repo` endpoint gbrain + never shipped. After init, the **next skill you run** will ask you ONE question about privacy mode: @@ -65,14 +65,15 @@ Your answer is persisted. You won't be asked again. ## Cross-machine workflow -On machine A: run `gstack-brain-init` once. That's it — every skill +On machine A: run `gstack-artifacts-init` once. That's it — every skill invocation now drains the sync queue at its start and end boundaries (~200–800 ms network pause per skill). On machine B: -1. Copy `~/.gstack-brain-remote.txt` from machine A to machine B - (password manager, dotfile repo, USB stick — your call). +1. Copy `~/.gstack-artifacts-remote.txt` from machine A to machine B + (password manager, dotfile repo, USB stick — your call; the legacy + `~/.gstack-brain-remote.txt` name is still recognized). 2. Run any gstack skill. The preamble sees the URL file and prints: ``` BRAIN_SYNC: brain repo detected: @@ -80,9 +81,7 @@ On machine B: ``` 3. Run `gstack-brain-restore`. That clones the repo, rehydrates your learnings/plans/retros, and re-registers the git merge drivers. -4. Re-enter consumer tokens (they're machine-local and NOT synced — - `gstack-config set gbrain_token `). -5. Next skill: your yesterday-on-machine-A learning surfaces. That's the +4. Next skill: your yesterday-on-machine-A learning surfaces. That's the magical moment. ## Status, health, and queue depth @@ -141,6 +140,12 @@ To remediate: There's a defense-in-depth hook at `~/.gstack/.git/hooks/pre-commit` that runs the same scan if you manually `git commit` against the repo. +Separately (v1.63.0.0+), every push writes a tamper-evident receipt to the +egress ledger (`~/.gstack/security/egress.jsonl`) *before* anything is +sent, fail-closed: if the receipt can't be written, the push is refused +and the queue is preserved. Inspect the ledger with `gstack-egress list` +and verify its hash chain with `gstack-egress verify`. + ## Two-machine conflicts If you write on machine A and machine B the same day, both will push @@ -176,7 +181,7 @@ This: Add `--delete-remote` to also delete the private GitHub repo (GitHub only, uses `gh repo delete`). -Re-init anytime with `gstack-brain-init`. +Re-init anytime with `gstack-artifacts-init`. ## Troubleshooting @@ -185,8 +190,8 @@ error message gstack-brain may print, with problem / cause / fix for each. ## Under the hood -For the architectural decisions behind this feature (allowlist vs -denylist, daemon vs preamble-boundary sync, JSONL merge driver, privacy -stop-gate), see the -[approved plan](../system-instruction-you-are-working-jaunty-kahn.md) in -the gstack plans directory. +The architectural decisions behind this feature: allowlist over denylist +(unknown files stay local by default), preamble-boundary sync over a daemon +(no background process to babysit), a JSONL merge driver so concurrent +machines union their queues instead of conflicting, and a privacy stop-gate +that asks once before anything syncs. diff --git a/extension/background.js b/extension/background.js index d0abe6328..249bd9f05 100644 --- a/extension/background.js +++ b/extension/background.js @@ -32,22 +32,34 @@ function getBaseUrl() { // ─── Auth Token Bootstrap ───────────────────────────────────── +// Token bootstrap: POST /extension-token. The server validates our Origin +// (chrome-extension:// — the manifest "key" pins the ID) before +// releasing the token. GET /health is liveness/status only and never +// carries a token. Returns true on success, false on failure; a 403 means +// the server doesn't trust this extension identity — treat as disconnected +// rather than retrying forever with a stale token. async function loadAuthToken() { - if (authToken) return; - // Get token from browse server /health endpoint (localhost-only, safe). - // Previously read from .auth.json in extension dir, but that breaks - // read-only .app bundles and codesigning. const base = getBaseUrl(); - if (!base) return; + if (!base) return false; try { - const resp = await fetch(`${base}/health`, { signal: AbortSignal.timeout(3000) }); + const resp = await fetch(`${base}/extension-token`, { + method: 'POST', + signal: AbortSignal.timeout(3000), + }); + if (resp.status === 403) { + console.error('[gstack bg] /extension-token 403 — extension identity not trusted by server'); + authToken = null; + setDisconnected(); + return false; + } if (resp.ok) { const data = await resp.json(); - if (data.token) authToken = data.token; + if (data.token) { authToken = data.token; return true; } } } catch (err) { console.error('[gstack bg] Failed to load auth token:', err.message); } + return false; } // ─── Health Polling ──────────────────────────────────────────── @@ -59,17 +71,16 @@ async function checkHealth() { return; } - // Retry loading auth token if we don't have one yet - if (!authToken) await loadAuthToken(); - try { const resp = await fetch(`${base}/health`, { signal: AbortSignal.timeout(3000) }); if (!resp.ok) { setDisconnected(); return; } const data = await resp.json(); if (data.status === 'healthy') { - // Always refresh auth token from /health — the server generates a new - // token on each restart, so the old one becomes stale. - if (data.token) authToken = data.token; + // Always refresh the auth token — the server generates a new token + // on each restart, so the old one becomes stale. loadAuthToken() + // already flips to disconnected on a 403. + const gotToken = await loadAuthToken(); + if (!gotToken && !authToken) return; // Forward chatEnabled so sidepanel can show/hide chat tab setConnected({ ...data, chatEnabled: !!data.chatEnabled }); } else { @@ -577,27 +588,50 @@ chrome.tabs.onUpdated.addListener((_id, changeInfo) => { } }); +// ─── identity-pin migration notice (v1.63) ──────────────────── +// +// The manifest "key" added in v1.63 pins the extension ID, which changes +// the ID for existing installs — chrome.storage.local is keyed by +// extension ID, so panel-local state (saved port, snoozes) resets once. +// Explain that in-product, one time. The flag name is version-free so a +// release re-slot never orphans an already-set flag. +async function announceIdentityPinOnce() { + try { + const data = await chrome.storage.local.get('gstack_id_pin_migrated'); + if (data.gstack_id_pin_migrated) return; + console.log('[gstack] gstack sidebar: extension identity pinned in v1.63 — panel state reset once.'); + chrome.runtime.sendMessage({ + type: 'gstack-migration-notice', + message: 'gstack sidebar: extension identity pinned in v1.63 — panel state reset once.', + }).catch(() => { + // Expected: panel not open. The console line above still lands. + }); + await chrome.storage.local.set({ gstack_id_pin_migrated: true }); + } catch (err) { + console.debug('[gstack] identity-pin notice failed (non-fatal):', err.message); + } +} + // ─── Startup ──────────────────────────────────────────────────── // Fast-retry health check on startup. The server may not be listening yet // (Chromium launches before Bun.serve starts). Retry every 1s for the // first 15 seconds, then switch to 10s polling. -loadAuthToken().then(() => { - loadPort().then(() => { - let startupAttempts = 0; - const startupCheck = setInterval(async () => { - startupAttempts++; - await checkHealth(); - if (isConnected || startupAttempts >= 15) { - clearInterval(startupCheck); - // Switch to slow polling now that we're connected (or gave up) - if (!healthInterval) { - healthInterval = setInterval(checkHealth, 10000); - } - if (!isConnected) { - console.log('[gstack] Startup health checks failed after 15 attempts, falling back to 10s polling'); - } +announceIdentityPinOnce(); +loadPort().then(() => { + let startupAttempts = 0; + const startupCheck = setInterval(async () => { + startupAttempts++; + await checkHealth(); + if (isConnected || startupAttempts >= 15) { + clearInterval(startupCheck); + // Switch to slow polling now that we're connected (or gave up) + if (!healthInterval) { + healthInterval = setInterval(checkHealth, 10000); } - }, 1000); - }); + if (!isConnected) { + console.log('[gstack] Startup health checks failed after 15 attempts, falling back to 10s polling'); + } + } + }, 1000); }); diff --git a/extension/manifest.json b/extension/manifest.json index 962562646..c8194cac7 100644 --- a/extension/manifest.json +++ b/extension/manifest.json @@ -3,6 +3,7 @@ "name": "gstack browse", "version": "0.1.0", "description": "Live activity feed and @ref overlays for gstack browse", + "key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApp4uyDmQADJ/MPoKybEvBPpuGWxsXiNMo5jJFFaEaC3yPJnB4y8E0UuvE56n2KlQzaqlnBOt4T8w0ApTbNABZpEnSGQJVmkbT8a62WXefYZm79bMgzW/bNIZ4QYWNEAtZb0wvncMNSOyU9mga1s3eGWtukHs2Zf5spXRLQGV/on9l8iN9QPRM/VB0AxtUc2DTYjwkTGCAOMFiaq02miP0/hW6AeltBW9R0aHgbnJw2H2YVrgQXRvGxD1DMQe6NzGVVqKGhpUdYGPw4ONWOKijdignz+j+90HSrCK06HUy80jiKAYmdZePnn++N5meIJY+bWk7RqxS6er8Ow2U65TywIDAQAB", "permissions": ["sidePanel", "storage", "activeTab", "scripting", "tabs"], "host_permissions": ["http://127.0.0.1:*/", "ws://127.0.0.1:*/"], "action": { diff --git a/extension/sidepanel-terminal.js b/extension/sidepanel-terminal.js index e6287abca..80b47246b 100644 --- a/extension/sidepanel-terminal.js +++ b/extension/sidepanel-terminal.js @@ -504,7 +504,8 @@ window.gstackScanForPTYInject = async function (text, origin) { if (!text) return { allow: false, verdict: 'BLOCK', reasons: ['empty-text'] }; try { - const resp = await fetch('http://127.0.0.1:34567/pty-inject-scan', { + const serverPort = getServerPort() || 34567; + const resp = await fetch(`http://127.0.0.1:${serverPort}/pty-inject-scan`, { method: 'POST', headers: { 'Content-Type': 'application/json', @@ -529,21 +530,13 @@ }; // The auth token for /pty-inject-scan comes from the same source the - // sidepanel uses for /pty-session — a runtime fetch from /health (which - // already returns AUTH_TOKEN in headed mode per CLAUDE.md's v1.1 TODO). - // We don't echo the token here; this helper is a thin proxy around the - // existing pattern. + // sidepanel uses for /pty-session — window.gstackAuthToken, set by + // sidepanel.js after the pinned-origin POST /extension-token bootstrap. + // The old fallback here fetched /health and read token keys the server + // never sent (AUTH_TOKEN/authToken) — dead code since /health stopped + // carrying any token. async function getAuthTokenForScan() { - if (window.__gstackPtyScanToken) return window.__gstackPtyScanToken; - try { - const resp = await fetch('http://127.0.0.1:34567/health'); - const body = await resp.json(); - const token = body.AUTH_TOKEN || body.authToken || ''; - if (token) window.__gstackPtyScanToken = token; - return token; - } catch { - return ''; - } + return getAuthToken() || ''; } async function connect() { diff --git a/extension/sidepanel.js b/extension/sidepanel.js index 5856ebdfb..9490e3786 100644 --- a/extension/sidepanel.js +++ b/extension/sidepanel.js @@ -1304,21 +1304,36 @@ async function tryConnect() { }); if (healthResp.ok) { const data = await healthResp.json(); - if (data.status === 'healthy' && data.token) { + if (data.status === 'healthy') { + // /health is liveness-only — the token comes from the pinned-origin + // POST /extension-token bootstrap (our chrome-extension:// Origin + // is validated server-side against the manifest-pinned ID). + const tokenResp = await fetch(`http://127.0.0.1:${port}/extension-token`, { + method: 'POST', + signal: AbortSignal.timeout(2000), + }); + const tokenData = tokenResp.ok ? await tokenResp.json() : null; + if (tokenData?.token) { + setLoadingStatus( + `Server healthy on port ${port}, connecting...`, + `token: yes (from /extension-token)\nStarting SSE + activity feed...` + ); + updateConnection(`http://127.0.0.1:${port}`, tokenData.token); + // The SEC shield used to drive off /health.security via the chat + // path's classifier; with the chat path ripped, the indicator is + // not driven yet. Leaving the shield element hidden by default. + return; + } setLoadingStatus( - `Server healthy on port ${port}, connecting...`, - `token: yes (from /health)\nStarting SSE + activity feed...` + `Server healthy but token bootstrap failed (attempt ${connectAttempts})`, + `POST /extension-token → ${tokenResp.status}${tokenResp.status === 403 ? ' (extension identity not trusted)' : ''}` + ); + } else { + setLoadingStatus( + `Server responded but not healthy (attempt ${connectAttempts})`, + `status: ${data.status}` ); - updateConnection(`http://127.0.0.1:${port}`, data.token); - // The SEC shield used to drive off /health.security via the chat - // path's classifier; with the chat path ripped, the indicator is - // not driven yet. Leaving the shield element hidden by default. - return; } - setLoadingStatus( - `Server responded but not healthy (attempt ${connectAttempts})`, - `status: ${data.status}\ntoken: ${data.token ? 'yes' : 'no'}` - ); } else { setLoadingStatus( `Server returned ${healthResp.status} (attempt ${connectAttempts})`, @@ -1355,6 +1370,23 @@ chrome.runtime.onMessage.addListener((msg) => { fetchRefs(); } } + // One-time v1.62 identity-pin notice from background.js. Transient banner — + // no dedicated element in sidepanel.html since this fires once per install. + if (msg.type === 'gstack-migration-notice' && msg.message) { + console.log('[gstack sidebar]', msg.message); + try { + const banner = document.createElement('div'); + banner.textContent = msg.message; + banner.style.cssText = + 'position:fixed;left:8px;right:8px;bottom:40px;z-index:9999;' + + 'background:#1f2937;color:#f5a623;border:1px solid #f5a623;' + + 'border-radius:6px;padding:8px 10px;font-size:12px;text-align:left;'; + document.body.appendChild(banner); + setTimeout(() => banner.remove(), 8000); + } catch (err) { + console.debug('[gstack sidebar] migration banner failed:', err && err.message); + } + } if (msg.type === 'inspectResult') { inspectorPickerActive = false; inspectorPickBtn.classList.remove('active'); diff --git a/lib/context-bill.ts b/lib/context-bill.ts new file mode 100644 index 000000000..bd191004a --- /dev/null +++ b/lib/context-bill.ts @@ -0,0 +1,1003 @@ +/** + * gstack context-bill — token bill-of-materials for an installed gstack skills tree. + * + * Read-only, offline, deterministic. Ledgers over pure file reads: + * ALWAYS-ON per-skill YAML frontmatter bytes (what every session's skill + * scanner loads), flagging frontmatter keys the router never + * reads and foreign-host files in scanner scope. + * EAGER SKILL.md plus any references the skill's prose forces "for + * every invocation". + * + * This is a STRIPPED port of the v2 fork's six-ledger bill: the CONDITIONAL, + * TRANSITIVE, LAZY, and FAST-PATH parsers only understand the fork's + * dispatcher-skill layout, which this repo's skills don't use, so they were + * dropped rather than shipped dead. The tier fields stay in the report shape + * (empty arrays / zeros / nulls) so re-adding a parser is additive: nothing + * downstream needs a schema change. + * + * Token figures come from one of two sources, always named in the output: + * ESTIMATE (default, offline) bytes / TOKEN_DIVISOR, calibrated against real + * count_tokens measurements. + * EXACT (--exact, opt-in) Anthropic's count_tokens for every file the + * bill touches. Sends file content off-machine, + * so it is never implicit: an egress receipt is + * written before the POSTs (sink + * 'context-bill-exact'), and if the receipt + * cannot be written the run degrades to the + * offline estimate with a warning instead of + * sending unrecorded. + * Both bytes and tokens are always shown, and the estimate's measured error + * band is printed with it. The tool never writes state anywhere (the egress + * receipt under --exact is the one exception, and it is the point). + */ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { writeReceipt } from "./egress-receipt"; + +const FORCED_PHRASE = "for every invocation"; +// Backticked reference in prose. `<...>` is excluded: a path template such as +// `references/templates/.md` names a family of files, not one on disk. +const PROSE_REF = /`(references\/[^`<>]+\.md)`/g; +// Upstream frontmatter contract: the keys the router/host actually reads. +const ROUTER_KEYS = new Set(["name", "description", "version", "allowed-tools", "triggers", "preamble-tier"]); +// Skill-shaped files other hosts drop into scanner scope. +const FOREIGN_SKILL_FILE = /^(skill\.(ya?ml|json)|agents?\.md|\.cursorrules|\.windsurfrules)$/i; + +/** + * Bytes per token, per content class, fitted to real count_tokens measurements. + * + * Calibration corpus: 219 `.md` skill files plus their frontmatter blocks, + * measured 2026-08-01 against `claude-opus-4-5` with the per-request message + * envelope subtracted. Regenerate with `gstack-context-bill --exact + * --json` and read the `calibration` block, which grades this estimate + * against measured counts file by file. + * + * Why classes and not one divisor: measured bytes-per-token spans 2.36 to 4.72 + * across the corpus, and the spread is largely structural. Legacy specialist + * modules cluster at 3.47 (n=50, range 3.10-3.83) and SKILL.md bodies at 4.21 + * (n=50, range 3.65-4.50) -- tight enough that one divisor for both charges + * some ledgers about 19% under while charging others about right. Splitting on + * path roles cuts mean per-file error from 11.1% to 7.4% and removes the + * systematic bias, which is what a cost tool owes. + * + * What classes do NOT fix: the `reference` class is genuinely heterogeneous + * (2.36 to 4.72 -- dense path/table files sit at one end, prose at the other), + * so worst-case per-file error stays near 40%. Use --exact when a single + * file's number has to be right. + * + * These divisors are tokenizer-specific. Opus 4.7 and later tokenize + * differently; on those models use --exact. + */ +export const TOKEN_DIVISORS: Record = { + frontmatter: 3.99, + skillmd: 4.21, + reference: 4.15, + artifact: 3.67, + legacy: 3.47, +}; +/** Fallback for content that matches no class. Corpus-wide aggregate. */ +export const TOKEN_DIVISOR = 3.9; +/** Worst-case per-file residual of the estimate over the calibration corpus. */ +export const TOKEN_ESTIMATE_ERROR_PCT = 40; + +export type TokensOf = (key: string, bytes: number) => number; + +export interface RefEntry { + path: string; + bytes: number; + tokens: number; + missing: boolean; + via?: string; + condition?: string; +} + +export interface SkillBill { + name: string; + dir: string; + frontmatterBytes: number; + frontmatterTokens: number; + frontmatterKeys: string[]; + deadKeys: string[]; + skillMdBytes: number; + skillMdTokens: number; + forcedRefs: RefEntry[]; + eagerBytes: number; + eagerTokens: number; + /** Stripped tiers: kept in the shape (empty/zero/null) so re-adding the + * fork's parsers is additive. */ + fastPath: null; + conditionalRefs: RefEntry[]; + conditionalBytes: number; + conditionalTokens: number; + transitiveRefs: RefEntry[]; + transitiveBytes: number; + transitiveTokens: number; + perInvocationBytes: number; + perInvocationTokens: number; + routeCeiling: { label: string; bytes: number; tokens: number } | null; + lazy: { label: string; modules: RefEntry[]; bytes: number; tokens: number }[]; + orphans: RefEntry[]; + foreignFiles: { path: string; bytes: number; tokens: number }[]; + totalMdBytes: number; + totalMdTokens: number; +} + +/** + * Content class from the path role. Legacy/artifact roles are kept even + * though their tiers are stripped: the divisors are per-content measurements + * and --exact calibration still grades them. + */ +export function contentClass(key: string): string { + if (key.endsWith("#frontmatter")) return "frontmatter"; + if (/references[/\\]legacy[/\\]/.test(key)) return "legacy"; + if (/references[/\\](artifacts|sections|support)[/\\]/.test(key)) return "artifact"; + if (/(^|[/\\])SKILL\.md$/.test(key)) return "skillmd"; + if (/references[/\\]/.test(key)) return "reference"; + return "other"; +} + +/** Path-less callers get the corpus-wide aggregate divisor. */ +export function estimateTokens(bytes: number): number { + return Math.round(bytes / TOKEN_DIVISOR); +} + +/** Default token source: the calibrated offline estimate. Unrounded, so sums round once. */ +function estimateTokensOf(key: string, bytes: number): number { + return bytes / (TOKEN_DIVISORS[contentClass(key)] ?? TOKEN_DIVISOR); +} + +function bytesOf(file: string): number | null { + try { + const st = fs.statSync(file); + return st.isFile() ? st.size : null; + } catch { + return null; + } +} + +function refEntry(skillDir: string, rel: string, tokensOf: TokensOf): RefEntry { + const abs = path.join(skillDir, rel); + const bytes = bytesOf(abs); + return { + path: rel, + bytes: bytes ?? 0, + tokens: bytes == null ? 0 : tokensOf(abs, bytes), + missing: bytes == null, + }; +} + +function sumBytes(entries: { bytes: number }[]): number { + return entries.reduce((n, e) => n + e.bytes, 0); +} + +function sumTokens(entries: { tokens: number }[]): number { + return entries.reduce((n, e) => n + e.tokens, 0); +} + +/** Cache key for a SKILL.md's frontmatter block, which is a slice, not a whole file. */ +function frontmatterKey(skillMdPath: string): string { + return `${skillMdPath}#frontmatter`; +} + +function parseFrontmatter(text: string): { bytes: number; keys: string[]; block: string } { + if (!text.startsWith("---")) return { bytes: 0, keys: [], block: "" }; + const end = text.indexOf("\n---", 3); + if (end === -1) return { bytes: 0, keys: [], block: "" }; + const closeEol = text.indexOf("\n", end + 1); + const block = text.slice(0, closeEol === -1 ? text.length : closeEol + 1); + const inner = text.slice(text.indexOf("\n") + 1, end); + const keys: string[] = []; + for (const line of inner.split("\n")) { + const m = /^([A-Za-z0-9_-]+)\s*:/.exec(line); + if (m) keys.push(m[1]); + } + return { bytes: Buffer.byteLength(block, "utf8"), keys, block }; +} + +/** + * Every .md file under a tree, for the on-disk total and for exact + * measurement. Skips node_modules and dot-directories: a skills tree that is + * also a repo checkout (dev symlink installs) would otherwise bill its + * dependency tree and CI state as skill content. + */ +export function walkMd(dir: string): string[] { + const out: string[] = []; + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return out; + } + for (const e of entries) { + if (e.name.startsWith(".") || e.name === "node_modules") continue; + const p = path.join(dir, e.name); + if (e.isDirectory()) out.push(...walkMd(p)); + else if (e.isFile() && e.name.endsWith(".md")) out.push(p); + } + return out; +} + +function totalMd(dir: string, tokensOf: TokensOf): { bytes: number; tokens: number } { + let bytes = 0; + let tokens = 0; + for (const p of walkMd(dir)) { + const b = bytesOf(p) ?? 0; + bytes += b; + tokens += tokensOf(p, b); + } + return { bytes, tokens }; +} + +export function parseSkill(skillDir: string, name: string, tokensOf: TokensOf = estimateTokensOf): SkillBill { + const skillMdPath = path.join(skillDir, "SKILL.md"); + const text = fs.readFileSync(skillMdPath, "utf8"); + const skillMdBytes = bytesOf(skillMdPath) ?? 0; + const skillMdTokens = tokensOf(skillMdPath, skillMdBytes); + const fm = parseFrontmatter(text); + // The frontmatter block is a slice of SKILL.md, so it carries its own key. + const frontmatterTokens = tokensOf(frontmatterKey(skillMdPath), fm.bytes); + const deadKeys = fm.keys.filter((k) => !ROUTER_KEYS.has(k)); + + // EAGER: references a prose CLAUSE forces "for every invocation". Clause + // granularity matters: a line can carry a forced clause and a conditional + // one, and only the forced clause's references are eager. Routing tables + // never count (they were the fork's LAZY tier). + const forcedRefs: RefEntry[] = []; + const seenForced = new Set(); + for (const line of text.split("\n")) { + if (line.trim().startsWith("|")) continue; + for (const clause of line.split(/(?<=[.;])\s+/)) { + if (!clause.includes(FORCED_PHRASE)) continue; + for (const m of clause.matchAll(PROSE_REF)) { + const p = m[1]; + if (seenForced.has(p)) continue; + seenForced.add(p); + forcedRefs.push(refEntry(skillDir, p, tokensOf)); + } + } + } + + // Foreign-host skill files sitting next to SKILL.md. + const foreignFiles: { path: string; bytes: number; tokens: number }[] = []; + for (const entry of fs.readdirSync(skillDir, { withFileTypes: true })) { + if (entry.isFile() && FOREIGN_SKILL_FILE.test(entry.name)) { + const abs = path.join(skillDir, entry.name); + const bytes = bytesOf(abs) ?? 0; + foreignFiles.push({ path: entry.name, bytes, tokens: tokensOf(abs, bytes) }); + } + } + + const total = totalMd(skillDir, tokensOf); + const eagerBytes = skillMdBytes + sumBytes(forcedRefs); + const eagerTokens = skillMdTokens + sumTokens(forcedRefs); + return { + name, + dir: skillDir, + frontmatterBytes: fm.bytes, + frontmatterTokens, + frontmatterKeys: fm.keys, + deadKeys, + skillMdBytes, + skillMdTokens, + forcedRefs, + eagerBytes, + eagerTokens, + // Stripped tiers, shape preserved (see the module docblock). + fastPath: null, + conditionalRefs: [], + conditionalBytes: 0, + conditionalTokens: 0, + transitiveRefs: [], + transitiveBytes: 0, + transitiveTokens: 0, + // With the conditional/transitive tiers stripped, the per-invocation + // ceiling IS the eager figure. Re-adding a tier changes these sums only. + perInvocationBytes: eagerBytes, + perInvocationTokens: eagerTokens, + routeCeiling: null, + lazy: [], + orphans: [], + foreignFiles, + totalMdBytes: total.bytes, + totalMdTokens: total.tokens, + }; +} + +/** + * Every skill directory under a tree. + * + * Root-as-container (upstream fix): this repo's ROOT has a router SKILL.md + * AND fifty skill directories under it — the fork's walker short-circuited at + * the root and billed one "skill". The root is counted as a skill (the router + * costs what it costs) and the walk continues into its children. A NON-root + * dir with SKILL.md is still a leaf: its subtree (references/, test + * fixtures) is never another skill. + * + * Repo-checkout subdirs are skipped (upstream install layout fix): an + * installed ~/.claude/skills tree contains flat skill dirs PLUS a full gstack + * repo checkout (`gstack/`, with .git). Its nested SKILL.md files are the + * repo's sources, not installed skills of the tree being billed. + * + * Directory symlinks are followed (setup's shell glob follows them, so a + * symlinked skill like connect-chrome/ is real scanner load); a realpath + * seen-set breaks cycles. + */ +export function findSkillDirs(root: string): string[] { + const out: string[] = []; + const visited = new Set(); + const walk = (dir: string, isRoot: boolean) => { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + if (entries.some((e) => e.isFile() && e.name === "SKILL.md")) { + out.push(dir); + // Two symlinked paths to the same skill dir are BOTH billed (each is + // real scanner load); only container recursion below is cycle-guarded. + if (!isRoot) return; + } + // Cycle guard for container recursion (a symlink loop of directories). + let real: string; + try { + real = fs.realpathSync(dir); + } catch { + return; + } + if (visited.has(real)) return; + visited.add(real); + for (const e of entries) { + if (e.name.startsWith(".") || e.name === "node_modules") continue; + const child = path.join(dir, e.name); + let isDir = e.isDirectory(); + if (!isDir && e.isSymbolicLink()) { + try { + isDir = fs.statSync(child).isDirectory(); + } catch { + continue; // dangling symlink + } + } + if (!isDir) continue; + if (fs.existsSync(path.join(child, ".git"))) continue; // repo checkout, not a skill + walk(child, false); + } + }; + walk(path.resolve(root), true); + return out.sort(); +} + +export interface Bill { + root: string; + tokenSource: string; + tokenEstimate: Record; + tokenEstimateErrorPct: number; + calibration?: Calibration; + skills: SkillBill[]; + totals: { + skillCount: number; + alwaysOnBytes: number; + alwaysOnTokens: number; + eagerBytesBySkill: Record; + eagerTokensBySkill: Record; + perInvocationBytesBySkill: Record; + perInvocationTokensBySkill: Record; + totalMdBytes: number; + totalMdTokens: number; + }; +} + +export function buildBill( + root: string, + { tokensOf = estimateTokensOf, tokenSource, calibration }: { + tokensOf?: TokensOf; + tokenSource?: string; + calibration?: Calibration; + } = {}, +): Bill { + const resolved = path.resolve(root); + if (!fs.existsSync(resolved)) throw new Error(`No such tree: ${resolved}`); + const skills = findSkillDirs(resolved).map((dir) => + parseSkill(dir, path.relative(resolved, dir) || path.basename(resolved), tokensOf), + ); + const total = skills.reduce((n, s) => n + s.totalMdBytes, 0); + const totalTokens = skills.reduce((n, s) => n + s.totalMdTokens, 0); + return { + root: resolved, + // Named so a reader never has to guess whether a figure was measured. + tokenSource: tokenSource ?? "estimate: calibrated bytes/token per content class", + tokenEstimate: TOKEN_DIVISORS, + tokenEstimateErrorPct: tokenSource ? 0 : TOKEN_ESTIMATE_ERROR_PCT, + // Present only under --exact: how far the offline estimate was off, per file. + ...(calibration ? { calibration } : {}), + skills, + totals: { + skillCount: skills.length, + alwaysOnBytes: skills.reduce((n, s) => n + s.frontmatterBytes, 0), + alwaysOnTokens: skills.reduce((n, s) => n + s.frontmatterTokens, 0), + eagerBytesBySkill: Object.fromEntries(skills.map((s) => [s.name, s.eagerBytes])), + eagerTokensBySkill: Object.fromEntries(skills.map((s) => [s.name, Math.round(s.eagerTokens)])), + perInvocationBytesBySkill: Object.fromEntries(skills.map((s) => [s.name, s.perInvocationBytes])), + perInvocationTokensBySkill: Object.fromEntries( + skills.map((s) => [s.name, Math.round(s.perInvocationTokens)]), + ), + totalMdBytes: total, + totalMdTokens: totalTokens, + }, + }; +} + +export interface DiffRow { + ledger: string; + label: string; + before: number; + after: number; + delta: number; + tokenDelta: number; +} + +export function diffBills(a: Bill, b: Bill): { rows: DiffRow[]; grew: boolean } { + const rows: DiffRow[] = []; + const push = (ledger: string, label: string, before: number, after: number, tokBefore: number, tokAfter: number) => { + if (before !== after) { + rows.push({ ledger, label, before, after, delta: after - before, tokenDelta: Math.round(tokAfter - tokBefore) }); + } + }; + const skillNames = [...new Set([...a.skills, ...b.skills].map((s) => s.name))].sort(); + for (const name of skillNames) { + const sa = a.skills.find((s) => s.name === name); + const sb = b.skills.find((s) => s.name === name); + push( + "always-on", name, + sa?.frontmatterBytes ?? 0, sb?.frontmatterBytes ?? 0, + sa?.frontmatterTokens ?? 0, sb?.frontmatterTokens ?? 0, + ); + push("eager", name, sa?.eagerBytes ?? 0, sb?.eagerBytes ?? 0, sa?.eagerTokens ?? 0, sb?.eagerTokens ?? 0); + // Stripped tiers stay in the diff contract so re-adding them is additive. + push( + "conditional", name, + sa?.conditionalBytes ?? 0, sb?.conditionalBytes ?? 0, + sa?.conditionalTokens ?? 0, sb?.conditionalTokens ?? 0, + ); + push( + "transitive", name, + sa?.transitiveBytes ?? 0, sb?.transitiveBytes ?? 0, + sa?.transitiveTokens ?? 0, sb?.transitiveTokens ?? 0, + ); + } + rows.sort((x, y) => Math.abs(y.delta) - Math.abs(x.delta)); + const grew = + b.totals.alwaysOnBytes > a.totals.alwaysOnBytes || + rows.some((r) => ["eager", "conditional", "transitive"].includes(r.ledger) && r.delta > 0); + return { rows, grew }; +} + +export interface BudgetViolation { + ceiling: string; + limit: number; + actual: number | null; + files: string[]; +} + +/** + * Budget file: user-authored plain JSON, ceilings in ~tokens. + * { "alwaysOnTotal": 4000, "eagerPerInvocation": { "qa": 5000 }, + * "perInvocation": { "qa": 9000 } } + * With the conditional/transitive tiers stripped, `perInvocation` and + * `routeCeiling` gate the same figure as `eagerPerInvocation`; the keys stay + * accepted so budgets survive the tiers returning. + */ +export function checkBudget(bill: Bill, budget: Record): BudgetViolation[] { + const violations: BudgetViolation[] = []; + if (typeof budget.alwaysOnTotal === "number") { + const actual = Math.round(bill.totals.alwaysOnTokens); + if (actual > budget.alwaysOnTotal) { + violations.push({ + ceiling: "alwaysOnTotal", + limit: budget.alwaysOnTotal, + actual, + files: bill.skills.map((s) => `${s.name}/SKILL.md (frontmatter ${s.frontmatterBytes}B)`), + }); + } + } + for (const key of ["eagerPerInvocation", "perInvocation", "routeCeiling"]) { + for (const [name, limit] of Object.entries(budget[key] ?? {}) as [string, number][]) { + const skill = bill.skills.find((s) => s.name === name); + if (!skill) { + violations.push({ ceiling: `${key}.${name}`, limit, actual: null, files: [""] }); + continue; + } + const tokens = + key === "routeCeiling" + ? (skill.routeCeiling?.tokens ?? skill.perInvocationTokens) + : key === "perInvocation" + ? skill.perInvocationTokens + : skill.eagerTokens; + const actual = Math.round(tokens); + if (actual > limit) { + violations.push({ + ceiling: `${key}.${name}`, + limit, + actual, + files: [ + `${skill.name}/SKILL.md (${skill.skillMdBytes}B)`, + ...skill.forcedRefs.map((r) => `${skill.name}/${r.path} (${r.bytes}B)`), + ], + }); + } + } + } + return violations; +} + +function fmtBytes(b: number): string { + if (b >= 1024 * 1024) return `${(b / 1024 / 1024).toFixed(1)}MB`; + if (b >= 1024) return `${(b / 1024).toFixed(1)}KB`; + return `${b}B`; +} + +/** Exact counts are measurements, so they lose the "~" the estimate wears. */ +function fmtTok(tokens: number, exact: boolean): string { + const t = Math.round(tokens); + const tilde = exact ? "" : "~"; + return t >= 1000 ? `${tilde}${(t / 1000).toFixed(1)}K tok` : `${tilde}${t} tok`; +} + +export function renderBill(bill: Bill, { skill }: { skill?: string } = {}): string { + const skills = skill ? bill.skills.filter((s) => s.name === skill) : bill.skills; + const exact = bill.tokenEstimateErrorPct === 0; + const size = (bytes: number, tokens: number) => `${fmtBytes(bytes)} (${fmtTok(tokens, exact)})`; + const lines = [`Context bill for ${bill.root}`, `Token source: ${bill.tokenSource}`, ""]; + + lines.push( + `ALWAYS-ON (every session): ${skills.length} skills, ` + + `${size(skills.reduce((n, s) => n + s.frontmatterBytes, 0), skills.reduce((n, s) => n + s.frontmatterTokens, 0))}`, + ); + // The host wraps each skill's frontmatter in its own available_skills XML + // element before the model sees it. That wrapper is host-specific and cannot + // be read from this tree, so it is excluded here — the real always-on cost is + // this figure plus one wrapper per skill. + lines.push(" (frontmatter only; excludes the host's per-skill available_skills XML wrapper)"); + for (const s of skills) lines.push(` ${s.name.padEnd(20)} ${size(s.frontmatterBytes, s.frontmatterTokens)}`); + for (const s of skills) { + if (s.deadKeys.length) lines.push(` ! ${s.name}: frontmatter key(s) the router never reads: ${s.deadKeys.join(", ")}`); + for (const f of s.foreignFiles) lines.push(` ! ${s.name}: foreign-host file in scanner scope: ${f.path} (${size(f.bytes, f.tokens)})`); + } + lines.push(""); + + lines.push("EAGER (per invocation): SKILL.md + forced-read references"); + for (const s of skills) { + const refs = s.forcedRefs.length + ? ` = SKILL.md ${fmtBytes(s.skillMdBytes)} + refs ${fmtBytes(sumBytes(s.forcedRefs))} (${s.forcedRefs.map((r) => path.basename(r.path)).join(", ")})` + : ""; + lines.push(` ${s.name.padEnd(20)} ${size(s.eagerBytes, s.eagerTokens)}${refs}`); + for (const r of s.forcedRefs.filter((r) => r.missing)) lines.push(` ! ${s.name}: forced-read reference missing on disk: ${r.path}`); + } + lines.push(""); + + lines.push( + `TOTAL on disk: ${size(bill.totals.totalMdBytes, bill.totals.totalMdTokens)} across ${bill.totals.skillCount} skill(s).`, + ); + lines.push(tokenDisclaimer(bill)); + return lines.join("\n") + "\n"; +} + +/** + * Names the error band instead of hand-waving about "estimates". The band is the + * worst-case residual measured over the calibration corpus, not a guess. + */ +export function tokenDisclaimer(bill: Pick): string { + if (bill.tokenEstimateErrorPct === 0) { + return `Token counts measured with ${bill.tokenSource}. Bytes are exact.`; + } + const per = Object.entries(TOKEN_DIVISORS).map(([k, v]) => `${k} /${v}`).join(", "); + return ( + `Token counts are ESTIMATES: bytes divided per content class (${per}), calibrated against ` + + `count_tokens on 219 skill files. Measured accuracy of that estimate: mean ` + + `per-file error 7.4%, systematic bias under 0.5%, worst single file ` + + `${bill.tokenEstimateErrorPct}% (dense path/table files). Ledger rows ` + + `land tighter than single files because errors partly cancel across a sum. Run --exact for ` + + `measured counts when a number has to be right. Bytes are always exact.` + ); +} + +export function renderDiff(diff: { rows: DiffRow[]; grew: boolean }): string { + if (diff.rows.length === 0) return "No context-cost changes between trees.\n"; + const lines = ["Context-cost changes (sorted by |delta|):", ""]; + for (const r of diff.rows) { + const sign = r.delta > 0 ? "+" : "-"; + lines.push( + ` ${r.ledger.padEnd(11)} ${r.label.padEnd(28)} ${sign}${fmtBytes(Math.abs(r.delta))} (${sign}${Math.abs(r.tokenDelta)} tok) ${fmtBytes(r.before)} -> ${fmtBytes(r.after)}`, + ); + } + lines.push(""); + lines.push( + diff.grew + ? "RESULT: context cost GREW (always-on or eager)." + : "RESULT: no always-on or eager growth.", + ); + return lines.join("\n") + "\n"; +} + +// --exact defaults to the model the offline divisor was calibrated against, so +// `--exact` and the estimate are comparable. Later tokenizers differ. +export const EXACT_DEFAULT_MODEL = "claude-opus-4-5"; +const COUNT_TOKENS_URL = "https://api.anthropic.com/v1/messages/count_tokens"; +const EXACT_CONCURRENCY = 8; + +/** Typed failures, so callers branch on a code rather than on message text. */ +export class ExactModeError extends Error { + code: string; + constructor(code: string, message: string) { + super(message); + this.name = "ExactModeError"; + this.code = code; + } +} + +type FetchLike = typeof globalThis.fetch; + +interface CountTokensOptions { + model: string; + apiKey: string; + fetchImpl: FetchLike; +} + +async function countTokens(text: string, { model, apiKey, fetchImpl }: CountTokensOptions): Promise { + let res: Response; + try { + res = await fetchImpl(COUNT_TOKENS_URL, { + method: "POST", + headers: { + "content-type": "application/json", + "x-api-key": apiKey, + "anthropic-version": "2023-06-01", + }, + body: JSON.stringify({ model, messages: [{ role: "user", content: text }] }), + }); + } catch (error) { + throw new ExactModeError("exact_network_unreachable", `count_tokens unreachable: ${(error as Error)?.message ?? error}`); + } + if (!res.ok) { + const body = await res.text().catch(() => ""); + const code = res.status === 401 || res.status === 403 ? "exact_auth_rejected" : "exact_request_failed"; + throw new ExactModeError(code, `count_tokens returned ${res.status}: ${body.slice(0, 200)}`); + } + const json: any = await res.json(); + if (typeof json?.input_tokens !== "number") { + throw new ExactModeError("exact_response_malformed", "count_tokens response had no input_tokens"); + } + return json.input_tokens; +} + +export interface ExactMeasurement { + tokenSource: string; + tokensOf: TokensOf; + measuredFiles: number; + counts: Map; + /** Keys priced by estimate because measurement missed them. Read after buildBill. */ + missedKeys: Set; +} + +/** + * Measures every text the bill will bill for. Returns a `tokensOf` lookup. + * + * count_tokens prices a whole request, so it includes a fixed message envelope. + * That envelope is measured once and subtracted, leaving the tokens each file's + * own content contributes — otherwise every small reference is overcharged by a + * constant that has nothing to do with the file. + * + * Egress receipt BEFORE any POST (sink 'context-bill-exact'): if the receipt + * cannot be written this throws exact_egress_receipt_failed, which the CLI + * degrades to the offline estimate — nothing is sent unrecorded. + */ +export async function measureExactTokens( + root: string, + { model, apiKey, fetchImpl = fetch, onProgress, egressHome }: { + model: string; + apiKey: string; + fetchImpl?: FetchLike; + onProgress?: (done: number, total: number) => void; + egressHome?: string; + }, +): Promise { + if (!apiKey) { + throw new ExactModeError( + "exact_missing_api_key", + "--exact needs ANTHROPIC_API_KEY. Without it the offline estimate is used; nothing was sent.", + ); + } + const opts: CountTokensOptions = { model, apiKey, fetchImpl }; + + const texts = new Map(); + // Resolve before keying. buildBill resolves its root, so a relative root here + // would produce keys that never match and every lookup would fall back to the + // estimate -- exact mode silently degrading to the thing it replaces. + for (const file of walkMd(path.resolve(root))) { + const text = fs.readFileSync(file, "utf8"); + texts.set(file, text); + if (path.basename(file) === "SKILL.md") { + const fm = parseFrontmatter(text); + if (fm.block) texts.set(frontmatterKey(file), fm.block); + } + } + + // Receipt-before-send. Content-free: file count + total bytes only. + try { + let totalBytes = 0; + for (const t of texts.values()) totalBytes += Buffer.byteLength(t, "utf8"); + writeReceipt({ + home: egressHome, + sink: "context-bill-exact", + host: "api.anthropic.com", + payloadClass: `count-tokens skill-tree texts=${texts.size} (${totalBytes}B across ${EXACT_CONCURRENCY}-way POSTs)`, + bytes: totalBytes, + sha256: null, + consent: "user passed --exact", + }); + } catch (error) { + throw new ExactModeError( + "exact_egress_receipt_failed", + `egress receipt could not be written (${(error as Error)?.message ?? error}); refusing to send unrecorded`, + ); + } + + // One-char body: subtracting its single content token leaves the envelope. + const envelope = (await countTokens("x", opts)) - 1; + + const counts = new Map(); + const keys = [...texts.keys()]; + let next = 0; + let done = 0; + const worker = async () => { + while (next < keys.length) { + const key = keys[next++]; + const raw = await countTokens(texts.get(key)!, opts); + counts.set(key, Math.max(0, raw - envelope)); + onProgress?.(++done, keys.length); + } + }; + await Promise.all(Array.from({ length: Math.min(EXACT_CONCURRENCY, keys.length) }, worker)); + + // A key the walk never saw (a non-.md foreign-host file) falls back to the + // estimate rather than billing zero. Misses are counted, not swallowed: a bill + // that is part-measured and part-estimated must not present itself as measured. + const missed = new Set(); + const tokensOf: TokensOf = (key, bytes) => { + const exact = counts.get(key); + if (exact !== undefined) return exact; + missed.add(key); + return estimateTokensOf(key, bytes); + }; + return { + tokenSource: `count_tokens (${model})`, + tokensOf, + measuredFiles: counts.size, + counts, + missedKeys: missed, + }; +} + +export interface Calibration { + rows: { + path: string; + contentClass: string; + bytes: number; + estimatedTokens: number; + tokens: number; + bytesPerToken: number; + errorPct: number; + }[]; + worstErrorPct: number; + meanAbsErrorPct: number; + biasPct: number; +} + +/** + * Estimate-vs-measured residual per file. This is what makes the divisor + * auditable: run --exact and the tool grades its own offline estimate. + */ +export function calibrationTable(counts: Map, root: string): Calibration { + const rows: Calibration["rows"] = []; + for (const [key, tokens] of counts) { + if (key.endsWith("#frontmatter") || tokens === 0) continue; + const bytes = bytesOf(key); + if (bytes == null) continue; + // Grade the estimate the tool actually uses, class divisor included. + const estimated = Math.round(estimateTokensOf(key, bytes)); + rows.push({ + path: path.relative(root, key), + contentClass: contentClass(key), + bytes, + estimatedTokens: estimated, + tokens, + bytesPerToken: Number((bytes / tokens).toFixed(3)), + errorPct: Number((((estimated - tokens) / tokens) * 100).toFixed(1)), + }); + } + rows.sort((a, b) => Math.abs(b.errorPct) - Math.abs(a.errorPct)); + const abs = rows.map((r) => Math.abs(r.errorPct)); + return { + rows, + worstErrorPct: abs.length ? Math.max(...abs) : 0, + meanAbsErrorPct: abs.length ? Number((abs.reduce((a, b) => a + b, 0) / abs.length).toFixed(2)) : 0, + biasPct: rows.length + ? Number((rows.reduce((n, r) => n + r.errorPct, 0) / rows.length).toFixed(2)) + : 0, + }; +} + +// Where installed skills actually live: `.agents/skills` (the host-neutral +// canonical path) alongside `.claude/skills`, project then user. +const DEFAULT_TREES: string[][] = [ + ["cwd", "skills"], + ["cwd", ".agents", "skills"], + ["cwd", ".claude", "skills"], + ["home", ".agents", "skills"], + ["home", ".claude", "skills"], +]; + +function defaultTreeCandidates(cwd: string, homeDir: string): string[] { + return DEFAULT_TREES.map(([base, ...rest]) => path.join(base === "cwd" ? cwd : homeDir, ...rest)); +} + +function detectDefaultTree(cwd: string, homeDir: string): string | null { + return defaultTreeCandidates(cwd, homeDir).find((c) => fs.existsSync(c)) ?? null; +} + +const USAGE = + "Usage:\n" + + " gstack-context-bill [TREE] [--json] [--skill ]\n" + + " gstack-context-bill --diff [--json]\n" + + " gstack-context-bill [TREE] --budget [--json]\n" + + "\n" + + " --exact measure tokens with Anthropic's count_tokens instead of\n" + + " estimating. Off by default: it sends the content of every\n" + + " .md file in the tree to api.anthropic.com. Needs\n" + + " ANTHROPIC_API_KEY; passing --exact is the consent. An\n" + + " egress receipt is written before the send (see\n" + + " gstack-egress); if it cannot be written, the run falls\n" + + " back to the offline estimate.\n" + + " --exact also recalibrates: the --json output's\n" + + " `calibration` block grades the offline divisors\n" + + " (TOKEN_DIVISORS) file by file against measured counts.\n" + + " --exact-model model whose tokenizer to count against\n" + + ` (default ${EXACT_DEFAULT_MODEL}, the calibration model).\n`; + +export interface MainOptions { + cwd?: string; + stdout?: { write(s: string): unknown }; + stderr?: { write(s: string): unknown }; + homeDir?: string; + apiKey?: string; + fetchImpl?: FetchLike; + egressHome?: string; +} + +export async function contextBillMain(argv: string[], options: MainOptions = {}): Promise { + const cwd = options.cwd ?? process.cwd(); + const stdout = options.stdout ?? process.stdout; + const stderr = options.stderr ?? process.stderr; + const homeDir = options.homeDir ?? os.homedir(); + + const positional: string[] = []; + const flags: { json: boolean; diff: boolean; exact: boolean; exactModel: string; skill?: string; budget?: string } = + { json: false, diff: false, exact: false, exactModel: EXACT_DEFAULT_MODEL }; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === "--json") flags.json = true; + else if (arg === "--diff") flags.diff = true; + else if (arg === "--skill") flags.skill = argv[++i]; + else if (arg === "--budget") flags.budget = argv[++i]; + else if (arg === "--exact") flags.exact = true; + else if (arg === "--exact-model") flags.exactModel = argv[++i]; + else if (arg === "--help" || arg === "-h") { + stdout.write(USAGE); + return 0; + } else if (arg.startsWith("--")) { + stderr.write(`Unknown flag: ${arg}\n${USAGE}`); + return 2; + } else positional.push(arg); + } + + // Exact mode is the only path that leaves the machine. Announce what is sent + // before sending it, and degrade to the estimate rather than failing the run. + const exactFor = async (tree: string): Promise<{ + tokensOf?: TokensOf; + tokenSource?: string; + calibration?: Calibration; + onDone?: () => void; + }> => { + if (!flags.exact) return {}; + const files = walkMd(tree).length; + stderr.write( + `--exact: sending the content of ${files} .md file(s) under ${tree} to ` + + `api.anthropic.com for count_tokens (${flags.exactModel}). No other data leaves this machine.\n`, + ); + try { + const measured = await measureExactTokens(tree, { + model: flags.exactModel, + apiKey: options.apiKey ?? process.env.ANTHROPIC_API_KEY ?? "", + fetchImpl: options.fetchImpl, + egressHome: options.egressHome, + }); + return { + tokensOf: measured.tokensOf, + tokenSource: measured.tokenSource, + calibration: calibrationTable(measured.counts, tree), + onDone: () => { + if (measured.missedKeys.size) { + stderr.write( + `--exact: ${measured.missedKeys.size} item(s) had no measurement and were estimated ` + + `(${[...measured.missedKeys].slice(0, 3).join(", ")}). Those figures are not measurements.\n`, + ); + } + }, + }; + } catch (error) { + if (!(error instanceof ExactModeError)) throw error; + stderr.write(`--exact unavailable [${error.code}]: ${error.message}\nFalling back to the offline estimate.\n`); + return {}; + } + }; + + try { + if (flags.diff) { + if (positional.length !== 2) { + stderr.write(`--diff needs exactly two trees.\n${USAGE}`); + return 2; + } + const treeA = path.resolve(cwd, positional[0]); + const treeB = path.resolve(cwd, positional[1]); + const optsA = await exactFor(treeA); + const optsB = await exactFor(treeB); + const diff = diffBills(buildBill(treeA, optsA), buildBill(treeB, optsB)); + optsA.onDone?.(); + optsB.onDone?.(); + stdout.write(flags.json ? JSON.stringify(diff, null, 2) + "\n" : renderDiff(diff)); + return diff.grew ? 2 : 0; + } + + const tree = positional[0] ? path.resolve(cwd, positional[0]) : detectDefaultTree(cwd, homeDir); + if (!tree) { + stderr.write( + `No skills tree found (tried ${defaultTreeCandidates(cwd, homeDir).join(", ")}). Pass a path.\n`, + ); + return 2; + } + const exactOpts = await exactFor(tree); + const bill = buildBill(tree, exactOpts); + exactOpts.onDone?.(); + if (bill.skills.length === 0) { + stderr.write(`No SKILL.md files found under ${tree}.\n`); + return 2; + } + if (flags.skill && !bill.skills.some((s) => s.name === flags.skill)) { + stderr.write(`No skill named "${flags.skill}" in ${tree}. Skills: ${bill.skills.map((s) => s.name).join(", ")}\n`); + return 2; + } + + if (flags.budget) { + const budget = JSON.parse(fs.readFileSync(path.resolve(cwd, flags.budget), "utf8")); + const violations = checkBudget(bill, budget); + if (flags.json) { + stdout.write(JSON.stringify({ ok: violations.length === 0, violations }, null, 2) + "\n"); + } else if (violations.length === 0) { + stdout.write("Within budget.\n"); + } else { + for (const v of violations) { + stdout.write(`OVER BUDGET: ${v.ceiling} at ~${v.actual} tok (ceiling ~${v.limit} tok)\n`); + for (const f of v.files) stdout.write(` ${f}\n`); + } + } + return violations.length === 0 ? 0 : 2; + } + + stdout.write(flags.json ? JSON.stringify(bill, null, 2) + "\n" : renderBill(bill, flags)); + return 0; + } catch (error) { + stderr.write(`${(error as Error)?.message ?? error}\n`); + return 1; + } +} diff --git a/lib/egress-receipt.ts b/lib/egress-receipt.ts new file mode 100644 index 000000000..61b374430 --- /dev/null +++ b/lib/egress-receipt.ts @@ -0,0 +1,392 @@ +/** + * egress-receipt — hash-chained, content-free receipts for every + * gstack-initiated off-machine send (`~/.gstack/security/egress.jsonl`, 0600). + * + * THREAT MODEL: the egress ledger is forensic observability — it records + * ATTEMPTED egress so accidents are auditable; it is not an exfiltration + * control. Receipts are written before send, outcomes are best-effort, and + * fail-open sinks can send unrecorded with a warning. + * + * Semantics: + * - Receipt-before-send: the receipt line is appended BEFORE the network + * call. Fail-closed sinks MUST refuse the send with the typed code + * EGRESS_RECEIPT_FAILED when it cannot be written; fail-open sinks warn + * on stderr and proceed. + * - Content-free: never payload text, never credentials — a sha256 of the + * exact bytes sent plus a byte count only (semantic-reviews.jsonl + * precedent). Sinks where a subprocess/SDK owns the bytes record + * sha256: null. + * - Tamper-evident: each line carries `prev` = sha256 of the previous raw + * line ("" for line 1). `verifyLedger` recomputes the chain. + * + * Node builtins only, so bun TS binaries and the compiled browse binary can + * both import it (same constraint as browse/src/security.ts: no native + * modules). + */ + +import { createHash } from 'node:crypto'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +export const EGRESS_RECEIPT_FAILED = 'EGRESS_RECEIPT_FAILED'; + +const SHA256_HEX = /^[0-9a-f]{64}$/; + +/** + * WARN-at-size threshold. Above this the ledger still appends (never blocks + * on size), but writeReceipt emits one stderr warning per process so the + * user learns the file exists and how to inspect it before it gets silly. + */ +export const LEDGER_WARN_BYTES = 25 * 1024 * 1024; + +/** + * Tail window for reading the last raw line. Receipt lines are ~300 bytes; + * 4KB covers any legal line with an order of magnitude to spare. + */ +const TAIL_READ_BYTES = 4096; + +// TODO(rotation): ledger rotation. Chain-genesis sketch: when the ledger +// exceeds the size threshold, rename it to egress.jsonl.1 and start a new +// generation whose FIRST record embeds `genesis: sha256()`, so verifyLedger can walk generations end-to-end +// (verify each file's internal chain, then check each genesis hash against +// the previous generation's last line). Until then we warn at 25MB. + +type Env = Record; + +export interface WriteReceiptOptions { + /** gstack home; resolved from env when omitted */ + home?: string; + /** env for home resolution (tests) */ + env?: Env; + /** which gstack component is sending */ + sink: string; + /** destination host[:port] */ + host: string; + /** content-free payload description */ + payloadClass: string; + /** exact byte count sent (0 for bodyless requests) */ + bytes?: number; + /** sha256 hex of the exact bytes sent; null when a subprocess/SDK owns the bytes */ + sha256?: string | null; + /** the consent key+value that authorizes this send */ + consent: string; +} + +export interface WriteOutcomeOptions { + home?: string; + env?: Env; + /** receipt id returned by writeReceipt */ + receipt: string; + status?: string | number; +} + +export interface LedgerLine { + lineNo: number; + raw: string; + record: Record | null; +} + +export interface VerifyResult { + ok: boolean; + count: number; + brokenLine: number | null; + reason: string | null; + /** present when the ledger file exceeds LEDGER_WARN_BYTES */ + sizeWarning: string | null; +} + +/** + * Same resolution order as the rest of gstack (shell sinks, selection code): + * GSTACK_HOME, legacy GSTACK_STATE_DIR, then $HOME/.gstack. + */ +export function resolveEgressHome(env: Env = process.env): string { + const configured = env.GSTACK_HOME || env.GSTACK_STATE_DIR; + if (configured) return path.resolve(configured); + return path.join(env.HOME || os.homedir(), '.gstack'); +} + +export function egressLedgerPath(home: string): string { + return path.join(home, 'security', 'egress.jsonl'); +} + +export function sha256Hex(data: string | Uint8Array): string { + return createHash('sha256').update(data).digest('hex'); +} + +function receiptError(message: string, cause?: unknown): Error & { code: string } { + const error = (cause === undefined ? new Error(message) : new Error(message, { cause })) as Error & { code: string }; + error.code = EGRESS_RECEIPT_FAILED; + return error; +} + +// A serialized receipt line must stay under TAIL_READ_BYTES so the O(1) +// tail-read always captures the FULL previous line before hashing it into the +// chain. A caller-controlled field (sink/host/payloadClass/consent — e.g. +// context-bill builds payloadClass dynamically) long enough to push the line +// past the tail window would make the next append hash a truncated prior line, +// and verifyLedger would then report a permanent false TAMPER. Cap each field +// well under the window so the invariant holds by construction. +const MAX_FIELD_BYTES = 512; + +function requireString(value: unknown, name: string): string { + if (typeof value !== 'string' || !value) throw receiptError(`Egress receipt requires a non-empty ${name}`); + if (Buffer.byteLength(value) > MAX_FIELD_BYTES) { + throw receiptError(`Egress receipt ${name} exceeds ${MAX_FIELD_BYTES} bytes (${Buffer.byteLength(value)})`); + } + return value; +} + +/** + * mkdir spin lock, ~2.5s budget. Egress events are rare (minutes apart); the + * lock only protects the read-last-line → append window. + * + * Stale-lock reclaim: a crashed writer strands the lock dir. Once the spin + * budget is exhausted, a lock dir whose mtime is >10s old is stale by + * definition (appends take milliseconds), so the waiter removes it and + * retries instead of failing. The rmdir/stat races with a concurrent + * reclaimer or the owner's own cleanup are harmless — losers just loop. + */ +function withLedgerLock(ledger: string, callback: () => T): T { + const lock = `${ledger}.lock`; + const deadline = Date.now() + 2500; + for (;;) { + try { + fs.mkdirSync(lock); + break; + } catch (error) { + if ((error as NodeJS.ErrnoException)?.code !== 'EEXIST') throw error; + if (Date.now() > deadline) { + try { + const age = Date.now() - fs.statSync(lock).mtimeMs; + if (age > 10_000) { fs.rmdirSync(lock); continue; } + } catch { /* raced with the owner's cleanup — retry */ } + throw receiptError(`Egress ledger is locked: ${lock}`); + } + // Sync sleep (node + bun): the API is sync on purpose so shell, bun, + // and node callers all share one implementation. + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 10); + } + } + try { + return callback(); + } finally { + try { fs.rmdirSync(lock); } catch { /* best-effort unlock */ } + } +} + +/** + * Last raw line via a tail read: open the file, read the final + * TAIL_READ_BYTES, take the last newline-terminated chunk. Never loads the + * whole ledger, so appends stay O(1) as the file grows. + */ +function lastRawLine(ledger: string): string | null { + let fd: number; + try { + fd = fs.openSync(ledger, 'r'); + } catch (error) { + if ((error as NodeJS.ErrnoException)?.code === 'ENOENT') return null; + throw error; + } + try { + const size = fs.fstatSync(fd).size; + if (size === 0) return null; + const length = Math.min(size, TAIL_READ_BYTES); + const buffer = Buffer.alloc(length); + fs.readSync(fd, buffer, 0, length, size - length); + const tail = buffer.toString('utf8'); + const lines = tail.split('\n').filter((line) => line.length > 0); + return lines.length ? lines[lines.length - 1] : null; + } finally { + fs.closeSync(fd); + } +} + +let warnedLedgerSize = false; + +/** Test-only: re-arm the once-per-process size warning. */ +export function resetLedgerSizeWarningForTests(): void { + warnedLedgerSize = false; +} + +function warnLedgerSizeOnce(ledger: string): void { + // Short-circuit BEFORE the stat: the warning fires at most once per process, + // so after it has fired there is no reason to stat the ledger on every + // subsequent writeReceipt (this runs on the append hot path). + if (warnedLedgerSize) return; + let size: number; + try { + size = fs.statSync(ledger).size; + } catch { + return; // no file yet — nothing to warn about + } + if (size <= LEDGER_WARN_BYTES) return; + warnedLedgerSize = true; + process.stderr.write(ledgerSizeWarning(ledger, size) + '\n'); +} + +/** + * Self-explanatory size warning: says what the ledger is (records what + * gstack ATTEMPTS to send off-machine), how to inspect it, and that + * trimming arrives with rotation. + */ +export function ledgerSizeWarning(ledger: string, size: number): string { + const mb = (size / (1024 * 1024)).toFixed(1); + return ( + `gstack: egress ledger is large (${mb}MB): ${ledger}. ` + + `This file records what gstack ATTEMPTS to send off-machine (content-free receipts, for auditing). ` + + `Inspect it with 'gstack-egress list'. Trimming arrives with ledger rotation (TODO); until then it only grows.` + ); +} + +function appendChained( + homeOrNull: string | null, + record: Record, + env?: Env, +): { id: string; path: string } { + const home = homeOrNull ?? resolveEgressHome(env); + const ledger = egressLedgerPath(home); + try { + fs.mkdirSync(path.dirname(ledger), { recursive: true, mode: 0o700 }); + return withLedgerLock(ledger, () => { + const previous = lastRawLine(ledger); + const line = JSON.stringify({ ...record, prev: previous == null ? '' : sha256Hex(previous) }); + const existed = fs.existsSync(ledger); + fs.appendFileSync(ledger, `${line}\n`, { mode: 0o600 }); + if (!existed) fs.chmodSync(ledger, 0o600); // umask must not weaken the ledger + return { id: sha256Hex(line), path: ledger }; + }); + } catch (error) { + if ((error as NodeJS.ErrnoException)?.code === EGRESS_RECEIPT_FAILED) throw error; + throw receiptError( + `Egress receipt could not be written to ${ledger}: ${(error as Error)?.message ?? error}`, + error, + ); + } +} + +/** + * Append one content-free receipt BEFORE a network send. + * + * @returns id = sha256 of the written line (for writeOutcome) + * @throws Error with code EGRESS_RECEIPT_FAILED — fail-closed callers must + * refuse the send; fail-open callers warn and proceed + */ +export function writeReceipt(opts: WriteReceiptOptions): { id: string; path: string } { + const sink = requireString(opts.sink, 'sink'); + const host = requireString(opts.host, 'host'); + const payloadClass = requireString(opts.payloadClass, 'payloadClass'); + const consent = requireString(opts.consent, 'consent'); + const bytes = opts.bytes ?? 0; + if (!Number.isSafeInteger(bytes) || bytes < 0) throw receiptError('Egress receipt bytes must be a non-negative integer'); + const sha256 = opts.sha256 ?? null; + if (sha256 !== null && !SHA256_HEX.test(String(sha256))) throw receiptError('Egress receipt sha256 must be 64 lowercase hex chars or null'); + const home = opts.home ?? resolveEgressHome(opts.env); + warnLedgerSizeOnce(egressLedgerPath(home)); + return appendChained(home, { + ts: new Date().toISOString(), + type: 'egress', + sink, + host, + payload_class: payloadClass, + bytes, + sha256, + consent, + }, opts.env); +} + +/** + * Append the response status for an earlier receipt (best-effort companion + * record — the pre-send receipt is the invariant, the outcome is + * bookkeeping). Chained like every other line. + */ +export function writeOutcome(opts: WriteOutcomeOptions): { id: string; path: string } { + const receipt = requireString(opts.receipt, 'receipt id'); + return appendChained(opts.home ?? null, { + ts: new Date().toISOString(), + type: 'outcome', + receipt, + status: String(opts.status ?? 'unknown'), + }, opts.env); +} + +/** Raw parsed lines: [{lineNo, raw, record|null}]. Missing ledger → []. */ +export function readLedger(home: string): LedgerLine[] { + const ledger = egressLedgerPath(home); + let content: string; + try { + content = fs.readFileSync(ledger, 'utf8'); + } catch (error) { + if ((error as NodeJS.ErrnoException)?.code === 'ENOENT') return []; + throw error; + } + return content.split('\n').filter((line) => line.length > 0).map((raw, index) => { + let record: Record | null = null; + try { + const parsed = JSON.parse(raw); + if (parsed && typeof parsed === 'object') record = parsed; + } catch { /* malformed line — verifyLedger reports it */ } + return { lineNo: index + 1, raw, record }; + }); +} + +export interface Receipt { + ts: string; + type: 'egress'; + sink: string; + host: string; + payload_class: string; + bytes: number; + sha256: string | null; + consent: string; + prev: string; + id: string; + status: string | null; +} + +/** Receipts with their joined outcome status (`status: null` = none recorded). */ +export function listReceipts(home: string): Receipt[] { + const lines = readLedger(home); + const receipts: Receipt[] = []; + const byId = new Map(); + for (const { raw, record } of lines) { + if (!record) continue; + if (record.type === 'egress') { + const entry = { ...(record as unknown as Omit), id: sha256Hex(raw), status: null }; + receipts.push(entry); + byId.set(entry.id, entry); + } else if (record.type === 'outcome' && byId.has(record.receipt as string)) { + byId.get(record.receipt as string)!.status = String(record.status); + } + } + return receipts; +} + +/** + * Recompute the hash chain. `brokenLine` is the 1-indexed first line whose + * `prev` no longer matches the sha256 of the previous raw line (or that + * fails to parse). `sizeWarning` is set when the ledger exceeds + * LEDGER_WARN_BYTES. + */ +export function verifyLedger(home: string): VerifyResult { + const ledger = egressLedgerPath(home); + let sizeWarning: string | null = null; + try { + const size = fs.statSync(ledger).size; + if (size > LEDGER_WARN_BYTES) sizeWarning = ledgerSizeWarning(ledger, size); + } catch { /* missing ledger — verify of an empty chain below */ } + const lines = readLedger(home); + let previousRaw: string | null = null; + for (const { lineNo, raw, record } of lines) { + if (!record || typeof record.prev !== 'string') { + return { ok: false, count: lines.length, brokenLine: lineNo, reason: 'unparseable or missing prev', sizeWarning }; + } + const expected = previousRaw == null ? '' : sha256Hex(previousRaw); + if (record.prev !== expected) { + return { ok: false, count: lines.length, brokenLine: lineNo, reason: 'prev hash does not match previous line', sizeWarning }; + } + previousRaw = raw; + } + return { ok: true, count: lines.length, brokenLine: null, reason: null, sizeWarning }; +} diff --git a/package.json b/package.json index db5d2a5d0..bbffcc3b1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.62.0.0", + "version": "1.63.0.0", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", @@ -27,6 +27,8 @@ "test:e2e:all": "EVALS=1 EVALS_ALL=1 bun test --retry 2 --concurrent --max-concurrency ${EVALS_CONCURRENCY:-15} test/skill-e2e-*.test.ts test/skill-routing-e2e.test.ts test/codex-e2e.test.ts test/gemini-e2e.test.ts", "test:gate": "EVALS=1 EVALS_TIER=gate bun test --retry 2 --concurrent --max-concurrency ${EVALS_CONCURRENCY:-15} test/skill-llm-eval.test.ts test/skill-e2e-*.test.ts test/skill-routing-e2e.test.ts test/codex-e2e.test.ts test/gemini-e2e.test.ts", "test:periodic": "EVALS=1 EVALS_TIER=periodic EVALS_ALL=1 bun test --retry 2 --concurrent --max-concurrency ${EVALS_CONCURRENCY:-15} test/skill-e2e-*.test.ts test/skill-routing-e2e.test.ts test/codex-e2e.test.ts test/gemini-e2e.test.ts", + "test:gate:sharded": "bun run scripts/test-paid-shards.ts --tier gate", + "test:periodic:sharded": "EVALS_ALL=1 bun run scripts/test-paid-shards.ts --tier periodic", "test:codex": "EVALS=1 bun test test/codex-e2e.test.ts", "test:codex:all": "EVALS=1 EVALS_ALL=1 bun test test/codex-e2e.test.ts", "test:gemini": "EVALS=1 bun test test/gemini-e2e.test.ts", @@ -36,8 +38,8 @@ "start": "bun run browse/src/server.ts", "eval:bg": "bin/gstack-detach --label evals --lock gstack-evals --timeout 5400 -- bun run test:evals", "eval:bg:all": "bin/gstack-detach --label evals-all --lock gstack-evals --timeout 7200 -- bun run test:evals:all", - "eval:bg:gate": "bin/gstack-detach --label evals-gate --lock gstack-evals --timeout 3600 -- bun run test:gate", - "eval:bg:periodic": "bin/gstack-detach --label evals-periodic --lock gstack-evals --timeout 5400 -- bun run test:periodic", + "eval:bg:gate": "bin/gstack-detach --label evals-gate --lock gstack-evals --timeout 25200 -- bun run test:gate:sharded", + "eval:bg:periodic": "bin/gstack-detach --label evals-periodic --lock gstack-evals --timeout 28800 -- bun run test:periodic:sharded", "eval:list": "bun run scripts/eval-list.ts", "eval:compare": "bun run scripts/eval-compare.ts", "eval:summary": "bun run scripts/eval-summary.ts", diff --git a/scripts/eval-compare.ts b/scripts/eval-compare.ts index 3cb30d5fb..a7b0dba03 100644 --- a/scripts/eval-compare.ts +++ b/scripts/eval-compare.ts @@ -10,12 +10,13 @@ import * as fs from 'fs'; import * as path from 'path'; -import * as os from 'os'; import { findPreviousRun, compareEvalResults, formatComparison, getProjectEvalDir, + isPartialEval, + listEvalJsonFiles, } from '../test/helpers/eval-store'; import type { EvalResult } from '../test/helpers/eval-store'; @@ -52,25 +53,34 @@ if (args.length === 2) { } beforeFile = prev; } else { - // No args — find two most recent of the same tier - let files: string[]; - try { - files = fs.readdirSync(EVAL_DIR) - .filter(f => f.endsWith('.json')) - .sort() - .reverse(); - } catch { + // No args — find two most recent of the same tier. Scans the flat dir plus + // one level of shards//; in-progress accumulators are never the + // "after" run (comparing against a half-finished run says nothing). + const files = listEvalJsonFiles(EVAL_DIR) + .sort((a, b) => path.basename(b).localeCompare(path.basename(a))); + + if (files.length === 0) { console.log('No eval runs yet. Run: EVALS=1 bun run test:evals'); process.exit(0); } - if (files.length < 2) { console.log('Need at least 2 eval runs to compare. Run evals again.'); process.exit(0); } - // Most recent file - afterFile = path.join(EVAL_DIR, files[0]); + // Most recent finalized file + const latest = files.find(f => { + try { + return !isPartialEval(JSON.parse(fs.readFileSync(f, 'utf-8')), f); + } catch { + return false; + } + }); + if (!latest) { + console.log('No completed eval runs yet. Run: EVALS=1 bun run test:evals'); + process.exit(0); + } + afterFile = latest; const afterResult = loadResult(afterFile); const prev = findPreviousRun(EVAL_DIR, afterResult.tier, afterResult.branch, afterFile); if (!prev) { diff --git a/scripts/eval-list.ts b/scripts/eval-list.ts index 67d3f71a2..bb4ee9a83 100644 --- a/scripts/eval-list.ts +++ b/scripts/eval-list.ts @@ -6,9 +6,7 @@ */ import * as fs from 'fs'; -import * as path from 'path'; -import * as os from 'os'; -import { getProjectEvalDir } from '../test/helpers/eval-store'; +import { getProjectEvalDir, listEvalJsonFiles } from '../test/helpers/eval-store'; const EVAL_DIR = getProjectEvalDir(); @@ -37,14 +35,8 @@ for (let i = 0; i < args.length; i++) { else if (args[i] === '--limit') { limit = parseLimit(args[++i]); } } -// Read eval files -let files: string[]; -try { - files = fs.readdirSync(EVAL_DIR).filter(f => f.endsWith('.json')); -} catch { - console.log('No eval runs yet. Run: EVALS=1 bun run test:evals'); - process.exit(0); -} +// Read eval files (flat dir plus one level of shards//) +const files = listEvalJsonFiles(EVAL_DIR); if (files.length === 0) { console.log('No eval runs yet. Run: EVALS=1 bun run test:evals'); @@ -68,7 +60,7 @@ interface RunSummary { const runs: RunSummary[] = []; for (const file of files) { try { - const data = JSON.parse(fs.readFileSync(path.join(EVAL_DIR, file), 'utf-8')); + const data = JSON.parse(fs.readFileSync(file, 'utf-8')); if (filterBranch && data.branch !== filterBranch) continue; if (filterTier && data.tier !== filterTier) continue; const totalTurns = (data.tests || []).reduce((s: number, t: any) => s + (t.turns_used || 0), 0); diff --git a/scripts/eval-summary.ts b/scripts/eval-summary.ts index fba682c21..a3b8d8bc6 100644 --- a/scripts/eval-summary.ts +++ b/scripts/eval-summary.ts @@ -6,20 +6,13 @@ */ import * as fs from 'fs'; -import * as path from 'path'; -import * as os from 'os'; import type { EvalResult } from '../test/helpers/eval-store'; -import { getProjectEvalDir } from '../test/helpers/eval-store'; +import { getProjectEvalDir, listEvalJsonFiles } from '../test/helpers/eval-store'; const EVAL_DIR = getProjectEvalDir(); -let files: string[]; -try { - files = fs.readdirSync(EVAL_DIR).filter(f => f.endsWith('.json')); -} catch { - console.log('No eval runs yet. Run: EVALS=1 bun run test:evals'); - process.exit(0); -} +// Flat dir plus one level of shards// +const files = listEvalJsonFiles(EVAL_DIR); if (files.length === 0) { console.log('No eval runs yet. Run: EVALS=1 bun run test:evals'); @@ -30,7 +23,7 @@ if (files.length === 0) { const results: EvalResult[] = []; for (const file of files) { try { - results.push(JSON.parse(fs.readFileSync(path.join(EVAL_DIR, file), 'utf-8'))); + results.push(JSON.parse(fs.readFileSync(file, 'utf-8'))); } catch { continue; } } diff --git a/scripts/test-free-shards.ts b/scripts/test-free-shards.ts index 5be84a1f7..8bf98c066 100755 --- a/scripts/test-free-shards.ts +++ b/scripts/test-free-shards.ts @@ -27,22 +27,12 @@ import * as fs from 'fs'; import * as path from 'path'; import { spawnSync } from 'child_process'; +import { isPaidTestFile } from '../test/helpers/paid-test-set'; const ROOT = path.resolve(import.meta.dir, '..'); const TEST_ROOTS = ['browse/test', 'test', 'make-pdf/test'] as const; const TEST_FILE_REGEX = /\.test\.(?:[cm]?[jt]s|tsx|jsx)$/; -// Tests that require API spend, external services, or e2e harnesses. -// These are filtered out before any sharding or curation. -const PAID_EVAL_TESTS = [ - /^browse\/test\/security-review-fullstack\.test\.ts$/, - /^test\/skill-e2e-.*\.test\.ts$/, - /^test\/skill-llm-eval\.test\.ts$/, - /^test\/skill-routing-e2e\.test\.ts$/, - /^test\/codex-e2e\.test\.ts$/, - /^test\/gemini-e2e\.test\.ts$/, -] as const; - // POSIX-only patterns that indicate a test will fail on windows-latest no // matter how the runner shards. Codex's v1.18.0.0 review flagged the first // three as concrete examples in the existing free suite (test/ship-version-sync.test.ts:72, @@ -118,7 +108,7 @@ export function normalizeRelativePath(filePath: string): string { export function isFreeTestFile(relativePath: string): boolean { const normalized = normalizeRelativePath(relativePath); if (!TEST_FILE_REGEX.test(normalized)) return false; - return !PAID_EVAL_TESTS.some(pattern => pattern.test(normalized)); + return !isPaidTestFile(normalized); } /** diff --git a/scripts/test-paid-shards.ts b/scripts/test-paid-shards.ts new file mode 100644 index 000000000..fad74e868 --- /dev/null +++ b/scripts/test-paid-shards.ts @@ -0,0 +1,479 @@ +#!/usr/bin/env bun +/** + * test-paid-shards — enumerate, shard, and run the paid (gate/periodic) tier. + * + * The single-process `test:gate` fan-out has never completed a run: one wedged + * or spinning file takes the whole tier down, and an in-process `--timeout` + * cannot save it because a spinning main thread never fires a timer. This + * runner applies the free tier's proven fix — one Bun process per shard — plus + * the two things the paid tier additionally needs: + * + * - an EXTERNAL wall-clock timeout that kills the shard's process GROUP, and + * - an aggregate that distinguishes failed from timed-out from never-started, + * so 26% execution can never again look like a pass. + * + * Why not Bun 1.3.13's native `--shard` / isolated runs? Three gaps, each one + * fatal for this tier: + * 1. No detached-process-group SIGKILL. Paid tests spawn `claude` / `codex` + * PTY grandchildren; when a shard hangs, in-process isolation kills the + * Bun worker but the grandchildren survive and burn cores for hours. + * 2. No never-started taxonomy. A run that aborts partway reports only what + * executed — the shards that never ran are invisible, which is exactly + * the 26%-execution-looks-like-a-pass bug. + * 3. No per-shard env / eval dir. Each shard needs its own GSTACK_EVAL_DIR + * so eval baselines are per-test-file instead of last-flush-wins. + * + * Worst-case wall clock (all shards hit the 30min timeout, 4 parallel jobs): + * gate tier is 49 shards × 30min / 4 jobs ≈ 6.2h; periodic is 59 shards ≈ 7.4h. + * The eval:bg:* detach timeouts (25200s / 28800s) are sized against these. + * + * Enumeration matches package.json's `test:gate` globs (via the shared + * test/helpers/paid-test-set.ts) and honors EVALS_TIER against the E2E_TIERS + * map in test/helpers/touchfiles.ts. Output classification reuses + * scripts/test-strict-output.ts rather than reimplementing it. + * + * Parallelism now lives ACROSS shards (--jobs), not inside one Bun process, so + * each shard runs its own file sequentially and can be killed independently. + * + * Usage: + * bun run scripts/test-paid-shards.ts --list # shard plan only + * bun run scripts/test-paid-shards.ts --tier gate # run gate tier + * bun run scripts/test-paid-shards.ts --timeout 600 --jobs 2 + */ + +import { spawn, type ChildProcess } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { normalizeRelativePath } from './test-free-shards'; +import { + BunTestOutputClassifier, + exactTestFileSelectors, + forwardAndClassify, + installChildSignalForwarding, + strictTestExitCode, +} from './test-strict-output'; +import { PAID_TEST_GLOBS, isPaidTestFile } from '../test/helpers/paid-test-set'; +import { getProjectEvalDir } from '../test/helpers/eval-store'; + +export { PAID_TEST_GLOBS, isPaidTestFile }; + +const ROOT = path.resolve(import.meta.dir, '..'); + +export type PaidTier = 'gate' | 'periodic'; + +export const DEFAULT_TIER: PaidTier = 'gate'; +export const DEFAULT_SHARD_TIMEOUT_MS = 30 * 60_000; +export const DEFAULT_MAX_FILES_PER_SHARD = 1; +export const DEFAULT_JOBS = 4; + +export function collectPaidTestFiles(rootDir = ROOT): string[] { + const testDir = path.join(rootDir, 'test'); + if (!fs.existsSync(testDir)) return []; + return fs.readdirSync(testDir) + .map((name) => `test/${name}`) + .filter(isPaidTestFile) + .sort(); +} + +export interface TierClassification { + included: boolean; + reason: string; +} + +/** + * Decide whether a paid test file has anything to run in `tier`. + * + * Per-TEST tier filtering already happens at runtime: test/helpers/e2e-helpers.ts + * intersects the selected tests with E2E_TIERS whenever EVALS_TIER is set, and + * this runner passes EVALS_TIER down to every shard. So this file-level pass is + * only an optimization — skipping a file merely saves one near-instant shard. + * + * Exclusion is the dangerous direction (a wrongly-skipped gate test is exactly + * the invisible-non-execution bug this runner exists to kill), so the only + * exclusion evidence accepted is an explicit whole-file `EVALS_TIER === ''` + * guard. Inferring a file's tier from which E2E_TIERS names appear in its source + * is guesswork that silently drops real work: short keys like 'retro' match + * unrelated strings, and LLM-judge tests are keyed off LLM_JUDGE_TOUCHFILES and + * carry no E2E_TIERS name at all. Everything without an explicit other-tier + * guard runs and self-skips. + */ +export function classifyPaidTestFile(source: string, tier: PaidTier): TierClassification { + const other: PaidTier = tier === 'gate' ? 'periodic' : 'gate'; + const declares = (candidate: PaidTier) => + new RegExp(`EVALS_TIER\\s*===\\s*['"\`]${candidate}['"\`]`).test(source); + + if (declares(tier)) return { included: true, reason: `declares EVALS_TIER === '${tier}'` }; + if (declares(other)) return { included: false, reason: `declares EVALS_TIER === '${other}' only` }; + return { included: true, reason: 'no whole-file tier guard — runtime E2E_TIERS filter decides' }; +} + +export interface TierSelection { + selected: string[]; + excluded: Array<{ file: string; reason: string }>; +} + +export function selectPaidTestFiles(files: string[], tier: PaidTier, rootDir = ROOT): TierSelection { + const selected: string[] = []; + const excluded: Array<{ file: string; reason: string }> = []; + for (const file of files) { + const source = fs.readFileSync(path.join(rootDir, file), 'utf8'); + const classification = classifyPaidTestFile(source, tier); + if (classification.included) selected.push(file); + else excluded.push({ file, reason: classification.reason }); + } + return { selected, excluded }; +} + +export function planPaidShards( + files: string[], + options: { maxFilesPerShard?: number } = {}, +): string[][] { + const size = Math.max(1, options.maxFilesPerShard ?? DEFAULT_MAX_FILES_PER_SHARD); + const unique = [...new Set(files.map(normalizeRelativePath))].sort(); + const shards: string[][] = []; + for (let index = 0; index < unique.length; index += size) shards.push(unique.slice(index, index + size)); + return shards; +} + +export function buildPaidShardArgs(files: string[], timeoutMs: number): string[] { + return ['test', ...files, '--retry', '2', `--timeout=${timeoutMs}`]; +} + +/** + * Stable per-shard eval-dir slug: test filename sans extension, sanitized. + * Stable across runs so each shard baselines against its own prior run. + */ +export function shardSlug(files: string[]): string { + return files + .map((file) => path.basename(normalizeRelativePath(file)).replace(/\.test\.(?:[cm]?[jt]s|tsx|jsx)$/, '')) + .join('+') + .replace(/[^a-zA-Z0-9._+-]/g, '-'); +} + +export type ShardStatus = 'passed' | 'failed' | 'timed-out' | 'never-started'; + +export interface ShardOutcome { + shard: number; + files: string[]; + status: ShardStatus; + exitCode: number | null; + elapsedMs: number; + groupPid: number | null; +} + +export interface ShardCommand { + command: string; + args: string[]; +} + +export interface RunShardsOptions { + timeoutMs?: number; + jobs?: number; + rootDir?: string; + env?: NodeJS.ProcessEnv; + /** When set, each shard child gets GSTACK_EVAL_DIR=/shards//. */ + evalDirBase?: string; + /** Override the spawned command. Tests inject fake slow/spinning commands. */ + commandFor?: (files: string[]) => ShardCommand; + log?: (line: string) => void; +} + +/** + * SIGKILL the shard's whole process group. Orphaned grandchildren (browsers, + * claude sessions) are how a stalled run once burned a core for 15.7 hours. + */ +function killProcessGroup(child: ChildProcess, signal: NodeJS.Signals): void { + if (process.platform === 'win32' || typeof child.pid !== 'number') { + child.kill(signal); + return; + } + try { + process.kill(-child.pid, signal); + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + if (code === 'ESRCH') return; // group already gone + if (code !== 'EPERM') throw err; + // Observed on macOS after a SIGKILLed group is reaped: signalling the + // now-empty group id returns EPERM, not ESRCH. Throwing here loses the + // shard's real outcome (a timeout gets recorded as a failure) and, from + // the timeout timer, leaves the shard promise unsettled — a hang, which + // is the exact failure class this runner exists to kill. Fall back to the + // direct pid so a genuinely-live child is still signalled. + try { + child.kill(signal); + } catch { + // Best-effort reap: nothing actionable is left if this fails too. + } + } +} + +export async function runPaidShard( + files: string[], + shardNumber: number, + totalShards: number, + options: RunShardsOptions = {}, +): Promise { + if (files.length === 0) throw new Error('Cannot run an empty paid-test shard.'); + const rootDir = options.rootDir ?? ROOT; + const timeoutMs = options.timeoutMs ?? DEFAULT_SHARD_TIMEOUT_MS; + const streamLive = (options.jobs ?? DEFAULT_JOBS) === 1; + const log = options.log ?? ((line: string) => console.log(line)); + const label = `[test:paid] shard ${shardNumber}/${totalShards}`; + + const { command, args } = options.commandFor + ? options.commandFor(files) + : { + command: process.execPath, + args: buildPaidShardArgs(exactTestFileSelectors(files, rootDir), timeoutMs), + }; + + const env = { ...(options.env ?? process.env) }; + if (options.evalDirBase) { + env.GSTACK_EVAL_DIR = path.join(options.evalDirBase, 'shards', shardSlug(files)); + } + + const startedAt = Date.now(); + log(`${label} START ${files.join(' ')} (timeout ${Math.round(timeoutMs / 1000)}s)`); + + const child = spawn(command, args, { + cwd: rootDir, + env, + stdio: ['ignore', 'pipe', 'pipe'], + detached: process.platform !== 'win32', + windowsHide: true, + }); + const groupPid = child.pid ?? null; + // Group-kill on parent SIGINT/SIGTERM too, not just on timeout. + const forwarding = installChildSignalForwarding({ + kill: (signal?: NodeJS.Signals | number) => { + killProcessGroup(child, (signal as NodeJS.Signals) ?? 'SIGTERM'); + return true; + }, + }); + + const classifier = new BunTestOutputClassifier(); + const buffered: Buffer[] = []; + const sink = (destination: NodeJS.WriteStream): NodeJS.WriteStream => (streamLive + ? destination + : ({ write: (chunk: Buffer | string) => buffered.push(Buffer.from(chunk)) } as unknown as NodeJS.WriteStream)); + + let timedOut = false; + const killTimer = setTimeout(() => { + timedOut = true; + killProcessGroup(child, 'SIGKILL'); + }, timeoutMs); + + let exitCode: number | null = null; + try { + const streams: Array> = []; + if (child.stdout) streams.push(forwardAndClassify(child.stdout, sink(process.stdout), classifier)); + if (child.stderr) streams.push(forwardAndClassify(child.stderr, sink(process.stderr), classifier)); + exitCode = await new Promise((resolve, reject) => { + child.once('error', reject); + child.once('close', (code) => resolve(code)); + }); + await Promise.all(streams); + } finally { + clearTimeout(killTimer); + forwarding.dispose(); + // Reap survivors of this shard even on the clean path. + killProcessGroup(child, 'SIGKILL'); + } + + const summary = classifier.end(); + if (!streamLive && buffered.length > 0) process.stdout.write(Buffer.concat(buffered)); + + // Pass expectedFiles so a shard whose bun child ran fewer files than planned + // (or zero, all self-skipped) with exit 0 is NOT recorded 'passed' — the + // invisible-non-execution class this runner exists to kill. bun prints + // "Ran N tests across M files" with M = selected files even when every test + // self-skips, so terminalFileCounts must include files.length. Only enforced + // on the real bun path: an injected commandFor (tests) isn't bun and emits no + // terminal summary, so there's no file count to check against. + const expectedFiles = options.commandFor ? undefined : files.length; + const status: ShardStatus = timedOut + ? 'timed-out' + : strictTestExitCode(exitCode ?? 1, summary, expectedFiles) === 0 ? 'passed' : 'failed'; + const elapsedMs = Date.now() - startedAt; + log(`${label} ${status.toUpperCase()} in ${Math.round(elapsedMs / 1000)}s (exit ${exitCode ?? 'signal'})`); + + return { shard: shardNumber, files, status, exitCode, elapsedMs, groupPid }; +} + +export interface RunSummary { + total: number; + executed: number; + passed: number; + failed: number; + timedOut: number; + neverStarted: number; + outcomes: ShardOutcome[]; +} + +export function summarize(outcomes: ShardOutcome[]): RunSummary { + const count = (status: ShardStatus) => outcomes.filter((o) => o.status === status).length; + return { + total: outcomes.length, + executed: outcomes.length - count('never-started'), + passed: count('passed'), + failed: count('failed'), + timedOut: count('timed-out'), + neverStarted: count('never-started'), + outcomes, + }; +} + +/** Run every shard in its own process. A timeout or failure never aborts the run. */ +export async function runPaidShards( + shards: string[][], + options: RunShardsOptions = {}, +): Promise { + const jobs = Math.max(1, options.jobs ?? DEFAULT_JOBS); + const outcomes: ShardOutcome[] = shards.map((files, index) => ({ + shard: index + 1, + files, + status: 'never-started', + exitCode: null, + elapsedMs: 0, + groupPid: null, + })); + + let next = 0; + const worker = async (): Promise => { + while (true) { + const index = next; + next += 1; + if (index >= shards.length) return; + try { + outcomes[index] = await runPaidShard(shards[index], index + 1, shards.length, { ...options, jobs }); + } catch (error) { + outcomes[index] = { + shard: index + 1, + files: shards[index], + status: 'failed', + exitCode: null, + elapsedMs: 0, + groupPid: null, + }; + console.error(`[test:paid] shard ${index + 1} could not run: ${error instanceof Error ? error.message : String(error)}`); + } + } + }; + + await Promise.all(Array.from({ length: Math.min(jobs, shards.length) }, worker)); + return summarize(outcomes); +} + +export function formatSummary(summary: RunSummary): string[] { + const lines = [ + '', + `[test:paid] ${summary.executed}/${summary.total} shards executed — ` + + `${summary.passed} passed, ${summary.failed} failed, ` + + `${summary.timedOut} timed out, ${summary.neverStarted} never started`, + ]; + for (const outcome of summary.outcomes) { + lines.push( + ` ${outcome.status.padEnd(13)} ${String(Math.round(outcome.elapsedMs / 1000)).padStart(5)}s ` + + outcome.files.join(' '), + ); + } + return lines; +} + +type CliOptions = { + tier: PaidTier; + listOnly: boolean; + timeoutMs: number; + jobs: number; + maxFilesPerShard: number; +}; + +function parsePositiveInt(value: string | undefined, flag: string): number { + const parsed = Number.parseInt(value ?? '', 10); + if (!Number.isInteger(parsed) || parsed <= 0) throw new Error(`${flag} needs a positive integer. Received: ${value}`); + return parsed; +} + +function validatedTier(value: string | undefined, source: string): PaidTier { + if (value === undefined || value === '') return DEFAULT_TIER; + // A typo'd EVALS_TIER (e.g. 'e2e', the tier string eval-store uses) would + // otherwise cast through unchecked, match nothing in the runtime E2E_TIERS + // filter, self-skip every test, and exit 0 with all shards 'passed' — the + // exact 0%-execution-looks-like-a-pass class this runner exists to kill. + if (value !== 'gate' && value !== 'periodic') { + throw new Error(`${source} must be gate or periodic. Received: ${value}`); + } + return value; +} + +export function parseCliOptions(argv: string[], env: NodeJS.ProcessEnv = process.env): CliOptions { + const options: CliOptions = { + tier: validatedTier(env.EVALS_TIER, 'EVALS_TIER'), + listOnly: false, + timeoutMs: env.EVALS_SHARD_TIMEOUT_MS + ? parsePositiveInt(env.EVALS_SHARD_TIMEOUT_MS, 'EVALS_SHARD_TIMEOUT_MS') + : DEFAULT_SHARD_TIMEOUT_MS, + jobs: env.EVALS_CONCURRENCY ? parsePositiveInt(env.EVALS_CONCURRENCY, 'EVALS_CONCURRENCY') : DEFAULT_JOBS, + maxFilesPerShard: DEFAULT_MAX_FILES_PER_SHARD, + }; + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === '--list') { options.listOnly = true; continue; } + if (arg === '--tier') { + const value = argv[index += 1]; + if (value !== 'gate' && value !== 'periodic') throw new Error(`--tier must be gate or periodic. Received: ${value}`); + options.tier = value; + continue; + } + if (arg === '--timeout') { options.timeoutMs = parsePositiveInt(argv[index += 1], '--timeout') * 1000; continue; } + if (arg === '--jobs') { options.jobs = parsePositiveInt(argv[index += 1], '--jobs'); continue; } + if (arg === '--files-per-shard') { options.maxFilesPerShard = parsePositiveInt(argv[index += 1], '--files-per-shard'); continue; } + throw new Error(`Unknown argument: ${arg}`); + } + return options; +} + +async function main(): Promise { + const options = parseCliOptions(process.argv.slice(2)); + const discovered = collectPaidTestFiles(); + if (discovered.length === 0) throw new Error('No paid test files were discovered.'); + + const { selected, excluded } = selectPaidTestFiles(discovered, options.tier); + const shards = planPaidShards(selected, { maxFilesPerShard: options.maxFilesPerShard }); + console.log( + `[test:paid] tier=${options.tier}: ${selected.length}/${discovered.length} files, ` + + `${shards.length} shards, jobs=${options.jobs}, timeout=${Math.round(options.timeoutMs / 1000)}s`, + ); + + if (options.listOnly) { + for (let index = 0; index < shards.length; index += 1) { + console.log(` shard ${index + 1}/${shards.length}: ${shards[index].join(' ')}`); + } + if (excluded.length > 0) { + console.log(`\nExcluded (${excluded.length}):`); + for (const { file, reason } of excluded) console.log(` - ${file} [${reason}]`); + } + return 0; + } + + const summary = await runPaidShards(shards, { + // Tier reaches the children only via EVALS_TIER below; the runtime + // E2E_TIERS filter inside each child is the real selection mechanism. + timeoutMs: options.timeoutMs, + jobs: options.jobs, + env: { ...process.env, EVALS: '1', EVALS_TIER: options.tier }, + evalDirBase: process.env.GSTACK_EVAL_DIR || getProjectEvalDir(), + }); + for (const line of formatSummary(summary)) console.log(line); + return summary.passed === summary.total ? 0 : 1; +} + +if (import.meta.main) { + try { + process.exitCode = await main(); + } catch (error) { + console.error(`[test:paid] ${error instanceof Error ? error.message : String(error)}`); + process.exitCode = 1; + } +} diff --git a/scripts/test-strict-output.ts b/scripts/test-strict-output.ts new file mode 100644 index 000000000..16c073c0a --- /dev/null +++ b/scripts/test-strict-output.ts @@ -0,0 +1,200 @@ +/** + * Strict Bun-test output classification + child lifecycle helpers. + * + * Works around a Bun test runner bug where failures can be printed even though + * the child exits successfully: output is forwarded byte-for-byte as it + * arrives, and only complete Bun result lines and terminal summaries are + * classified. `strictTestExitCode` then refuses to trust a zero exit when the + * output shows failures (or when fewer files ran than expected). + * + * Shared by the sharded paid-tier runner (scripts/test-paid-shards.ts) and any + * future strict wrapper around `bun test`. + */ + +import { type ChildProcess } from 'node:child_process'; +import { StringDecoder } from 'node:string_decoder'; +import * as path from 'node:path'; + +const ROOT = path.resolve(import.meta.dir, '..'); +const ANSI_ESCAPE = /\u001B\[[0-?]*[ -/]*[@-~]/g; +const BUN_FAIL_RESULT = /^\(fail\) .+ \[(?:\d+(?:\.\d+)?)(?:ns|us|µs|ms|s)\]$/; +const BUN_BETWEEN_TESTS_ERROR = '# Unhandled error between tests'; +const BUN_TERMINAL_SUMMARY = /^Ran \d+ tests? across (\d+) files?\. \[(?:\d+(?:\.\d+)?)(?:ns|us|µs|ms|s)\]$/; + +export type BunTestOutputFinding = 'failed-test' | 'unhandled-between-tests'; + +export interface BunTestOutputSummary { + failedTests: number; + unhandledBetweenTests: number; + terminalFileCounts: number[]; +} + +export type ForwardedTerminationSignal = 'SIGINT' | 'SIGTERM'; + +export interface TerminationSignalSource { + on(event: string, listener: () => void): unknown; + off(event: string, listener: () => void): unknown; +} + +export interface TerminationTimerApi { + schedule(callback: () => void, delayMs: number): unknown; + cancel(handle: unknown): void; +} + +export interface ChildSignalForwarding { + readonly receivedSignal: ForwardedTerminationSignal | null; + dispose(): void; +} + +const DEFAULT_TERMINATION_TIMER: TerminationTimerApi = { + schedule: (callback, delayMs) => setTimeout(callback, delayMs), + cancel: (handle) => clearTimeout(handle as ReturnType), +}; + +/** + * Bind one active child to the parent's termination lifecycle. SIGINT and + * SIGTERM get a grace period so Bun can clean up; a repeated signal, timeout, + * or synchronous parent exit uses SIGKILL so the child cannot be orphaned. + */ +export function installChildSignalForwarding( + child: Pick, + source: TerminationSignalSource = process, + timer: TerminationTimerApi = DEFAULT_TERMINATION_TIMER, + graceMs = 5_000, +): ChildSignalForwarding { + let receivedSignal: ForwardedTerminationSignal | null = null; + let forceTimer: unknown = null; + let disposed = false; + + const forward = (signal: ForwardedTerminationSignal): void => { + if (disposed) return; + if (receivedSignal !== null) { + child.kill('SIGKILL'); + return; + } + receivedSignal = signal; + child.kill(signal); + forceTimer = timer.schedule(() => { + forceTimer = null; + child.kill('SIGKILL'); + }, graceMs); + }; + const onSigint = () => forward('SIGINT'); + const onSigterm = () => forward('SIGTERM'); + const onExit = () => { child.kill('SIGKILL'); }; + + source.on('SIGINT', onSigint); + source.on('SIGTERM', onSigterm); + source.on('exit', onExit); + + return { + get receivedSignal() { + return receivedSignal; + }, + dispose() { + if (disposed) return; + disposed = true; + source.off('SIGINT', onSigint); + source.off('SIGTERM', onSigterm); + source.off('exit', onExit); + if (forceTimer !== null) timer.cancel(forceTimer); + forceTimer = null; + }, + }; +} + +export function classifyBunTestOutputLine(rawLine: string): BunTestOutputFinding | null { + const line = rawLine.replace(ANSI_ESCAPE, '').replace(/\r$/, ''); + if (BUN_FAIL_RESULT.test(line)) return 'failed-test'; + if (line === BUN_BETWEEN_TESTS_ERROR) return 'unhandled-between-tests'; + return null; +} + +export function parseBunTerminalSummaryLine(rawLine: string): number | null { + const line = rawLine.replace(ANSI_ESCAPE, '').replace(/\r$/, ''); + const match = BUN_TERMINAL_SUMMARY.exec(line); + return match ? Number.parseInt(match[1], 10) : null; +} + +/** Incrementally classifies output without assuming process chunks align to lines. */ +export class BunTestOutputClassifier { + private readonly decoder = new StringDecoder('utf8'); + private pending = ''; + private failedTests = 0; + private unhandledBetweenTests = 0; + private terminalFileCounts: number[] = []; + + write(chunk: Uint8Array | string): void { + this.pending += typeof chunk === 'string' + ? chunk + : this.decoder.write(Buffer.from(chunk)); + this.consumeCompleteLines(); + } + + end(): BunTestOutputSummary { + this.pending += this.decoder.end(); + if (this.pending.length > 0) this.classify(this.pending); + this.pending = ''; + return this.summary(); + } + + summary(): BunTestOutputSummary { + return { + failedTests: this.failedTests, + unhandledBetweenTests: this.unhandledBetweenTests, + terminalFileCounts: [...this.terminalFileCounts], + }; + } + + private consumeCompleteLines(): void { + let newline = this.pending.indexOf('\n'); + while (newline !== -1) { + this.classify(this.pending.slice(0, newline)); + this.pending = this.pending.slice(newline + 1); + newline = this.pending.indexOf('\n'); + } + } + + private classify(line: string): void { + const finding = classifyBunTestOutputLine(line); + if (finding === 'failed-test') this.failedTests += 1; + if (finding === 'unhandled-between-tests') this.unhandledBetweenTests += 1; + const terminalFileCount = parseBunTerminalSummaryLine(line); + if (terminalFileCount !== null) this.terminalFileCounts.push(terminalFileCount); + } +} + +export function strictTestExitCode( + childExitCode: number, + summary: BunTestOutputSummary, + expectedFiles?: number, +): number { + if (childExitCode !== 0) return childExitCode; + if (summary.failedTests > 0 || summary.unhandledBetweenTests > 0) return 1; + if (expectedFiles !== undefined && !summary.terminalFileCounts.includes(expectedFiles)) return 1; + return 0; +} + +/** + * Bun treats positional test paths as substring filters. Resolve every + * canonical relative path before spawning so `test/foo.test.ts` cannot also + * select `browse/test/foo.test.ts`. + */ +export function exactTestFileSelectors(files: string[], rootDir = ROOT): string[] { + return files.map((file) => path.isAbsolute(file) ? path.normalize(file) : path.resolve(rootDir, file)); +} + +export function forwardAndClassify( + stream: NodeJS.ReadableStream, + destination: NodeJS.WriteStream, + classifier: BunTestOutputClassifier, +): Promise { + return new Promise((resolve, reject) => { + stream.on('data', (chunk: Buffer | string) => { + classifier.write(chunk); + destination.write(chunk); + }); + stream.on('end', resolve); + stream.on('error', reject); + }); +} diff --git a/test/brain-sync.test.ts b/test/brain-sync.test.ts index 2e7c121d2..a4587f806 100644 --- a/test/brain-sync.test.ts +++ b/test/brain-sync.test.ts @@ -34,7 +34,11 @@ function run(argv: string[], opts: { env?: Record; input?: strin const bin = argv[0]; const full = bin.startsWith('/') ? bin : path.join(BIN, bin); const res = spawnSync(full, argv.slice(1), { - env: { ...process.env, GSTACK_HOME: tmpHome, ...(opts.env || {}) }, + // HOME is overridden too: gstack-artifacts-init writes + // $HOME/.gstack-artifacts-remote.txt (plain $HOME, not GSTACK_HOME), so + // without this every free-suite run clobbers the operator's real + // artifacts-remote pointer. Keep it inside tmpHome, which afterEach removes. + env: { ...process.env, HOME: tmpHome, GSTACK_HOME: tmpHome, ...(opts.env || {}) }, encoding: 'utf-8', input: opts.input, cwd: ROOT, @@ -56,13 +60,18 @@ beforeEach(() => { afterEach(() => { fs.rmSync(tmpHome, { recursive: true, force: true }); fs.rmSync(bareRemote, { recursive: true, force: true }); - // Clean up any remote-helper file init may have written. - const remoteFile = path.join(os.homedir(), '.gstack-brain-remote.txt'); - // Only remove if it points at OUR bare remote (don't clobber a real user file). - try { - const contents = fs.readFileSync(remoteFile, 'utf-8').trim(); - if (contents === bareRemote) fs.unlinkSync(remoteFile); - } catch {} + // Clean up any remote-helper file init may have written. run() now pins + // HOME to tmpHome so these land inside the removed temp dir, but scrub the + // real home too as defense in depth — and cover BOTH the legacy brain-remote + // name and the current artifacts-remote name (init writes the latter). + for (const name of ['.gstack-brain-remote.txt', '.gstack-artifacts-remote.txt']) { + const remoteFile = path.join(os.homedir(), name); + // Only remove if it points at OUR bare remote (don't clobber a real user file). + try { + const contents = fs.readFileSync(remoteFile, 'utf-8').trim(); + if (contents === bareRemote) fs.unlinkSync(remoteFile); + } catch {} + } }); // --------------------------------------------------------------- @@ -330,6 +339,72 @@ describe('gstack-brain-sync secret scan', () => { }); }); +// --------------------------------------------------------------- +// Egress receipt gate: receipt-before-commit, queue intact on refusal +// --------------------------------------------------------------- +describe('gstack-brain-sync egress receipt gate', () => { + test('refused receipt leaves the queue intact, makes no commit, and next run retries', () => { + if (process.platform === 'win32' || process.getuid?.() === 0) return; // chmod is advisory there + run(['gstack-artifacts-init', '--remote', bareRemote]); + run(['gstack-config', 'set', 'artifacts_sync_mode', 'full']); + fs.mkdirSync(path.join(tmpHome, 'projects', 'p'), { recursive: true }); + fs.writeFileSync(path.join(tmpHome, 'projects/p/learnings.jsonl'), + '{"skill":"x","insight":"y","ts":"2026-04-22T10:00:00Z"}\n'); + run(['gstack-brain-enqueue', 'projects/p/learnings.jsonl']); + const commitsBefore = git(['rev-list', '--count', 'HEAD']).stdout.trim(); + + // Make the receipt unwritable: security dir exists but is read-only. + // (artifacts-init may have created it already — mkdirSync's mode is a + // no-op on an existing dir, so chmod explicitly.) + fs.mkdirSync(path.join(tmpHome, 'security'), { recursive: true }); + fs.chmodSync(path.join(tmpHome, 'security'), 0o500); + try { + const refused = run(['gstack-brain-sync', '--once']); + expect(refused.status).toBe(1); + // DX contract: problem + cause + fix, plain language. + expect(refused.stderr).toContain('NOT sent'); + expect(refused.stderr).toContain('EGRESS_RECEIPT_FAILED'); + expect(refused.stderr).toContain('Fix: chmod -R u+w'); + expect(refused.stderr).toContain('ATTEMPTS to send off-machine'); + // Queue intact (receipt is written BEFORE the commit consumes it). + const queue = fs.readFileSync(path.join(tmpHome, '.brain-queue.jsonl'), 'utf-8'); + expect(queue).toContain('projects/p/learnings.jsonl'); + // No local commit was created. + expect(git(['rev-list', '--count', 'HEAD']).stdout.trim()).toBe(commitsBefore); + // Nothing reached the remote. + const remoteLog = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + expect(remoteLog.stdout).not.toMatch(/sync: 1 file/); + const status = JSON.parse(fs.readFileSync(path.join(tmpHome, '.brain-sync-status.json'), 'utf-8')); + expect(status.status).toBe('push_failed'); + expect(status.message).toContain('EGRESS_RECEIPT_FAILED'); + } finally { + fs.chmodSync(path.join(tmpHome, 'security'), 0o700); + } + + // Next run (ledger writable again) drains the intact queue and pushes. + const retry = run(['gstack-brain-sync', '--once']); + expect(retry.status).toBe(0); + const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + expect(log.stdout).toMatch(/sync: 1 file/); + }); + + test('successful push writes a git-class receipt before the send', () => { + run(['gstack-artifacts-init', '--remote', bareRemote]); + run(['gstack-config', 'set', 'artifacts_sync_mode', 'full']); + fs.mkdirSync(path.join(tmpHome, 'projects', 'p'), { recursive: true }); + fs.writeFileSync(path.join(tmpHome, 'projects/p/learnings.jsonl'), + '{"skill":"x","insight":"y","ts":"2026-04-22T10:00:00Z"}\n'); + run(['gstack-brain-enqueue', 'projects/p/learnings.jsonl']); + const r = run(['gstack-brain-sync', '--once']); + expect(r.status).toBe(0); + const ledger = fs.readFileSync(path.join(tmpHome, 'security', 'egress.jsonl'), 'utf-8'); + const records = ledger.trim().split('\n').map((l) => JSON.parse(l)); + const pushReceipt = records.find((rec) => rec.sink === 'brain-sync' && rec.payload_class === 'curated-memory-git-push'); + expect(pushReceipt).toBeTruthy(); + expect(pushReceipt.sha256).toBeNull(); // git owns the bytes + }); +}); + // --------------------------------------------------------------- // Uninstall preserves user data // --------------------------------------------------------------- diff --git a/test/catalog-budget.test.ts b/test/catalog-budget.test.ts new file mode 100644 index 000000000..b2ad347b5 --- /dev/null +++ b/test/catalog-budget.test.ts @@ -0,0 +1,122 @@ +import { describe, expect, test } from 'bun:test'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { skillCensus } from './helpers/skill-census'; + +/** + * Aggregate discovery-surface budget: the sum of every skill's frontmatter + * `name` + `description` is what EVERY host loads at discovery, every session. + * + * This is the missing enforcement layer over the existing catalog-trim + * mechanism: `applyCatalogTrim` in scripts/gen-skill-docs.ts (~line 865) + * shapes each description, and the 160KB per-file warn (~line 1015) covers + * BODY size — neither caps the aggregate frontmatter the catalog is made of. + * + * Import-free by design: parses skills' SKILL.md frontmatter directly. Do not + * import gen-skill-docs internals here — this test must survive generator + * refactors. + * + * Budget derivation (re-derive it, do not trust the number): + * ref this commit + * method for each authored skill (test/helpers/skill-census.ts + * authoredSkills — symlink-deduped, root router excluded) plus the + * root router's `_gstack-command` alias frontmatter as one separate + * line item, run parseFrontmatter() below and sum + * Buffer.byteLength(name) + Buffer.byteLength(description); + * token-equivalents = ceil(bytes / 4). + * result 53 authored skills = 4,371 bytes (1,093 token-equivalents); + * + root router alias 49 bytes = 4,420 bytes total + * = 1,105 token-equivalents (measured 2026-08-12) + * Ceiling is 1,150 token-equivalents (4,600 bytes), so headroom is 180 bytes + * (~4%). Dominant skill: design-consultation at 229 bytes name+description. + */ +const CATALOG_BUDGET_TOKEN_EQUIVALENTS = 1_150; + +// Largest today: design-consultation at 229 bytes. A description that needs +// more than 260 bytes is a body paragraph, not a catalog entry. +const PER_SKILL_BYTE_CAP = 260; + +const RATCHET_PROTOCOL = + 'Adding a skill? Re-measure with: bun test test/catalog-budget.test.ts ' + + '(the failure prints the new total). Update CATALOG_BUDGET_TOKEN_EQUIVALENTS ' + + 'AND the derivation comment (ref/date/value/which skill moved it) in the ' + + 'SAME commit. Growing an existing description? Trim it instead — the ' + + 'catalog is what every host loads at discovery, every session.'; + +const ROOT = join(import.meta.dir, '..'); + +function parseFrontmatter(body: string): { name: string; description: string } { + const name = body.match(/^name:\s*(.+)$/m)?.[1]?.trim() ?? ''; + // Folded block scalar (description: >-) with two-space-indented continuation + // lines, falling back to a single-line description. + const folded = body.match(/^description:\s*>-?\r?\n((?: .*\r?\n)+)/m)?.[1]; + const description = folded + ? folded.split(/\r?\n/).map((line) => line.trim()).filter(Boolean).join(' ') + : body.match(/^description:\s*(?!>-?\s*$)(.+)$/m)?.[1]?.trim() ?? ''; + return { name, description }; +} + +interface CatalogEntry { + skill: string; + name: string; + description: string; + bytes: number; +} + +function catalogEntries(): CatalogEntry[] { + const entries: CatalogEntry[] = []; + for (const skill of skillCensus(ROOT).authoredSkills) { + const body = readFileSync(join(ROOT, skill, 'SKILL.md'), 'utf8'); + const { name, description } = parseFrontmatter(body); + entries.push({ + skill, + name, + description, + bytes: Buffer.byteLength(name) + Buffer.byteLength(description), + }); + } + // The root SKILL.md is a router, registered by setup as the + // `_gstack-command` alias — not an authored skill, but its frontmatter + // still ships in the catalog, so it counts as one line item. + const router = parseFrontmatter(readFileSync(join(ROOT, 'SKILL.md'), 'utf8')); + if (router.name && router.description) { + entries.push({ + skill: '(root router)', + name: router.name, + description: router.description, + bytes: Buffer.byteLength(router.name) + Buffer.byteLength(router.description), + }); + } + return entries; +} + +describe('catalog discovery-surface budget', () => { + test(`aggregate frontmatter stays within ${CATALOG_BUDGET_TOKEN_EQUIVALENTS} token-equivalents`, () => { + const entries = catalogEntries(); + const totalBytes = entries.reduce((sum, e) => sum + e.bytes, 0); + const estimatedTokens = Math.ceil(totalBytes / 4); + const delta = estimatedTokens - CATALOG_BUDGET_TOKEN_EQUIVALENTS; + expect( + estimatedTokens, + `Catalog is ${estimatedTokens} token-equivalents (${totalBytes} bytes), ` + + `${delta} over the ${CATALOG_BUDGET_TOKEN_EQUIVALENTS} budget. ${RATCHET_PROTOCOL}` + ).toBeLessThanOrEqual(CATALOG_BUDGET_TOKEN_EQUIVALENTS); + }); + + test(`every skill's name + description stays under ${PER_SKILL_BYTE_CAP} bytes`, () => { + for (const entry of catalogEntries()) { + expect( + entry.bytes, + `${entry.skill}: name + description is ${entry.bytes} bytes, ` + + `${entry.bytes - PER_SKILL_BYTE_CAP} over the ${PER_SKILL_BYTE_CAP}-byte ` + + `per-skill cap. ${RATCHET_PROTOCOL}` + ).toBeLessThanOrEqual(PER_SKILL_BYTE_CAP); + } + }); + + test('every skill has a non-empty description', () => { + for (const entry of catalogEntries()) { + expect(entry.description, `${entry.skill}: empty or missing frontmatter description`).not.toBe(''); + } + }); +}); diff --git a/test/codex-e2e.test.ts b/test/codex-e2e.test.ts index 2f2817f90..e2f33b110 100644 --- a/test/codex-e2e.test.ts +++ b/test/codex-e2e.test.ts @@ -37,15 +37,24 @@ const CODEX_AVAILABLE = (() => { const evalsEnabled = !!process.env.EVALS; -// Skip all tests if codex is not available or EVALS is not set. +// External-service tests are periodic-tier (CLAUDE.md tiering rule 3): +// "Requires external service (Codex, Gemini)? -> periodic". The positive +// form below is the canonical whole-file guard shape — the sharded runner's +// classifyPaidTestFile greps for it to exclude this file from gate. +const tierOk = process.env.EVALS_TIER === 'periodic'; + +// Skip all tests if codex is not available, EVALS is not set, or we're in +// the gate tier. // Note: Codex uses its own auth from ~/.codex/ config — no OPENAI_API_KEY env var needed. -const SKIP = !CODEX_AVAILABLE || !evalsEnabled; +const SKIP = !CODEX_AVAILABLE || !evalsEnabled || !tierOk; const describeCodex = SKIP ? describe.skip : describe; // Log why we're skipping (helpful for debugging CI) if (!evalsEnabled) { // Silent — same as Claude E2E tests, EVALS=1 required +} else if (!tierOk) { + process.stderr.write('\nCodex E2E: SKIPPED — external-service test, periodic tier only (EVALS_TIER === \'periodic\')\n'); } else if (!CODEX_AVAILABLE) { process.stderr.write('\nCodex E2E: SKIPPED — codex binary not found (install: npm i -g @openai/codex)\n'); } diff --git a/test/context-bill.test.ts b/test/context-bill.test.ts new file mode 100644 index 000000000..79cf79aab --- /dev/null +++ b/test/context-bill.test.ts @@ -0,0 +1,679 @@ +/** + * gstack-context-bill — token bill-of-materials for an installed skills tree. + * + * Free tier, no network, no API keys. Covers the STRIPPED port: + * - ALWAYS-ON ledger: exact frontmatter byte sums, dead-key flag against + * the upstream router-key contract, foreign-host file flag + * - EAGER ledger: SKILL.md + forced-read refs from the "for every + * invocation" phrase; stripped tiers stay zero/empty (shape preserved) + * - the three upstream fixes: (a) root-as-container walking + + * node_modules/dotdir exclusion in walkMd, (b) repo-checkout subdir skip + * for installed trees, (c) widened ROUTER_KEYS + * - token estimate calibration, --json shape, --diff, --budget exit codes + * - --exact via injected fetch: envelope subtraction, measurement + * replacement, typed failures, egress receipt-before-send + fail-open + * - ground truth against THIS repo via test/helpers/skill-census.ts + */ +import { describe, it, expect, beforeAll, afterAll } from "bun:test"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { + buildBill, + calibrationTable, + checkBudget, + contentClass, + contextBillMain, + diffBills, + estimateTokens, + findSkillDirs, + measureExactTokens, + renderBill, + tokenDisclaimer, + walkMd, + ExactModeError, + TOKEN_DIVISOR, + TOKEN_DIVISORS, +} from "../lib/context-bill"; +import { listReceipts, sha256Hex } from "../lib/egress-receipt"; +import { skillCensus } from "./helpers/skill-census"; + +const ROOT = path.join(import.meta.dir, ".."); +const TREE_A = path.join(import.meta.dir, "fixtures", "context-bill", "tree-a"); + +function fileBytes(...segments: string[]): number { + return fs.statSync(path.join(...segments)).size; +} + +/** Frontmatter block bytes, computed independently of the implementation. */ +function frontmatterBytes(file: string): number { + const text = fs.readFileSync(file, "utf8"); + const close = text.indexOf("\n---\n", 3); + return Buffer.byteLength(text.slice(0, close + 5), "utf8"); +} + +function capture() { + let buf = ""; + return { + stream: { write: (s: string) => ((buf += s), true) } as unknown as NodeJS.WriteStream, + text: () => buf, + }; +} + +describe("always-on ledger", () => { + const bill = buildBill(TREE_A); + + it("sums per-skill frontmatter bytes exactly", () => { + const alpha = bill.skills.find((s) => s.name === "alpha")!; + const beta = bill.skills.find((s) => s.name === "beta")!; + expect(alpha.frontmatterBytes).toBe(frontmatterBytes(path.join(TREE_A, "alpha", "SKILL.md"))); + expect(beta.frontmatterBytes).toBe(frontmatterBytes(path.join(TREE_A, "beta", "SKILL.md"))); + expect(bill.totals.alwaysOnBytes).toBe(alpha.frontmatterBytes + beta.frontmatterBytes); + }); + + it("flags only keys outside the upstream router contract (fix c: widened ROUTER_KEYS)", () => { + const alpha = bill.skills.find((s) => s.name === "alpha")!; + // `triggers` is part of the upstream frontmatter contract now — flagging + // it was the fork's contract, not this repo's. + expect(alpha.deadKeys).toEqual(["x-dead-key"]); + expect(bill.skills.find((s) => s.name === "beta")!.deadKeys).toEqual([]); + }); + + it("upstream contract keys are never dead: name/description/version/allowed-tools/triggers/preamble-tier", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-keys-")); + fs.mkdirSync(path.join(tmp, "s")); + fs.writeFileSync( + path.join(tmp, "s", "SKILL.md"), + "---\nname: s\ndescription: d\nversion: 1\nallowed-tools: Bash\ntriggers: t\npreamble-tier: 2\n---\n\n# S\n", + ); + const s = buildBill(tmp).skills[0]; + expect(s.deadKeys).toEqual([]); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("flags foreign-host skill-shaped files in scanner scope", () => { + const beta = bill.skills.find((s) => s.name === "beta")!; + expect(beta.foreignFiles.map((f) => f.path)).toEqual(["agents.md"]); + expect(bill.skills.find((s) => s.name === "alpha")!.foreignFiles).toEqual([]); + }); +}); + +describe("eager ledger", () => { + const bill = buildBill(TREE_A); + const alpha = bill.skills.find((s) => s.name === "alpha")!; + + it("eager = SKILL.md + only the forced-read references", () => { + expect(alpha.forcedRefs.map((r) => r.path)).toEqual([ + "references/CORE.md", + "references/POLICY.md", + ]); + const expected = + fileBytes(TREE_A, "alpha", "SKILL.md") + + fileBytes(TREE_A, "alpha", "references", "CORE.md") + + fileBytes(TREE_A, "alpha", "references", "POLICY.md"); + expect(alpha.eagerBytes).toBe(expected); + }); + + it("a skill with no forced reads bills only its SKILL.md", () => { + const beta = bill.skills.find((s) => s.name === "beta")!; + expect(beta.eagerBytes).toBe(fileBytes(TREE_A, "beta", "SKILL.md")); + }); + + it("stripped tiers stay zero/empty but keep their shape (re-adding is additive)", () => { + // OPTIONAL.md is mandated under a condition and the mode table routes two + // legacy modules — the fork billed those in CONDITIONAL and LAZY. The + // stripped port must not bill them anywhere NOR lose the fields. + expect(alpha.conditionalRefs).toEqual([]); + expect(alpha.conditionalBytes).toBe(0); + expect(alpha.transitiveRefs).toEqual([]); + expect(alpha.transitiveBytes).toBe(0); + expect(alpha.lazy).toEqual([]); + expect(alpha.orphans).toEqual([]); + expect(alpha.fastPath).toBeNull(); + expect(alpha.routeCeiling).toBeNull(); + // With those tiers stripped, per-invocation == eager. + expect(alpha.perInvocationBytes).toBe(alpha.eagerBytes); + expect(alpha.perInvocationTokens).toBe(alpha.eagerTokens); + }); +}); + +describe("upstream fix a: root-as-container + walkMd exclusions", () => { + it("a root with its own SKILL.md is billed AND walked into (router + children)", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-root-")); + fs.writeFileSync(path.join(tmp, "SKILL.md"), "---\nname: router\ndescription: r\n---\n# Router\n"); + fs.mkdirSync(path.join(tmp, "qa")); + fs.writeFileSync(path.join(tmp, "qa", "SKILL.md"), "---\nname: qa\ndescription: q\n---\n# QA\n"); + const dirs = findSkillDirs(tmp).map((d) => path.relative(fs.realpathSync(tmp), fs.realpathSync(d)) || "."); + expect(dirs.sort()).toEqual([".", "qa"]); + // A NON-root skill dir is still a leaf: nothing nested under qa/ counts. + fs.mkdirSync(path.join(tmp, "qa", "nested")); + fs.writeFileSync(path.join(tmp, "qa", "nested", "SKILL.md"), "# not a skill\n"); + expect(findSkillDirs(tmp).length).toBe(2); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("walkMd skips node_modules and dot-directories", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-walk-")); + fs.writeFileSync(path.join(tmp, "real.md"), "x"); + fs.mkdirSync(path.join(tmp, "node_modules", "pkg"), { recursive: true }); + fs.writeFileSync(path.join(tmp, "node_modules", "pkg", "README.md"), "y".repeat(5000)); + fs.mkdirSync(path.join(tmp, ".git"), { recursive: true }); + fs.writeFileSync(path.join(tmp, ".git", "notes.md"), "z"); + const files = walkMd(tmp).map((f) => path.basename(f)); + expect(files).toEqual(["real.md"]); + fs.rmSync(tmp, { recursive: true, force: true }); + }); +}); + +describe("upstream fix b: installed-tree layout (repo-checkout subdir skip)", () => { + it("skips a subdir that is its own repo checkout (gstack/ inside ~/.claude/skills)", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-install-")); + // Flat installed skill dirs. + for (const name of ["qa", "ship"]) { + fs.mkdirSync(path.join(tmp, name)); + fs.writeFileSync(path.join(tmp, name, "SKILL.md"), `---\nname: ${name}\ndescription: d\n---\n# ${name}\n`); + } + // A full repo checkout dropped into the tree: has .git and its own router + // SKILL.md plus nested skill sources. None of it is an installed skill. + fs.mkdirSync(path.join(tmp, "gstack", ".git"), { recursive: true }); + fs.writeFileSync(path.join(tmp, "gstack", "SKILL.md"), "---\nname: _router\ndescription: d\n---\n# router\n"); + fs.mkdirSync(path.join(tmp, "gstack", "review")); + fs.writeFileSync(path.join(tmp, "gstack", "review", "SKILL.md"), "---\nname: review\ndescription: d\n---\n# r\n"); + + const names = buildBill(tmp).skills.map((s) => s.name).sort(); + expect(names).toEqual(["qa", "ship"]); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("follows a directory symlink to a sibling skill (connect-chrome shape)", () => { + if (process.platform === "win32") return; + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-symlink-")); + fs.mkdirSync(path.join(tmp, "open-browser")); + fs.writeFileSync(path.join(tmp, "open-browser", "SKILL.md"), "---\nname: ob\ndescription: d\n---\n# ob\n"); + fs.symlinkSync(path.join(tmp, "open-browser"), path.join(tmp, "connect-chrome")); + const names = buildBill(tmp).skills.map((s) => s.name).sort(); + // Both entries are real scanner load, so both are billed. + expect(names).toEqual(["connect-chrome", "open-browser"]); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("prefers ./skills, then .agents/skills, then .claude/skills, project before user", async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-detect-")); + const home = path.join(tmp, "home"); + const proj = path.join(tmp, "proj"); + const plant = (dir: string, name: string) => { + fs.mkdirSync(path.join(dir, name), { recursive: true }); + fs.cpSync(TREE_A, path.join(dir, name), { recursive: true }); + return path.join(dir, name); + }; + const homeClaude = plant(home, path.join(".claude", "skills")); + const homeAgents = plant(home, path.join(".agents", "skills")); + const projClaude = plant(proj, path.join(".claude", "skills")); + const projAgents = plant(proj, path.join(".agents", "skills")); + + const run = async () => { + const out = capture(); + const code = await contextBillMain([], { cwd: proj, homeDir: home, stdout: out.stream, stderr: out.stream }); + expect(code).toBe(0); + return out.text().split("\n")[0]; + }; + + expect(await run()).toContain(projAgents); + fs.rmSync(projAgents, { recursive: true, force: true }); + expect(await run()).toContain(projClaude); + fs.rmSync(projClaude, { recursive: true, force: true }); + expect(await run()).toContain(homeAgents); + fs.rmSync(homeAgents, { recursive: true, force: true }); + expect(await run()).toContain(homeClaude); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("names every candidate it tried when no tree exists", async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-none-")); + const out = capture(); + const code = await contextBillMain([], { cwd: tmp, homeDir: tmp, stdout: out.stream, stderr: out.stream }); + expect(code).toBe(2); + expect(out.text()).toContain(path.join(tmp, ".agents", "skills")); + expect(out.text()).toContain(path.join(tmp, ".claude", "skills")); + fs.rmSync(tmp, { recursive: true, force: true }); + }); +}); + +describe("token estimate calibration", () => { + it("uses the corpus-wide divisor for path-less callers, rounded", () => { + expect(estimateTokens(3900)).toBe(1000); + expect(estimateTokens(TOKEN_DIVISOR * 2)).toBe(2); + }); + + it("charges each content class its own measured divisor", () => { + expect(contentClass("/x/plan/SKILL.md")).toBe("skillmd"); + expect(contentClass("/x/plan/references/legacy/office.md")).toBe("legacy"); + expect(contentClass("/x/plan/references/RUNTIME.md")).toBe("reference"); + expect(contentClass("/x/plan/references/artifacts/qa/t.md")).toBe("artifact"); + expect(contentClass("/x/plan/SKILL.md#frontmatter")).toBe("frontmatter"); + expect(TOKEN_DIVISORS.legacy).toBeLessThan(TOKEN_DIVISORS.skillmd); + const alpha = buildBill(TREE_A).skills.find((s) => s.name === "alpha")!; + expect(alpha.skillMdTokens).toBeCloseTo(alpha.skillMdBytes / TOKEN_DIVISORS.skillmd, 6); + expect(alpha.forcedRefs[0].tokens).toBeCloseTo(alpha.forcedRefs[0].bytes / TOKEN_DIVISORS.reference, 6); + }); + + it("never claims more precision than it has: no divisor is a measurement", () => { + const bill = buildBill(TREE_A); + expect(bill.tokenEstimateErrorPct).toBeGreaterThan(0); + expect(bill.tokenSource).toContain("estimate"); + expect(bill.calibration).toBeUndefined(); + }); + + it("names the measured error band rather than a vague 'estimates'", () => { + const text = tokenDisclaimer(buildBill(TREE_A)); + expect(text).toContain("ESTIMATES"); + expect(text).toMatch(/worst single file 40%/); + expect(text).toContain("--exact"); + expect(text).toContain("Bytes are always exact"); + const exactText = tokenDisclaimer({ + tokenEstimateErrorPct: 0, + tokenSource: "count_tokens (claude-opus-4-5)", + }); + expect(exactText).toContain("measured with count_tokens"); + expect(exactText).not.toContain("ESTIMATES"); + }); +}); + +describe("rendering", () => { + it("text output carries the live ledgers plus flags", () => { + const text = renderBill(buildBill(TREE_A)); + expect(text).toContain("ALWAYS-ON (every session): 2 skills"); + expect(text).toContain("EAGER (per invocation)"); + expect(text).toContain("frontmatter key(s) the router never reads: x-dead-key"); + expect(text).toContain("foreign-host file in scanner scope: agents.md"); + expect(text).toContain("TOTAL on disk:"); + expect(text).toContain("Token source: estimate"); + expect(text).toContain("Token counts are ESTIMATES"); + }); + + it("states the always-on row's exclusion of the host's per-skill wrapper", () => { + const text = renderBill(buildBill(TREE_A)); + const alwaysOn = text.slice(text.indexOf("ALWAYS-ON"), text.indexOf("EAGER (")); + expect(alwaysOn).toContain("excludes the host's per-skill available_skills XML wrapper"); + }); + + it("--json shape is stable (stripped tiers keep their fields)", async () => { + const out = capture(); + const code = await contextBillMain([TREE_A, "--json"], { stdout: out.stream, stderr: out.stream }); + expect(code).toBe(0); + const bill = JSON.parse(out.text()); + expect(bill.tokenEstimate).toEqual(TOKEN_DIVISORS); + expect(bill.tokenEstimateErrorPct).toBe(40); + expect(Object.keys(bill.totals).sort()).toEqual([ + "alwaysOnBytes", + "alwaysOnTokens", + "eagerBytesBySkill", + "eagerTokensBySkill", + "perInvocationBytesBySkill", + "perInvocationTokensBySkill", + "skillCount", + "totalMdBytes", + "totalMdTokens", + ]); + const skill = bill.skills[0]; + for (const key of ["name", "frontmatterBytes", "frontmatterTokens", "deadKeys", "skillMdBytes", "skillMdTokens", "forcedRefs", "eagerBytes", "eagerTokens", "fastPath", "conditionalRefs", "conditionalBytes", "conditionalTokens", "transitiveRefs", "transitiveBytes", "transitiveTokens", "perInvocationBytes", "perInvocationTokens", "routeCeiling", "lazy", "orphans", "foreignFiles", "totalMdTokens"]) { + expect(skill).toHaveProperty(key); + } + for (const r of skill.forcedRefs) expect(typeof r.tokens).toBe("number"); + }); +}); + +describe("--diff", () => { + it("reports exactly the grown row and exits 2", async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-")); + const treeB = path.join(tmp, "tree-b"); + fs.cpSync(TREE_A, treeB, { recursive: true }); + fs.appendFileSync(path.join(treeB, "alpha", "SKILL.md"), "x".repeat(43000)); + + const diff = diffBills(buildBill(TREE_A), buildBill(treeB)); + expect(diff.rows).toHaveLength(1); + expect(diff.rows[0]).toMatchObject({ ledger: "eager", label: "alpha", delta: 43000 }); + expect(diff.grew).toBe(true); + + const out = capture(); + const code = await contextBillMain(["--diff", TREE_A, treeB], { stdout: out.stream, stderr: out.stream }); + expect(code).toBe(2); + expect(out.text()).toContain("eager"); + expect(out.text()).toContain("GREW"); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("identical trees diff clean and exit 0", async () => { + const out = capture(); + const code = await contextBillMain(["--diff", TREE_A, TREE_A], { stdout: out.stream, stderr: out.stream }); + expect(code).toBe(0); + expect(out.text()).toContain("No context-cost changes"); + }); +}); + +describe("--budget", () => { + const alphaEagerTok = Math.round(buildBill(TREE_A).skills.find((s) => s.name === "alpha")!.eagerTokens); + + it("exits 0 under budget, 2 over budget with offending files listed", async () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-budget-")); + const okBudget = path.join(tmp, "ok.json"); + const tightBudget = path.join(tmp, "tight.json"); + fs.writeFileSync(okBudget, JSON.stringify({ alwaysOnTotal: 10000, eagerPerInvocation: { alpha: alphaEagerTok } })); + fs.writeFileSync(tightBudget, JSON.stringify({ eagerPerInvocation: { alpha: alphaEagerTok - 1 } })); + + const ok = capture(); + expect(await contextBillMain([TREE_A, "--budget", okBudget], { stdout: ok.stream, stderr: ok.stream })).toBe(0); + expect(ok.text()).toContain("Within budget"); + + const over = capture(); + expect(await contextBillMain([TREE_A, "--budget", tightBudget], { stdout: over.stream, stderr: over.stream })).toBe(2); + expect(over.text()).toContain("OVER BUDGET: eagerPerInvocation.alpha"); + expect(over.text()).toContain("alpha/SKILL.md"); + expect(over.text()).toContain("references/CORE.md"); + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it("perInvocation and routeCeiling keys stay accepted (they gate the eager figure while tiers are stripped)", () => { + const bill = buildBill(TREE_A); + expect(checkBudget(bill, { perInvocation: { alpha: alphaEagerTok } })).toEqual([]); + expect(checkBudget(bill, { perInvocation: { alpha: alphaEagerTok - 1 } })).toHaveLength(1); + expect(checkBudget(bill, { routeCeiling: { alpha: alphaEagerTok } })).toEqual([]); + }); + + it("checkBudget flags a budgeted skill missing from the tree", () => { + const violations = checkBudget(buildBill(TREE_A), { eagerPerInvocation: { ghost: 100 } }); + expect(violations).toHaveLength(1); + expect(violations[0].ceiling).toBe("eagerPerInvocation.ghost"); + }); +}); + +describe("--exact (opt-in measurement; offline here via an injected fetch)", () => { + let egressHome: string; + beforeAll(() => { + egressHome = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-egress-")); + }); + afterAll(() => { + try { fs.chmodSync(path.join(egressHome, "security"), 0o700); } catch {} + fs.rmSync(egressHome, { recursive: true, force: true }); + }); + + const ENVELOPE = 7; + /** Deterministic stand-in for count_tokens: 1 token per 3 chars, plus envelope. */ + function fakeFetch(calls: { body: string }[] = []) { + return (async (_url: string, init: { body: string }) => { + calls.push({ body: init.body }); + const { messages } = JSON.parse(init.body); + const text: string = messages[0].content; + return { + ok: true, + json: async () => ({ input_tokens: Math.ceil(text.length / 3) + ENVELOPE }), + }; + }) as never; + } + + it("writes the egress receipt BEFORE the first count_tokens POST", async () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-receipt-")); + let receiptsAtFirstPost = -1; + const fetchImpl = (async (_url: string, init: { body: string }) => { + if (receiptsAtFirstPost === -1) receiptsAtFirstPost = listReceipts(home).length; + const { messages } = JSON.parse(init.body); + return { ok: true, json: async () => ({ input_tokens: Math.ceil(messages[0].content.length / 3) + ENVELOPE }) }; + }) as never; + await measureExactTokens(TREE_A, { model: "m", apiKey: "sk-test", fetchImpl, egressHome: home }); + expect(receiptsAtFirstPost).toBe(1); // the receipt existed before any POST + const receipts = listReceipts(home); + expect(receipts).toHaveLength(1); + expect(receipts[0].sink).toBe("context-bill-exact"); + expect(receipts[0].host).toBe("api.anthropic.com"); + expect(receipts[0].sha256).toBeNull(); + expect(receipts[0].bytes).toBeGreaterThan(0); + fs.rmSync(home, { recursive: true, force: true }); + }); + + it("fail-open: an unwritable ledger degrades --exact to the offline estimate, sending nothing", async () => { + if (process.platform === "win32" || process.getuid?.() === 0) return; + const home = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-refuse-")); + fs.mkdirSync(path.join(home, "security"), { recursive: true, mode: 0o500 }); + let called = false; + const stdout = capture(); + const stderr = capture(); + const code = await contextBillMain([TREE_A, "--exact", "--json"], { + stdout: stdout.stream, + stderr: stderr.stream, + apiKey: "sk-test", + fetchImpl: (() => ((called = true), Promise.reject(new Error("must not send")))) as never, + egressHome: home, + }); + expect(code).toBe(0); // the run still answers, with the estimate + expect(called).toBe(false); // NOTHING was sent unrecorded + expect(stderr.text()).toContain("exact_egress_receipt_failed"); + expect(stderr.text()).toContain("Falling back to the offline estimate"); + expect(JSON.parse(stdout.text()).tokenSource).toContain("estimate"); + fs.chmodSync(path.join(home, "security"), 0o700); + fs.rmSync(home, { recursive: true, force: true }); + }); + + it("subtracts the request envelope so small files are not overcharged", async () => { + const calls: { body: string }[] = []; + const measured = await measureExactTokens(TREE_A, { + model: "test-model", + apiKey: "sk-test", + fetchImpl: fakeFetch(calls), + egressHome, + }); + expect(measured.tokenSource).toBe("count_tokens (test-model)"); + const core = path.join(TREE_A, "alpha", "references", "CORE.md"); + const text = fs.readFileSync(core, "utf8"); + expect(measured.tokensOf(core, text.length)).toBe(Math.ceil(text.length / 3)); + // One probe call for the envelope, then one per measured text. + expect(calls.length).toBe(measured.measuredFiles + 1); + }); + + it("measures the frontmatter block, not the whole SKILL.md, for the always-on row", async () => { + const measured = await measureExactTokens(TREE_A, { + model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome, + }); + const skillMd = path.join(TREE_A, "alpha", "SKILL.md"); + const fmTokens = measured.tokensOf(`${skillMd}#frontmatter`, 0); + const bodyTokens = measured.tokensOf(skillMd, 0); + expect(fmTokens).toBeGreaterThan(0); + expect(fmTokens).toBeLessThan(bodyTokens); + }); + + it("exact tokens replace the estimate everywhere the bill prices content", async () => { + const measured = await measureExactTokens(TREE_A, { + model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome, + }); + const bill = buildBill(TREE_A, { tokensOf: measured.tokensOf, tokenSource: measured.tokenSource }); + const alpha = bill.skills.find((s) => s.name === "alpha")!; + expect(alpha.eagerTokens).toBeGreaterThanOrEqual(alpha.eagerBytes / 3); + expect(alpha.eagerTokens).toBeLessThan(alpha.eagerBytes / 3 + 4); + expect(alpha.eagerTokens / (alpha.eagerBytes / TOKEN_DIVISOR)).toBeGreaterThan(1.2); + expect(alpha.eagerTokens).toBe(alpha.skillMdTokens + alpha.forcedRefs.reduce((n, r) => n + r.tokens, 0)); + expect(bill.tokenEstimateErrorPct).toBe(0); + expect(renderBill(bill)).toContain("Token counts measured with count_tokens"); + expect(renderBill(bill)).not.toMatch(/\(~\d/); + }); + + it("measures against a relative root too, instead of silently estimating", async () => { + const relative = path.relative(process.cwd(), TREE_A); + const measured = await measureExactTokens(relative, { + model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome, + }); + const bill = buildBill(relative, { tokensOf: measured.tokensOf, tokenSource: measured.tokenSource }); + for (const s of bill.skills) { + expect(s.totalMdBytes / s.totalMdTokens).toBeLessThan(3.2); + } + expect(measured.missedKeys.size).toBe(0); + }); + + it("counts anything it could not measure instead of passing it off as measured", async () => { + const measured = await measureExactTokens(TREE_A, { + model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome, + }); + expect(measured.missedKeys.size).toBe(0); + const ghost = path.join(TREE_A, "alpha", "nope.md"); + expect(measured.tokensOf(ghost, 4150)).toBeGreaterThan(0); + expect(measured.missedKeys.has(ghost)).toBe(true); + }); + + it("grades its own estimate: calibrationTable reports the residual per file", async () => { + const measured = await measureExactTokens(TREE_A, { + model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome, + }); + const table = calibrationTable(measured.counts, TREE_A); + expect(table.rows.length).toBeGreaterThan(0); + for (const row of table.rows) { + expect(row).toHaveProperty("contentClass"); + expect(row.estimatedTokens).toBeGreaterThan(0); + expect(row.tokens).toBeGreaterThan(0); + expect(row.errorPct).toBeCloseTo(((row.estimatedTokens - row.tokens) / row.tokens) * 100, 1); + } + expect(typeof table.worstErrorPct).toBe("number"); + expect(typeof table.biasPct).toBe("number"); + }); + + it("refuses to go to the network without a key, and says nothing was sent", async () => { + let called = false; + await expect( + measureExactTokens(TREE_A, { + model: "m", + apiKey: "", + fetchImpl: (() => ((called = true), Promise.reject(new Error("should not run")))) as never, + egressHome, + }), + ).rejects.toMatchObject({ code: "exact_missing_api_key" }); + expect(called).toBe(false); + }); + + it("maps HTTP failures to typed codes", async () => { + const status = (code: number) => (async () => ({ ok: false, status: code, text: async () => "nope" })) as never; + for (const [code, expected] of [ + [401, "exact_auth_rejected"], + [403, "exact_auth_rejected"], + [500, "exact_request_failed"], + ] as const) { + await expect( + measureExactTokens(TREE_A, { model: "m", apiKey: "k", fetchImpl: status(code), egressHome }), + ).rejects.toMatchObject({ code: expected }); + } + await expect( + measureExactTokens(TREE_A, { + model: "m", apiKey: "k", + fetchImpl: (() => Promise.reject(new Error("offline"))) as never, + egressHome, + }), + ).rejects.toBeInstanceOf(ExactModeError); + }); + + it("offline by default: no --exact means no network call at all", async () => { + const out = capture(); + let called = false; + const code = await contextBillMain([TREE_A, "--json"], { + stdout: out.stream, + stderr: out.stream, + fetchImpl: (() => ((called = true), Promise.reject(new Error("no")))) as never, + apiKey: "sk-test", + egressHome, + }); + expect(code).toBe(0); + expect(called).toBe(false); + expect(JSON.parse(out.text()).tokenSource).toContain("estimate"); + }); + + it("discloses what --exact sends before sending it", async () => { + const stdout = capture(); + const stderr = capture(); + const code = await contextBillMain([TREE_A, "--exact", "--json"], { + stdout: stdout.stream, + stderr: stderr.stream, + apiKey: "sk-test", + fetchImpl: fakeFetch(), + egressHome, + }); + expect(code).toBe(0); + expect(stderr.text()).toContain("api.anthropic.com"); + expect(stderr.text()).toMatch(/sending the content of \d+ \.md file\(s\)/); + const bill = JSON.parse(stdout.text()); + expect(bill.tokenSource).toContain("count_tokens"); + expect(bill.calibration.rows.length).toBeGreaterThan(0); + }); + + it("degrades to the estimate when exact mode is unavailable, naming the code", async () => { + const stdout = capture(); + const stderr = capture(); + const code = await contextBillMain([TREE_A, "--exact", "--json"], { + stdout: stdout.stream, + stderr: stderr.stream, + apiKey: "", + fetchImpl: (() => Promise.reject(new Error("should not run"))) as never, + egressHome, + }); + expect(code).toBe(0); + expect(stderr.text()).toContain("exact_missing_api_key"); + expect(JSON.parse(stdout.text()).tokenSource).toContain("estimate"); + }); + + it("--help discloses --exact's egress and recalibration", async () => { + const out = capture(); + expect(await contextBillMain(["--help"], { stdout: out.stream, stderr: out.stream })).toBe(0); + expect(out.text()).toContain("api.anthropic.com"); + expect(out.text()).toContain("egress receipt"); + expect(out.text()).toContain("recalibrates"); + }); +}); + +describe("ground truth against THIS repo (skill-census parity)", () => { + const census = skillCensus(ROOT); + const dirs = findSkillDirs(ROOT); + const rels = dirs.map((d) => path.relative(ROOT, d) || "."); + + it("the walker reaches every physical SKILL.md the census counts", () => { + // physicalSkillFiles is the walker's expectation: every depth-1 skill dir + // (symlinked dirs included) plus the root router. + for (const rel of census.physicalSkillFiles) { + const dirRel = rel === "SKILL.md" ? "." : path.dirname(rel); + expect(rels, `walker missed ${rel}`).toContain(dirRel); + } + }); + + it("the root router SKILL.md is billed (fix a live on the real repo)", () => { + expect(census.physicalSkillFiles).toContain("SKILL.md"); // guard the guard + expect(rels).toContain("."); + }); + + it("a real skill's frontmatter bytes match an independent computation", () => { + const qaDir = dirs.find((d) => path.relative(ROOT, d) === "qa")!; + expect(qaDir).toBeTruthy(); + const bill = buildBill(qaDir); + expect(bill.skills).toHaveLength(1); + expect(bill.skills[0].frontmatterBytes).toBe(frontmatterBytes(path.join(qaDir, "SKILL.md"))); + expect(bill.skills[0].frontmatterBytes).toBeGreaterThan(0); + }); + + it("skill count is at least the census count (deeper trees may add more, never fewer)", () => { + expect(dirs.length).toBeGreaterThanOrEqual(census.physicalSkillFiles.length); + }); + + it("no skill dir is billed twice", () => { + expect(new Set(rels).size).toBe(rels.length); + }); +}); + +describe("CLI plumbing", () => { + it("unknown flag and missing tree are usage errors (exit 2)", async () => { + const a = capture(); + expect(await contextBillMain(["--nope"], { stdout: a.stream, stderr: a.stream })).toBe(2); + const b = capture(); + expect(await contextBillMain(["--diff", TREE_A], { stdout: b.stream, stderr: b.stream })).toBe(2); + const c = capture(); + expect(await contextBillMain([path.join(os.tmpdir(), "does-not-exist-xyz")], { stdout: c.stream, stderr: c.stream })).toBe(1); + }); + + it("bin/gstack-context-bill runs standalone", () => { + const result = Bun.spawnSync([path.join(ROOT, "bin", "gstack-context-bill"), TREE_A]); + expect(result.exitCode).toBe(0); + expect(result.stdout.toString()).toContain("ALWAYS-ON"); + expect(result.stdout.toString()).toContain("EAGER"); + }); +}); diff --git a/test/egress-lib.test.ts b/test/egress-lib.test.ts new file mode 100644 index 000000000..2cfb51c45 --- /dev/null +++ b/test/egress-lib.test.ts @@ -0,0 +1,185 @@ +/** + * gstack-egress-lib.sh — shared shell receipt helpers, tested end-to-end + * against a local listener. Free tier, loopback only. + * + * Pins the shell-sink contract: + * - receipt sha256 == sha256 of the exact bytes the listener received + * (same file is hashed and handed to curl via --data-binary @file) + * - fail-closed refusal never touches the network and its stderr carries + * problem + cause + fix in plain language + * - fail-open warns on stderr and proceeds unrecorded + * - payload temp files are consumed immediately (no EXIT traps) + */ + +import { describe, test, expect, beforeEach, afterEach, afterAll } from 'bun:test'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { listReceipts, sha256Hex } from '../lib/egress-receipt'; + +const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); +const LIB = path.join(ROOT, 'bin', 'gstack-egress-lib.sh'); + +const received: string[] = []; +const server = Bun.serve({ + port: 0, + fetch: async (req) => { + received.push(await req.text()); + return new Response('ok'); + }, +}); +const URL_BASE = `http://127.0.0.1:${server.port}`; + +afterAll(() => { + server.stop(true); +}); + +let home: string; + +beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-egress-lib-')); + received.length = 0; +}); + +afterEach(() => { + try { fs.chmodSync(path.join(home, 'security'), 0o700); } catch {} + fs.rmSync(home, { recursive: true, force: true }); +}); + +// Async spawn: spawnSync would block Bun's event loop, deadlocking the +// in-process listener the script curls against. +async function runBash(script: string) { + const proc = Bun.spawn(['bash', '-c', script], { + env: { ...process.env, GSTACK_HOME: home }, + stdout: 'pipe', + stderr: 'pipe', + }); + const [stdout, stderr, status] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { status, stdout, stderr }; +} + +describe('_receipted_curl', () => { + test('receipt sha256 equals sha256 of the body the listener received; payload file consumed', async () => { + const body = '[{"event":"lib-test","v":1}]'; + const result = await runBash(` + set -uo pipefail + . "${LIB}" + payload="$(mktemp)" + printf '%s' '${body}' > "$payload" + echo "PAYLOAD_PATH=$payload" + _receipted_curl closed lib-test 127.0.0.1:${server.port} test-events "telemetry=community" "$payload" \\ + curl -s --max-time 5 -X POST "${URL_BASE}/ingest" + echo "CURL_EXIT=$?" + [ -e "$payload" ] && echo "PAYLOAD_STILL_EXISTS" || echo "PAYLOAD_GONE" + `); + expect(result.status).toBe(0); + expect(result.stdout).toContain('CURL_EXIT=0'); + expect(result.stdout).toContain('PAYLOAD_GONE'); + expect(received.length).toBe(1); + expect(received[0]).toBe(body); + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].sha256).toBe(sha256Hex(received[0])); + expect(receipts[0].bytes).toBe(Buffer.byteLength(body)); + expect(receipts[0].sink).toBe('lib-test'); + expect(receipts[0].status).toBe('exit:0'); // best-effort outcome joined + }); + + test('fail-closed refusal never hits the network; stderr is problem + cause + fix', async () => { + if (process.platform === 'win32' || process.getuid?.() === 0) return; + fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 }); + const result = await runBash(` + set -uo pipefail + . "${LIB}" + payload="$(mktemp)" + printf '%s' 'secret payload' > "$payload" + _receipted_curl closed lib-test 127.0.0.1:${server.port} test-events "telemetry=community" "$payload" \\ + curl -s --max-time 5 -X POST "${URL_BASE}/ingest" + status=$? + [ -e "$payload" ] && echo "PAYLOAD_STILL_EXISTS" || echo "PAYLOAD_GONE" + exit $status + `); + expect(result.status).toBe(3); + expect(received.length).toBe(0); // the send was refused, not attempted + expect(result.stdout).toContain('PAYLOAD_GONE'); // cleaned up on refusal too + // Problem: what did not happen. + expect(result.stderr).toContain('lib-test NOT sent'); + // Cause: the bridge's typed error is quoted. + expect(result.stderr).toContain('EGRESS_RECEIPT_FAILED'); + // Fix: an actionable command. + expect(result.stderr).toContain('Fix: chmod -R u+w'); + // What this is: the ledger explained. + expect(result.stderr).toContain('ATTEMPTS to send off-machine'); + expect(result.stderr).toContain('gstack-egress'); + }); + + test('fail-open warns and proceeds when the receipt cannot be written', async () => { + if (process.platform === 'win32' || process.getuid?.() === 0) return; + fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 }); + const result = await runBash(` + set -uo pipefail + . "${LIB}" + _receipted_curl open lib-open 127.0.0.1:${server.port} version-fetch "user ran --force" --no-payload \\ + curl -s --max-time 5 "${URL_BASE}/VERSION" + `); + expect(result.status).toBe(0); + expect(received.length).toBe(1); // the send proceeded unrecorded + expect(result.stderr).toContain('sending anyway'); + expect(result.stderr).toContain('lib-open'); + }); + + test('--no-payload records sha256:null and does not append --data-binary', async () => { + const result = await runBash(` + set -uo pipefail + . "${LIB}" + _receipted_curl closed lib-get 127.0.0.1:${server.port} version-fetch "user ran --force" --no-payload \\ + curl -s --max-time 5 "${URL_BASE}/VERSION" + `); + expect(result.status).toBe(0); + expect(received.length).toBe(1); + expect(received[0]).toBe(''); // GET, no body + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].sha256).toBeNull(); + expect(receipts[0].bytes).toBe(0); + }); +}); + +describe('_receipted_git', () => { + test('writes a git-class sha256:null receipt then runs the command unmodified', async () => { + const result = await runBash(` + set -uo pipefail + . "${LIB}" + _receipted_git closed brain-sync github.com curated-memory-git-push "artifacts_sync_mode!=off" \\ + bash -c 'echo GIT_RAN "$@"' _ + `); + expect(result.status).toBe(0); + expect(result.stdout).toContain('GIT_RAN'); + expect(result.stdout).not.toContain('--data-binary'); + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].sha256).toBeNull(); + expect(receipts[0].payload_class).toBe('curated-memory-git-push'); + expect(receipts[0].status).toBe('exit:0'); + }); + + test('fail-closed git refusal returns 3 without running the command', async () => { + if (process.platform === 'win32' || process.getuid?.() === 0) return; + fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 }); + const marker = path.join(os.tmpdir(), `gstack-egress-git-${process.pid}`); + fs.rmSync(marker, { force: true }); + const result = await runBash(` + set -uo pipefail + . "${LIB}" + _receipted_git closed brain-sync github.com curated-memory-git-push "artifacts_sync_mode!=off" \\ + touch "${marker}" + `); + expect(result.status).toBe(3); + expect(fs.existsSync(marker)).toBe(false); // command never ran + expect(result.stderr).toContain('brain-sync NOT sent'); + }); +}); diff --git a/test/egress-receipt-wiring.test.ts b/test/egress-receipt-wiring.test.ts new file mode 100644 index 000000000..82dafa7de --- /dev/null +++ b/test/egress-receipt-wiring.test.ts @@ -0,0 +1,380 @@ +/** + * Static-grep tripwire for egress-receipt wiring. Free tier — no API. + * + * THREAT MODEL: the egress ledger is forensic observability — it records + * ATTEMPTED egress so accidents are auditable; it is not an exfiltration + * control. Receipts are written before send, outcomes are best-effort, and + * fail-open classes can send unrecorded with a warning. + * + * Every enumerated off-machine sink must route its send through the receipt + * ledger (lib/egress-receipt.ts), receipt BEFORE send. A future egress call + * site added without a receipt fails CI here instead of becoming a + * user-filed issue. The NEW-SINK SCANNER at the bottom sweeps the whole + * tree for outbound network ops and requires every hit to be either wired + * or in the REASONED exemption list — there is no KNOWN_UNWIRED bucket. + * + * Out of scope, documented here on purpose: the preamble-generated brain + * sync block (scripts/resolvers/preamble/generate-brain-sync-block.ts) + * renders a `git fetch` into skill PROSE that the agent executes — it is + * agent-executed instructions, not a gstack binary, so it is covered by the + * skill-prose exemption below rather than a receipt. + * + * Pattern mirrors test/hermetic-wiring.test.ts: read source files as text, + * assert invariants on their contents. Brittle by design — renaming a + * helper must force the author to look here. + */ + +import { describe, test, expect } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; + +const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); + +function read(rel: string): string { + return fs.readFileSync(path.join(ROOT, rel), 'utf-8'); +} + +function exists(rel: string): boolean { + return fs.existsSync(path.join(ROOT, rel)); +} + +// ── POLARITY TABLE (amendments T3/C8) ────────────────────────────────────── +// Pinned as data: which sinks refuse the send when the receipt cannot be +// written (fail-closed) vs warn and proceed (fail-open). Changing a sink's +// polarity is a security decision — update this table deliberately. +const POLARITY: Record = { + // fail-closed: gstack state leaving the machine unrecorded is worse than + // the operation failing. + 'brain-sync': 'fail-closed', + 'memory-ingest': 'fail-closed', + 'gbrain-sync': 'fail-closed', + 'telemetry-sync': 'fail-closed', + 'browse-tunnel (ngrok)': 'fail-closed', + 'gbrain-mcp-verify': 'fail-closed', + 'supabase-provision': 'fail-closed', + // fail-open: user-facing operations that must not die over an audit-log + // hiccup; they warn on stderr and proceed. + 'design-openai': 'fail-open', + 'update-check': 'fail-open', + 'security-dashboard': 'fail-open', + 'community-dashboard': 'fail-open', + 'git-class user ops (artifacts-init, brain-restore, session-update)': 'fail-open', + 'context-bill --exact': 'fail-open', +}; + +/** TS sinks: must import the canonical helper and call writeReceipt(). */ +const MODULE_SINKS = [ + 'bin/gstack-gbrain-sync.ts', + 'bin/gstack-memory-ingest.ts', + 'browse/src/server.ts', + // Unconditional: context-bill ships in the same tree as this tripwire. A + // missing file must fail loudly (a rename/move that drops its receipt wiring + // is exactly what this pins), not silently soften the assertion. + 'lib/context-bill.ts', +]; + +/** Shell sinks: must source the shared lib; every network op receipted. */ +const SHELL_SINKS = [ + 'bin/gstack-telemetry-sync', + 'bin/gstack-update-check', + 'bin/gstack-brain-sync', + 'bin/gstack-gbrain-mcp-verify', + 'bin/gstack-security-dashboard', + 'bin/gstack-community-dashboard', + 'bin/gstack-gbrain-supabase-provision', + 'bin/gstack-artifacts-init', + 'bin/gstack-brain-restore', + 'bin/gstack-session-update', +]; + +/** design files that talk to api.openai.com — all must use receiptedFetch. */ +const DESIGN_SINKS = [ + 'design/src/generate.ts', + 'design/src/variants.ts', + 'design/src/iterate.ts', + 'design/src/evolve.ts', + 'design/src/check.ts', + 'design/src/diff.ts', + 'design/src/design-to-code.ts', + 'design/src/memory.ts', +]; + +// ── NEW-SINK SCANNER exemptions ──────────────────────────────────────────── +// Every entry carries its reason. An unexplained network op anywhere in the +// swept tree fails the scanner — add real sinks to the wired lists above, +// not here. +const SCANNER_EXEMPT: Record = { + 'bin/gstack-team-init': + 'every git clone is inside an echoed instruction string (install docs); the script executes no network ops', + 'bin/gstack-gbrain-install': + 'user-invoked installer: bodyless HEAD reachability probe to github.com + clone of the public gbrain repo (user-directed install; no gstack state leaves the machine)', + 'bin/gstack-next-version': + 'fetches the user\'s own repo\'s base branch for version-claim freshness — a user-repo dev-workflow op, not gstack-state egress', + 'bin/gstack-version-bump': + 'git fetch appears only in an error-message string', + 'bin/gstack-redact-prepush': + 'git push mentions are hook documentation strings (bypass instructions)', + 'browse/src/security-classifier.ts': + 'HF model download: bodyless GET of a public classifier model (variable URL)', + 'browse/src/write-commands.ts': + 'user-directed page fetch — the browser command surface fetches what the user asked for', + 'browse/src/cli.ts': + 'health probe of the user\'s own pair-agent tunnel URL (reachability probe)', + 'browse/src/commands.ts': + 'git pull appears only in an upgrade-hint message string', + 'browse/src/cookie-picker-ui.ts': + 'served-page JS talking to its own loopback server (same-origin relative fetch)', + 'design/src/compare.ts': + 'served-page JS talking to its own loopback server (relative ./api fetch)', + // Skill prose templates: these render agent-executed instructions (the + // agent runs git in the USER\'S repo at the user\'s direction), they are + // not gstack binaries. Includes the preamble-generated brain-sync block — + // see the header. + 'scripts/resolvers': + 'skill prose templates — agent-executed instructions rendered into SKILL.md, not gstack binaries', +}; + +function isExempt(rel: string): string | undefined { + for (const [key, reason] of Object.entries(SCANNER_EXEMPT)) { + if (rel === key || rel.startsWith(`${key}/`)) return reason; + } + return undefined; +} + +// Receipt markers that make a nearby network op "wired". +const RECEIPT_MARKER = + /_receipted_(curl|git|version_fetch)\b|gstack-egress-receipt["']?\s+write\b|writeReceipt\(|receiptedFetch\(/; + +/** Was a receipt marker present on this line or the 30 preceding lines? */ +function guarded(lines: string[], i: number): boolean { + for (let j = i; j >= Math.max(0, i - 30); j--) { + if (RECEIPT_MARKER.test(lines[j])) return true; + } + return false; +} + +// git as a COMMAND followed by a remote op. Local ops (rev-parse, remote +// get-url, add, commit, merge) never match; neither does prose like +// "curated-memory-git-push" (hyphenated) or "'git fetch'" (quoted). +const GIT_REMOTE_OP = /(^|[;|&`($!]|\s)git(\s+-C\s+\S+)?\s+(push|pull|fetch|clone|ls-remote)\b/; +// git spawn-array form in TS: spawn("git", ["push", ...]). +const GIT_SPAWN_OP = /["'`]git["'`]\s*,\s*\[\s*["'`](push|pull|fetch|clone|ls-remote)/; +// curl as a command token. +const CURL_OP = /(^|[|&;(`]|\s|\$\()curl\s/; +// fetch() with an absolute http(s) URL (loopback filtered separately). +const FETCH_ABS = /(^|[^A-Za-z])fetch(Fn|Impl)?\(\s*[`'"]https?:\/\//; + +function isTextFile(full: string): boolean { + try { + const buf = fs.readFileSync(full); + return !buf.subarray(0, 1024).includes(0); + } catch { + return false; + } +} + +function* walk(dir: string): Generator { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + if (entry.name === 'node_modules' || entry.name === 'dist' || entry.name.startsWith('.')) continue; + const full = path.join(dir, entry.name); + if (entry.isDirectory()) yield* walk(full); + else if (entry.isFile()) yield full; + } +} + +/** + * Collect un-receipted outbound network ops in a file. Skips comments, + * loopback lines, `command -v` probes, message-emitting lines, and shell + * heredoc bodies (echoed instructions are not executed ops). + */ +function scanFile(rel: string): string[] { + const src = read(rel); + const isTs = /\.(ts|js|mjs|tsx)$/.test(rel); + const lines = src.split('\n'); + const offenders: string[] = []; + let heredocEnd: string | null = null; + for (let i = 0; i < lines.length; i++) { + const line = lines[i]; + if (heredocEnd !== null) { + if (line.trim() === heredocEnd) heredocEnd = null; + continue; + } + if (!isTs) { + const heredoc = line.match(/<<-?\s*['"]?([A-Za-z_][A-Za-z0-9_]*)['"]?/); + if (heredoc) heredocEnd = heredoc[1]; + } + const trimmed = line.trimStart(); + if (/^(#|\/\/|\*|\/\*)/.test(trimmed)) continue; + if (line.includes('127.0.0.1') || line.includes('localhost')) continue; + if (/command -v/.test(line)) continue; + if (/^(echo|printf|emit|die|fail|log)\b/.test(trimmed)) continue; + + // TS: shell-style git ops only count on lines that actually execute + // something (spawn/exec markers) — template-literal prose does not. + const tsExecGit = + GIT_SPAWN_OP.test(line) || + (GIT_REMOTE_OP.test(line) && /\b(spawn|spawnSync|exec|execSync|execFileSync|runCommand)\b/.test(line)); + const isNetOp = isTs + ? FETCH_ABS.test(line) || tsExecGit + : CURL_OP.test(line) || GIT_REMOTE_OP.test(line); + if (!isNetOp) continue; + if (guarded(lines, i)) continue; + offenders.push(`${rel}:${i + 1}: ${line.trim().slice(0, 100)}`); + } + return offenders; +} + +describe('egress receipt wiring tripwire', () => { + test('every TS sink imports lib/egress-receipt and calls writeReceipt()', () => { + for (const rel of MODULE_SINKS) { + const src = read(rel); + expect(src.includes('egress-receipt'), `${rel}: must import lib/egress-receipt`).toBe(true); + expect(src.includes('writeReceipt('), `${rel}: must call writeReceipt() before its send`).toBe(true); + } + }); + + test('every shell sink sources gstack-egress-lib.sh', () => { + for (const rel of SHELL_SINKS) { + const src = read(rel); + expect( + src.includes('gstack-egress-lib.sh'), + `${rel}: must source bin/gstack-egress-lib.sh for _receipted_* helpers`, + ).toBe(true); + } + }); + + test('every network op in a wired shell sink sits under a receipt', () => { + const offenders = SHELL_SINKS.flatMap((rel) => scanFile(rel)); + expect( + offenders, + 'un-receipted network call(s) — wrap in _receipted_curl/_receipted_git or write the receipt first:\n' + + offenders.join('\n'), + ).toEqual([]); + }); + + test('browse tunnel: every ngrok.forward() has a writeReceipt in the 30 preceding lines', () => { + const lines = read('browse/src/server.ts').split('\n'); + const offenders: string[] = []; + let sawForward = false; + for (let i = 0; i < lines.length; i++) { + if (!lines[i].includes('ngrok.forward(')) continue; + if (/^\s*(\/\/|\*)/.test(lines[i])) continue; + sawForward = true; + const context = lines.slice(Math.max(0, i - 30), i).join('\n'); + if (!context.includes('writeReceipt(')) offenders.push(`browse/src/server.ts:${i + 1}`); + } + expect(sawForward, 'expected ngrok.forward call sites in server.ts').toBe(true); + expect(offenders, 'tunnel session opened without a receipt: ' + offenders.join(', ')).toEqual([]); + }); + + test('design: every api.openai.com call routes through receiptedFetch', () => { + for (const rel of DESIGN_SINKS) { + const src = read(rel); + expect( + src.includes('receipted-fetch'), + `${rel}: must import design/src/receipted-fetch`, + ).toBe(true); + const lines = src.split('\n'); + for (let i = 0; i < lines.length; i++) { + if (!lines[i].includes('api.openai.com')) continue; + if (/^\s*(\/\/|\*)/.test(lines[i])) continue; + if (/\bfetch(Fn|Impl)?\(/.test(lines[i])) { + throw new Error( + `${rel}:${i + 1}: raw fetch to api.openai.com — route it through receiptedFetch()`, + ); + } + } + } + }); + + test('deprecated dead-endpoint brain consumer/reader scripts stay deleted', () => { + // lstat (not existsSync) so a dangling symlink also fails. + for (const rel of ['bin/gstack-brain-consumer', 'bin/gstack-brain-reader']) { + let present = true; + try { + fs.lstatSync(path.join(ROOT, rel)); + } catch { + present = false; + } + expect(present, `${rel} was deleted (dead /ingest-repo egress sink) — do not resurrect`).toBe(false); + } + }); + + test('polarity table names every wired sink exactly once per polarity', () => { + const closed = Object.entries(POLARITY).filter(([, p]) => p === 'fail-closed').map(([s]) => s); + const open = Object.entries(POLARITY).filter(([, p]) => p === 'fail-open').map(([s]) => s); + expect(closed.sort()).toEqual([ + 'brain-sync', + 'browse-tunnel (ngrok)', + 'gbrain-mcp-verify', + 'gbrain-sync', + 'memory-ingest', + 'supabase-provision', + 'telemetry-sync', + ]); + expect(open.sort()).toEqual([ + 'community-dashboard', + 'context-bill --exact', + 'design-openai', + 'git-class user ops (artifacts-init, brain-restore, session-update)', + 'security-dashboard', + 'update-check', + ]); + }); + + test('polarity spot-checks: closed sinks refuse, open sinks warn', () => { + // telemetry-sync (closed): the wrapped POST uses the `closed` policy. + expect(read('bin/gstack-telemetry-sync')).toMatch(/_receipted_curl closed telemetry-sync/); + // brain-sync (closed): refusal exits before the commit consumes the queue. + expect(read('bin/gstack-brain-sync')).toMatch(/gstack-egress-receipt["']? write/); + // update-check (open). + expect(read('bin/gstack-update-check')).toMatch(/_receipted_curl open update-check/); + // dashboards (open). + expect(read('bin/gstack-security-dashboard')).toMatch(/_receipted_curl open security-dashboard/); + expect(read('bin/gstack-community-dashboard')).toMatch(/_receipted_curl open community-dashboard/); + // mcp-verify + provision (closed). + expect(read('bin/gstack-gbrain-mcp-verify')).toMatch(/_receipted_curl closed gbrain-mcp-verify/); + expect(read('bin/gstack-gbrain-supabase-provision')).toMatch(/_receipted_curl closed supabase-provision/); + // design (open): the wrapper catches receipt errors and proceeds. + const rf = read('design/src/receipted-fetch.ts'); + expect(rf).toContain('fail-open'); + expect(rf.indexOf('writeReceipt(')).toBeLessThan(rf.indexOf('fetchImpl(url, init)')); + }); + + test('NEW-SINK SCANNER: every outbound network op in the tree is wired or reasoned-exempt', () => { + const SWEEP = ['bin', 'lib', 'scripts', 'design/src', 'browse/src']; + const offenders: string[] = []; + for (const dirRel of SWEEP) { + const dir = path.join(ROOT, dirRel); + if (!fs.existsSync(dir)) continue; + for (const full of walk(dir)) { + const rel = path.relative(ROOT, full).split(path.sep).join('/'); + if (!/\.(ts|js|mjs|sh|tsx)$/.test(rel) && !isTextFile(full)) continue; + if (isExempt(rel)) continue; + offenders.push(...scanFile(rel)); + } + } + expect( + offenders, + 'unwired outbound network op(s). Wire each through the receipt helpers ' + + '(_receipted_curl/_receipted_git in shell, writeReceipt/receiptedFetch in TS) ' + + 'or add a REASONED exemption with the honest why:\n' + + offenders.join('\n'), + ).toEqual([]); + }); + + test('shebang tripwire: no bin/gstack-* file carries a node shebang (amendment 2A)', () => { + const offenders: string[] = []; + for (const entry of fs.readdirSync(path.join(ROOT, 'bin'))) { + if (!entry.startsWith('gstack-')) continue; + const full = path.join(ROOT, 'bin', entry); + if (!fs.lstatSync(full).isFile()) continue; + if (!isTextFile(full)) continue; + const firstLine = fs.readFileSync(full, 'utf-8').split('\n', 1)[0]; + if (firstLine.startsWith('#!') && /\bnode\b/.test(firstLine)) { + offenders.push(`bin/${entry}: ${firstLine}`); + } + } + expect(offenders, 'node shebangs in bin/ (use #!/usr/bin/env bun): ' + offenders.join(', ')).toEqual([]); + }); +}); diff --git a/test/egress-receipt.test.ts b/test/egress-receipt.test.ts new file mode 100644 index 000000000..64e6f3928 --- /dev/null +++ b/test/egress-receipt.test.ts @@ -0,0 +1,258 @@ +/** + * Egress receipts — chain, fail-closed, verify, shell bridge. Free tier, no network. + * + * THREAT MODEL: the egress ledger is forensic observability — it records + * ATTEMPTED egress so accidents are auditable; it is not an exfiltration + * control. + * + * Pins the auditor contract: + * - receipt-before-send fail-closed (EGRESS_RECEIPT_FAILED, no ledger = no send) + * - content-free lines chained by prev = sha256(previous raw line) + * - tail-read correctness (last line found without loading the whole file) + * - abandoned-lock reclaim (stale lock dir >10s old is removed, not fatal) + * - WARN-at-size (one self-explanatory stderr warning per process >25MB) + */ + +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { spawnSync } from 'child_process'; +import { + EGRESS_RECEIPT_FAILED, + LEDGER_WARN_BYTES, + egressLedgerPath, + ledgerSizeWarning, + listReceipts, + resetLedgerSizeWarningForTests, + sha256Hex, + verifyLedger, + writeOutcome, + writeReceipt, +} from '../lib/egress-receipt'; + +const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); + +let home: string; + +beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-egress-')); +}); + +afterEach(() => { + try { fs.chmodSync(path.join(home, 'security'), 0o700); } catch {} // undo fail-closed fixtures + fs.rmSync(home, { recursive: true, force: true }); +}); + +describe('egress receipt library', () => { + test('receipts chain: prev = sha256 of the previous raw line, "" for line 1', () => { + writeReceipt({ home, sink: 'a', host: 'h1', payloadClass: 'c', bytes: 3, sha256: sha256Hex('abc'), consent: 'k=v' }); + writeReceipt({ home, sink: 'b', host: 'h2', payloadClass: 'c', bytes: 0, sha256: null, consent: 'k=v' }); + const lines = fs.readFileSync(egressLedgerPath(home), 'utf-8').trim().split('\n'); + expect(lines.length).toBe(2); + const first = JSON.parse(lines[0]); + const second = JSON.parse(lines[1]); + expect(first.prev).toBe(''); + expect(second.prev).toBe(sha256Hex(lines[0])); + expect(first.sha256).toBe(sha256Hex('abc')); + expect(second.sha256).toBeNull(); + expect(verifyLedger(home)).toMatchObject({ ok: true, count: 2 }); + }); + + test('ledger is 0600 and the security dir 0700', () => { + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + const ledger = egressLedgerPath(home); + expect(fs.statSync(ledger).mode & 0o777).toBe(0o600); + expect(fs.statSync(path.dirname(ledger)).mode & 0o777).toBe(0o700); + }); + + test('fail-closed: unwritable security dir throws typed EGRESS_RECEIPT_FAILED', () => { + if (process.platform === 'win32' || process.getuid?.() === 0) return; // chmod is advisory there + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + fs.chmodSync(path.join(home, 'security'), 0o500); + try { + expect(() => + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }), + ).toThrow(); + try { + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + } catch (err: any) { + expect(err.code).toBe(EGRESS_RECEIPT_FAILED); + } + } finally { + fs.chmodSync(path.join(home, 'security'), 0o700); + } + }); + + test('outcome records join back onto their receipt in listReceipts', () => { + const { id } = writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + writeOutcome({ home, receipt: id, status: 204 }); + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].status).toBe('204'); + expect(verifyLedger(home)).toMatchObject({ ok: true, count: 2 }); + }); + + test('tampering with a middle line breaks verification at that line', () => { + for (let i = 0; i < 3; i += 1) { + writeReceipt({ home, sink: `s${i}`, host: 'h', payloadClass: 'c', consent: 'telemetry=community' }); + } + const ledger = egressLedgerPath(home); + const lines = fs.readFileSync(ledger, 'utf-8').trim().split('\n'); + lines[1] = lines[1].replace('community', 'communitX'); + fs.writeFileSync(ledger, `${lines.join('\n')}\n`); + // Line 2's edited bytes no longer hash to line 3's recorded prev. + expect(verifyLedger(home)).toMatchObject({ ok: false, brokenLine: 3 }); + }); + + test('validation rejects garbage before touching the ledger', () => { + expect(() => writeReceipt({ home, sink: '', host: 'h', payloadClass: 'c', consent: 'k' } as any)).toThrow(); + expect(() => writeReceipt({ home, sink: 's', host: 'h', payloadClass: 'c', consent: 'k', bytes: -1 })).toThrow(); + expect(() => writeReceipt({ home, sink: 's', host: 'h', payloadClass: 'c', consent: 'k', sha256: 'nothex' })).toThrow(); + expect(fs.existsSync(egressLedgerPath(home))).toBe(false); + }); + + test('abandoned lock: a stale lock dir (>10s-old mtime) is reclaimed by the next writer', () => { + // First write creates the security dir so the lock path's parent exists. + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + const lock = `${egressLedgerPath(home)}.lock`; + fs.mkdirSync(lock); + const old = new Date(Date.now() - 60_000); + fs.utimesSync(lock, old, old); + const started = Date.now(); + const { id } = writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'k=v' }); + expect(id).toMatch(/^[0-9a-f]{64}$/); + // Spin budget is 2.5s; reclaim happens right after budget exhaustion. + expect(Date.now() - started).toBeLessThan(10_000); + expect(fs.existsSync(lock)).toBe(false); + expect(verifyLedger(home)).toMatchObject({ ok: true, count: 2 }); + }); + + test('a fresh (recent-mtime) lock held past the budget fails closed instead of being stolen', () => { + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + const lock = `${egressLedgerPath(home)}.lock`; + fs.mkdirSync(lock); + // Keep the mtime fresh so the reclaim path never fires: refresh it in the + // background while the writer spins out its 2.5s budget. + const refresher = setInterval(() => { + const now = new Date(); + try { fs.utimesSync(lock, now, now); } catch { /* test teardown race */ } + }, 1000); + try { + expect(() => + writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'k=v' }), + ).toThrow(/locked/); + } finally { + clearInterval(refresher); + fs.rmdirSync(lock); + } + }, 15_000); + + test('tail-read: last line is found correctly on a multi-record ledger larger than the tail window', () => { + // 30 records ≈ 9KB > the 4KB tail window, so the append path must find + // the true last line from a partial read. + for (let i = 0; i < 30; i += 1) { + writeReceipt({ + home, + sink: `sink-${i}`, + host: 'h', + payloadClass: `class-${'x'.repeat(200)}-${i}`, + consent: 'k=v', + }); + } + const ledger = egressLedgerPath(home); + expect(fs.statSync(ledger).size).toBeGreaterThan(4096); + const result = verifyLedger(home); + expect(result).toMatchObject({ ok: true, count: 30 }); + }); + + test('chain verify stays intact across 100+ records', () => { + for (let i = 0; i < 120; i += 1) { + writeReceipt({ home, sink: `s${i}`, host: 'h', payloadClass: 'c', consent: 'k=v' }); + if (i % 10 === 0) writeOutcome({ home, receipt: 'f'.repeat(64), status: 200 }); + } + const result = verifyLedger(home); + expect(result.ok).toBe(true); + expect(result.count).toBe(132); + expect(listReceipts(home).length).toBe(120); + }); + + test('WARN-at-size: >25MB ledger emits one self-explanatory stderr warning per process', () => { + const ledger = egressLedgerPath(home); + fs.mkdirSync(path.dirname(ledger), { recursive: true, mode: 0o700 }); + // Grow the file past the threshold with valid-looking filler; the warning + // keys off file size only. + const filler = `${JSON.stringify({ type: 'egress', pad: 'x'.repeat(1024) })}\n`; + const chunk = filler.repeat(1024); // ~1MB + const writes = Math.ceil(LEDGER_WARN_BYTES / chunk.length) + 1; + for (let i = 0; i < writes; i += 1) fs.appendFileSync(ledger, chunk); + expect(fs.statSync(ledger).size).toBeGreaterThan(LEDGER_WARN_BYTES); + + resetLedgerSizeWarningForTests(); + const captured: string[] = []; + const originalWrite = process.stderr.write.bind(process.stderr); + (process.stderr as any).write = (chunk: string) => { captured.push(String(chunk)); return true; }; + try { + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'k=v' }); + } finally { + (process.stderr as any).write = originalWrite; + } + const warnings = captured.filter((c) => c.includes('egress ledger is large')); + expect(warnings.length).toBe(1); // once per process, not per write + // Self-explanatory shape: what the ledger is, how to inspect, what's coming. + expect(warnings[0]).toContain('ATTEMPTS to send off-machine'); + expect(warnings[0]).toContain('gstack-egress list'); + expect(warnings[0]).toContain('rotation'); + expect(warnings[0]).toContain(ledger); + + // verifyLedger surfaces the same warning as data. + const message = ledgerSizeWarning(ledger, fs.statSync(ledger).size); + expect(message).toContain('MB'); + expect(verifyLedger(home).sizeWarning).toBe(message); + }); +}); + +describe('gstack-egress-receipt shell bridge', () => { + const bin = path.join(ROOT, 'bin', 'gstack-egress-receipt'); + + test('write hashes the exact payload file, prints the receipt id; outcome joins', () => { + const payload = path.join(home, 'payload.json'); + fs.writeFileSync(payload, '[{"v":1}]'); + const write = spawnSync(bin, ['write', '--sink', 'telemetry-sync', '--host', '127.0.0.1:8399', + '--class', 'telemetry-events', '--payload-file', payload, '--consent', 'telemetry=community'], + { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + expect(write.status).toBe(0); + const id = write.stdout.trim(); + expect(id).toMatch(/^[0-9a-f]{64}$/); + const outcome = spawnSync(bin, ['outcome', id, '204'], + { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + expect(outcome.status).toBe(0); + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].bytes).toBe(9); + expect(receipts[0].sha256).toBe(sha256Hex('[{"v":1}]')); + expect(receipts[0].status).toBe('204'); + }); + + test('--no-payload records sha256:null (git-class: a subprocess owns the bytes)', () => { + const write = spawnSync(bin, ['write', '--sink', 'brain-sync', '--host', 'github.com', + '--class', 'git-push', '--no-payload', '--consent', 'artifacts_sync_mode=auto'], + { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + expect(write.status).toBe(0); + const receipts = listReceipts(home); + expect(receipts.length).toBe(1); + expect(receipts[0].sha256).toBeNull(); + expect(receipts[0].bytes).toBe(0); + }); + + test('write exits 3 with EGRESS_RECEIPT_FAILED when the ledger is unwritable', () => { + if (process.platform === 'win32' || process.getuid?.() === 0) return; + fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 }); + const write = spawnSync(bin, ['write', '--sink', 's', '--host', 'h', '--class', 'c', '--no-payload'], + { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + expect(write.status).toBe(3); + expect(write.stderr).toContain('EGRESS_RECEIPT_FAILED'); + fs.chmodSync(path.join(home, 'security'), 0o700); + }); +}); diff --git a/test/fixtures/context-bill/tree-a/alpha/SKILL.md b/test/fixtures/context-bill/tree-a/alpha/SKILL.md new file mode 100644 index 000000000..98f55e48d --- /dev/null +++ b/test/fixtures/context-bill/tree-a/alpha/SKILL.md @@ -0,0 +1,21 @@ +--- +name: alpha +description: Fixture dispatcher with a mode table and forced-read references. +triggers: a-key-the-upstream-router-DOES-read +x-dead-key: a-key-no-router-reads +--- + +# Alpha + +## Dispatch protocol + +1. Infer the mode from the request. +2. Read `references/CORE.md` and `references/POLICY.md` for every invocation. Read `references/OPTIONAL.md` before public-web work. +3. When the target is a repository, read `references/CORE.md` once before specialist work; it is already billed eagerly. + +## Top-level modes + +| Mode | Infer when | Candidate internal specialists | +|---|---|---| +| `Discovery` | The idea is fluid. | `references/legacy/office.md` | +| `Full chain` | Everything at once. | `references/legacy/auto.md` | diff --git a/test/fixtures/context-bill/tree-a/alpha/references/CORE.md b/test/fixtures/context-bill/tree-a/alpha/references/CORE.md new file mode 100644 index 000000000..c81ca42ad --- /dev/null +++ b/test/fixtures/context-bill/tree-a/alpha/references/CORE.md @@ -0,0 +1,3 @@ +# Core + +Forced-read reference number one. Read on every invocation of alpha. diff --git a/test/fixtures/context-bill/tree-a/alpha/references/OPTIONAL.md b/test/fixtures/context-bill/tree-a/alpha/references/OPTIONAL.md new file mode 100644 index 000000000..d83a78384 --- /dev/null +++ b/test/fixtures/context-bill/tree-a/alpha/references/OPTIONAL.md @@ -0,0 +1,3 @@ +# Optional + +Conditional reference. Not part of the eager bill. diff --git a/test/fixtures/context-bill/tree-a/alpha/references/POLICY.md b/test/fixtures/context-bill/tree-a/alpha/references/POLICY.md new file mode 100644 index 000000000..54b723c51 --- /dev/null +++ b/test/fixtures/context-bill/tree-a/alpha/references/POLICY.md @@ -0,0 +1,3 @@ +# Policy + +Forced-read reference number two. diff --git a/test/fixtures/context-bill/tree-a/beta/SKILL.md b/test/fixtures/context-bill/tree-a/beta/SKILL.md new file mode 100644 index 000000000..5b52c3359 --- /dev/null +++ b/test/fixtures/context-bill/tree-a/beta/SKILL.md @@ -0,0 +1,8 @@ +--- +name: beta +description: Clean fixture tool skill with no forced reads and no mode table. +--- + +# Beta + +One-command tool skill. Nothing eager beyond this file, nothing lazy. diff --git a/test/fixtures/context-bill/tree-a/beta/agents.md b/test/fixtures/context-bill/tree-a/beta/agents.md new file mode 100644 index 000000000..01aa0a02f --- /dev/null +++ b/test/fixtures/context-bill/tree-a/beta/agents.md @@ -0,0 +1,3 @@ +# Foreign host file + +A skill-shaped file another host dropped into scanner scope (#1694). diff --git a/test/gbrain-refresh-install-render.test.ts b/test/gbrain-refresh-install-render.test.ts index f1494d47d..0b803ed2a 100644 --- a/test/gbrain-refresh-install-render.test.ts +++ b/test/gbrain-refresh-install-render.test.ts @@ -9,13 +9,18 @@ import * as fs from 'fs'; const ROOT = path.resolve(import.meta.dir, '..'); const SRC = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-config'), 'utf-8'); -// Pull out just the gbrain-refresh `ok)` branch so assertions can't be -// satisfied by unrelated text elsewhere in the file. +// Pull out just the gbrain-refresh healthy-status branch so assertions can't +// be satisfied by unrelated text elsewhere in the file. The case label grew +// from `ok)` to `ok|timeout|thin-client)` (#1964, #2051) and may grow again, +// so match any label that STARTS with `ok` followed by alternations. function okBranch(): string { const start = SRC.indexOf('gbrain-refresh)'); - const ok = SRC.indexOf('ok)', start); + if (start < 0) throw new Error('Could not locate gbrain-refresh case'); + const labelMatch = /^\s*ok(?:\|[\w-]+)*\)/m.exec(SRC.slice(start)); + if (!labelMatch) throw new Error('Could not locate gbrain-refresh ok) branch'); + const ok = start + labelMatch.index; const end = SRC.indexOf(';;', ok); - if (start < 0 || ok < 0 || end < 0) throw new Error('Could not locate gbrain-refresh ok) branch'); + if (end < 0) throw new Error('Could not locate gbrain-refresh ok) branch terminator'); return SRC.slice(ok, end); } diff --git a/test/gemini-e2e.test.ts b/test/gemini-e2e.test.ts index 307665ee6..afbb07f83 100644 --- a/test/gemini-e2e.test.ts +++ b/test/gemini-e2e.test.ts @@ -33,18 +33,42 @@ const GEMINI_AVAILABLE = (() => { } catch { return false; } })(); +// A binary on PATH is not enough: the CLI can be present but UNUSABLE — the +// individual code-assist auth path was deprecated upstream ("migrate to the +// Antigravity suite"), which fails every run before any model call, and flag +// churn (--skip-trust removed in 0.34) errors at argv parse. Probe with a +// bare --help: a CLI that can't even print usage is unusable, and a working +// one is cheap to confirm. Deeper auth failures are classified per-run. +const GEMINI_USABLE = GEMINI_AVAILABLE && (() => { + try { + const result = Bun.spawnSync(['gemini', '--help'], { timeout: 15_000 }); + return result.exitCode === 0; + } catch { return false; } +})(); + const evalsEnabled = !!process.env.EVALS; -// Skip all tests if gemini is not available or EVALS is not set. -const SKIP = !GEMINI_AVAILABLE || !evalsEnabled; +// External-service tests are periodic-tier (CLAUDE.md tiering rule 3): +// "Requires external service (Codex, Gemini)? -> periodic". The positive +// form below is the canonical whole-file guard shape — the sharded runner's +// classifyPaidTestFile greps for it to exclude this file from gate. +const tierOk = process.env.EVALS_TIER === 'periodic'; + +// Skip all tests if gemini is not available/usable, EVALS is not set, or +// we're in the gate tier. +const SKIP = !GEMINI_USABLE || !evalsEnabled || !tierOk; const describeGemini = SKIP ? describe.skip : describe; // Log why we're skipping (helpful for debugging CI) if (!evalsEnabled) { // Silent — same as Claude E2E tests, EVALS=1 required +} else if (!tierOk) { + process.stderr.write('\nGemini E2E: SKIPPED — external-service test, periodic tier only (EVALS_TIER === \'periodic\')\n'); } else if (!GEMINI_AVAILABLE) { process.stderr.write('\nGemini E2E: SKIPPED — gemini binary not found (install: npm i -g @google/gemini-cli)\n'); +} else if (!GEMINI_USABLE) { + process.stderr.write('\nGemini E2E: SKIPPED — gemini CLI present but unusable (auth path deprecated upstream or CLI broken; try updating @google/gemini-cli)\n'); } // --- Diff-based test selection --- diff --git a/test/gstack-egress-cli.test.ts b/test/gstack-egress-cli.test.ts new file mode 100644 index 000000000..4881de80f --- /dev/null +++ b/test/gstack-egress-cli.test.ts @@ -0,0 +1,151 @@ +/** + * gstack-egress CLI — list | verify | grants smoke tests. Free tier. + * + * Spawns the real bin against a temp GSTACK_HOME: list filters, verify + * exit-3-on-tamper (naming the first broken line), sizeWarning surfacing, + * and grants against the upstream config keys (telemetry, + * artifacts_sync_mode, redact_repo_visibility, redact_prepush_hook). + */ + +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { spawnSync } from 'child_process'; +import { + LEDGER_WARN_BYTES, + egressLedgerPath, + sha256Hex, + writeReceipt, +} from '../lib/egress-receipt'; + +const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); +const BIN = path.join(ROOT, 'bin', 'gstack-egress'); + +let home: string; + +beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-egress-cli-')); +}); + +afterEach(() => { + fs.rmSync(home, { recursive: true, force: true }); +}); + +function run(args: string[]) { + const result = spawnSync(BIN, args, { + encoding: 'utf-8', + env: { ...process.env, GSTACK_HOME: home }, + }); + return { code: result.status ?? -1, stdout: result.stdout || '', stderr: result.stderr || '' }; +} + +describe('gstack-egress list', () => { + test('fresh home prints "no receipts" and the ledger path, exit 0', () => { + const r = run(['list']); + expect(r.code).toBe(0); + expect(r.stdout).toContain('no receipts'); + expect(r.stdout).toContain(egressLedgerPath(home)); + }); + + test('--json returns receipts and honors --sink/--host/--since filters', () => { + writeReceipt({ home, sink: 'telemetry-sync', host: '10.0.0.1:8399', payloadClass: 'telemetry-events', bytes: 2, sha256: sha256Hex('[]'), consent: 'telemetry=community' }); + writeReceipt({ home, sink: 'design-openai', host: 'api.openai.com', payloadClass: 'generate-image-request', consent: 'user ran design command' }); + const all = run(['list', '--json']); + expect(all.code).toBe(0); + expect(JSON.parse(all.stdout).length).toBe(2); + const filtered = run(['list', '--json', '--sink', 'telemetry-sync']); + const rows = JSON.parse(filtered.stdout); + expect(rows.length).toBe(1); + expect(rows[0].host).toBe('10.0.0.1:8399'); + expect(rows[0].sha256).toBe(sha256Hex('[]')); + const none = run(['list', '--json', '--since', '2999-01-01T00:00:00Z']); + expect(JSON.parse(none.stdout).length).toBe(0); + }); + + test('unknown option exits 2 with usage', () => { + const r = run(['list', '--bogus']); + expect(r.code).toBe(2); + expect(r.stderr).toContain('Usage'); + }); +}); + +describe('gstack-egress verify', () => { + test('exits 0 on an intact chain and 3 naming the first broken line on tamper', () => { + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'telemetry=community' }); + writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'telemetry=community' }); + const ok = run(['verify']); + expect(ok.code).toBe(0); + expect(ok.stdout).toContain('chain intact: 2'); + + const ledger = egressLedgerPath(home); + const lines = fs.readFileSync(ledger, 'utf-8').trim().split('\n'); + lines[0] = lines[0].replace('community', 'communitX'); + fs.writeFileSync(ledger, `${lines.join('\n')}\n`); + const tampered = run(['verify']); + expect(tampered.code).toBe(3); + expect(tampered.stdout).toContain('line 2'); + }); + + test('prints the sizeWarning when the ledger exceeds the threshold', () => { + writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }); + const ledger = egressLedgerPath(home); + // verify keys the warning off file size only — pad with a trailing + // comment-free blank region by appending to a side channel is not + // possible in JSONL, so grow via many valid-shaped junk lines and + // assert on sizeWarning presence (chain will break; both surface). + const filler = `${JSON.stringify({ type: 'junk', pad: 'x'.repeat(1024) })}\n`.repeat(1024); + while (fs.statSync(ledger).size <= LEDGER_WARN_BYTES) fs.appendFileSync(ledger, filler); + const r = run(['verify', '--json']); + expect(r.code).toBe(3); // filler breaks the chain — expected + const parsed = JSON.parse(r.stdout); + expect(parsed.sizeWarning).toContain('egress ledger is large'); + expect(parsed.sizeWarning).toContain('gstack-egress list'); + const human = run(['verify']); + expect(human.stdout).toContain('egress ledger is large'); + }); +}); + +describe('gstack-egress grants', () => { + test('fresh home shows the four upstream grants off, each naming file and revoke command', () => { + const r = run(['grants']); + expect(r.code).toBe(0); + for (const grant of ['telemetry', 'brain-sync', 'redact_repo_visibility', 'redact_prepush_hook']) { + expect(r.stdout).toContain(grant); + } + expect(r.stdout).not.toContain('[GRANTED]'); + expect(r.stdout).toContain(path.join(home, 'config.yaml')); + expect(r.stdout).toContain('revoke:'); + }); + + test('--json flips granted=true when telemetry and sync mode are enabled', () => { + const config = spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'telemetry', 'community'], { + encoding: 'utf-8', + env: { ...process.env, GSTACK_HOME: home }, + }); + expect(config.status).toBe(0); + spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'artifacts_sync_mode', 'full'], { + encoding: 'utf-8', + env: { ...process.env, GSTACK_HOME: home }, + }); + const r = run(['grants', '--json']); + expect(r.code).toBe(0); + const grants = JSON.parse(r.stdout); + const telemetry = grants.find((g: any) => g.grant === 'telemetry'); + expect(telemetry.granted).toBe(true); + expect(telemetry.revoke).toContain('telemetry off'); + const sync = grants.find((g: any) => g.grant === 'brain-sync'); + expect(sync.granted).toBe(true); + expect(sync.value).toBe('full'); + const hook = grants.find((g: any) => g.grant === 'redact_prepush_hook'); + expect(hook.granted).toBe(false); + }); +}); + +describe('gstack-egress usage', () => { + test('no subcommand exits 2 with usage', () => { + const r = run([]); + expect(r.code).toBe(2); + expect(r.stderr).toContain('Usage'); + }); +}); diff --git a/test/gstack-gbrain-mcp-verify.test.ts b/test/gstack-gbrain-mcp-verify.test.ts index 3705230e1..4461d4dd5 100644 --- a/test/gstack-gbrain-mcp-verify.test.ts +++ b/test/gstack-gbrain-mcp-verify.test.ts @@ -51,13 +51,20 @@ function makeFakeCurl(opts: { printf 'CURL_CALL '"'"'%s'"'"' ' "$@" >> "${curlCallLog}" echo "" >> "${curlCallLog}" -# Walk argv to find -o and -d . +# Walk argv to find -o and the request body (-d or the +# receipted --data-binary @ shape). out="" data="" while [ $# -gt 0 ]; do case "$1" in -o) out="$2"; shift 2 ;; -d) data="$2"; shift 2 ;; + --data-binary) + case "$2" in + @*) data="$(cat "\${2#@}" 2>/dev/null)" ;; + *) data="$2" ;; + esac + shift 2 ;; *) shift ;; esac done @@ -83,6 +90,8 @@ function runVerify(token: string, url: string): { code: number; stdout: string; ...process.env, PATH: `${fakeBinDir}:${process.env.PATH}`, GBRAIN_MCP_TOKEN: token, + // The probe writes egress receipts — keep them in the temp home. + GSTACK_HOME: tmpDir, }, encoding: 'utf-8', }); diff --git a/test/helpers/claude-pty-runner.ts b/test/helpers/claude-pty-runner.ts index 516840126..669dacb48 100644 --- a/test/helpers/claude-pty-runner.ts +++ b/test/helpers/claude-pty-runner.ts @@ -24,7 +24,7 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; -import { hermeticChildEnv, isHermeticEnabled } from './hermetic-env'; +import { hermeticChildEnv, hermeticSkillsConfigDir, isHermeticEnabled } from './hermetic-env'; /** Strip ANSI escapes for pattern-matching against visible text. */ export function stripAnsi(s: string): string { @@ -61,6 +61,11 @@ export function resolveClaudeBinary(): string | null { } export interface ClaudePtyOptions { + /** Register the repo's shipped skills in the child's user scope via + * hermeticSkillsConfigDir(). Required by any test that types a /skill + * slash command; without it hermetic claude rejects the command as + * Unknown before any model turn. No effect when EVALS_HERMETIC=0. */ + seedSkills?: boolean; /** * Permission mode for the session. * - 'plan' (default) — launches with --permission-mode plan @@ -300,6 +305,17 @@ export function isPermissionDialogVisible(visible: string): boolean { } /** Detect any AskUserQuestion-shaped numbered option list with cursor. */ +/** + * Strip terminal residue that survives ANSI-stripping and can interleave + * with AUQ text: DEC cursor-visibility fragments (`[?25l` / `[?25h` — the ESC + * byte is gone but the bracket sequence remains) and the spinner frames + * rendered between them. Observed in plan-design-with-ui's failure buffer, + * where `[?25l✻Sprouting…[?25h` fragments sat inside the option lines. + */ +export function stripPtyResidue(visible: string): string { + return visible.replace(/\[\?25[lh]/g, ''); +} + export function isNumberedOptionListVisible(visible: string): boolean { // ❯ cursor + at least two numbered options 1-9. // Matches the trust dialog AND plan-ready prompt AND skill questions. @@ -311,7 +327,8 @@ export function isNumberedOptionListVisible(visible: string): boolean { // because `t-2` is a word-to-word transition. We use the weaker // `[^0-9]2\.` to require a non-digit before `2` (so we don't match // `12.0`) without requiring whitespace. - return /❯\s*1\./.test(visible) && /(^|[^0-9])2\./.test(visible); + const cleaned = stripPtyResidue(visible); + return /❯\s*1\./.test(cleaned) && /(^|[^0-9])2\./.test(cleaned); } // ──────────────────────────────────────────────────────────────────────────── @@ -687,6 +704,7 @@ export function isScopeGateAutoSelectVisible(visible: string): boolean { export function parseNumberedOptions( visible: string, ): Array<{ index: number; label: string }> { + visible = stripPtyResidue(visible); const tail = visible.length > 4096 ? visible.slice(-4096) : visible; // Split on lines, look for `❯ N.` or ` N.` patterns. Up to N=9. // The `\s*` after `.` (not `\s+`) is required because stripAnsi removes @@ -728,30 +746,41 @@ export function parseNumberedOptions( const seenIndices = new Set(); // Cursor line: option 1 may be inline after box dividers + prompt header - // (`...divider...header...❯1. label`). Use a non-anchored regex that - // captures `❯N. label` from anywhere on the line through end-of-line. - // Only used for the cursor line — subsequent options are parsed with the - // start-of-line `optionRe`. + // (`...divider...header...❯1. label`) — and, when the PTY reflows the whole + // AUQ onto ONE logical line, options 2..N sit on the SAME line after it + // (observed with /plan-design-review's Step-0 scope gate: `❯1.Branch diff + // ... 2.Plan or design doc ... 5.Chat about this ... Enter to select`). + // Parse the cursor line as a STREAM: find every `N.` token (not preceded + // by a digit, not followed by one — excludes "12." and "1.5"), require + // ascending indices starting from the cursor's option, and take each + // label as the text between successive number tokens. const cursorLine = lines[cursorLineIdx] ?? ''; - const cursorInlineRe = /❯\s*([1-9])\.\s*(\S.*?)\s*$/; - const inlineMatch = cursorInlineRe.exec(cursorLine); - if (inlineMatch) { - const idx = Number(inlineMatch[1]); - const label = (inlineMatch[2] ?? '').trim(); - if (label.length > 0 && !seenIndices.has(idx)) { - seenIndices.add(idx); - found.push({ index: idx, label }); - } - } else { - // No inline cursor match — fall back to start-of-line regex. - const startMatch = optionRe.exec(cursorLine); - if (startMatch) { - const idx = Number(startMatch[1]); - const label = (startMatch[2] ?? '').trim(); - if (label.length > 0 && !seenIndices.has(idx)) { - seenIndices.add(idx); - found.push({ index: idx, label }); - } + const cursorStart = cursorLine.indexOf('❯'); + const cursorSegment = cursorStart >= 0 ? cursorLine.slice(cursorStart) : cursorLine; + const tokenRe = /(?:^|[^0-9])([1-9])\.(?!\d)\s*/g; + const tokens: Array<{ idx: number; labelStart: number; matchStart: number }> = []; + for (let m = tokenRe.exec(cursorSegment); m !== null; m = tokenRe.exec(cursorSegment)) { + tokens.push({ + idx: Number(m[1]), + labelStart: m.index + m[0].length, + matchStart: m.index === 0 ? 0 : m.index + 1, // skip the [^0-9] guard char + }); + } + // Keep only the ascending run that starts the sequence (1, 2, 3, ...); + // stray numbers inside labels break ascension and end the run. + let expected = 1; + for (let t = 0; t < tokens.length; t++) { + const token = tokens[t]!; + if (token.idx !== expected) continue; + const next = tokens + .slice(t + 1) + .find((candidate) => candidate.idx === expected + 1 && candidate.matchStart > token.labelStart); + const labelEnd = next ? next.matchStart : cursorSegment.length; + const label = cursorSegment.slice(token.labelStart, labelEnd).trim(); + if (label.length > 0 && !seenIndices.has(token.idx)) { + seenIndices.add(token.idx); + found.push({ index: token.idx, label }); + expected += 1; } } @@ -1285,6 +1314,9 @@ export async function launchClaudePty( // Hermetic by default (test/helpers/hermetic-env.ts): operator session // context never reaches the child; per-test opts.env merges last. const childEnv = hermeticChildEnv(opts.env); + if (opts.seedSkills && hermetic && !opts.env?.CLAUDE_CONFIG_DIR) { + childEnv.CLAUDE_CONFIG_DIR = hermeticSkillsConfigDir(); + } // eslint-disable-next-line @typescript-eslint/no-explicit-any const proc = (Bun as any).spawn([claudePath, ...args], { @@ -1670,6 +1702,7 @@ export async function runPlanSkillObservation(opts: { extraArgs: opts.extraArgs, env: opts.env, model: opts.model, + seedSkills: true, }); try { @@ -1966,6 +1999,7 @@ export async function runPlanSkillCounting(opts: { timeoutMs: timeoutMs + 60_000, env: opts.env, model: opts.model, + seedSkills: true, }); const fingerprints: AskUserQuestionFingerprint[] = []; @@ -2199,6 +2233,7 @@ export async function runPlanSkillFloorCheck(opts: { timeoutMs: timeoutMs + 60_000, env: opts.env, model: opts.model, + seedSkills: true, }); try { diff --git a/test/helpers/codex-session-runner.ts b/test/helpers/codex-session-runner.ts index 404aa6bb2..6aa2ba74f 100644 --- a/test/helpers/codex-session-runner.ts +++ b/test/helpers/codex-session-runner.ts @@ -199,8 +199,12 @@ export async function runCodexSkill(opts: { } } - // Build codex exec command - const args = ['exec', prompt, '--json', '-s', sandbox]; + // Build codex exec command. + // --skip-git-repo-check: newer codex CLIs refuse exec in an untrusted + // non-git directory ("Not inside a trusted directory and + // --skip-git-repo-check was not specified") — our temp skill dirs are + // exactly that. Empirically verified against codex on this machine. + const args = ['exec', prompt, '--json', '-s', sandbox, '--skip-git-repo-check']; // Spawn codex with temp HOME so it discovers our installed skill. // Hermetic scrub (test/helpers/hermetic-env.ts) with codex's auth surface diff --git a/test/helpers/eval-store.test.ts b/test/helpers/eval-store.test.ts index c6aff1c95..540143aad 100644 --- a/test/helpers/eval-store.test.ts +++ b/test/helpers/eval-store.test.ts @@ -6,6 +6,9 @@ import { EvalCollector, extractToolSummary, findPreviousRun, + findLatestFinalizedRun, + isPartialEval, + listEvalJsonFiles, compareEvalResults, formatComparison, generateCommentary, @@ -58,6 +61,19 @@ function makeResult(overrides?: Partial): EvalResult { }; } +/** Capture everything a block writes to stderr (finalize prints there). */ +async function captureStderr(fn: () => Promise): Promise { + const original = process.stderr.write.bind(process.stderr); + let captured = ''; + (process.stderr as any).write = (chunk: any) => { captured += String(chunk); return true; }; + try { + await fn(); + } finally { + (process.stderr as any).write = original; + } + return captured; +} + // --- EvalCollector tests --- describe('EvalCollector', () => { @@ -119,6 +135,41 @@ describe('EvalCollector', () => { expect(fs.readdirSync(tmpDir).filter(f => f.endsWith('.json') && !f.startsWith('_partial'))).toHaveLength(1); }); + test('with no completed prior run, says NO BASELINE instead of comparing against its own partial', async () => { + // addTest writes the in-progress accumulator into the same dir. If that + // counted as a baseline, the run would compare against itself and print a + // reassuring all-clear forever. + const collector = new EvalCollector('e2e', tmpDir); + collector.addTest(makeEntry({ name: 'test-1', passed: true })); + + const output = await captureStderr(async () => { await collector.finalize(); }); + + expect(fs.existsSync(path.join(tmpDir, '_partial-e2e.json'))).toBe(true); // the trap exists + expect(output).toContain('NO BASELINE'); + expect(output).not.toContain('vs previous'); + expect(output).not.toContain('Stable run'); + }); + + test('with a genuine prior run, reports the real delta', async () => { + fs.writeFileSync( + path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'), + JSON.stringify(makeResult({ + timestamp: '2026-03-12T10:00:00Z', + tests: [makeEntry({ name: 'test-1', passed: true, turns_used: 5 })], + })), + ); + + const collector = new EvalCollector('e2e', tmpDir); + collector.addTest(makeEntry({ name: 'test-1', passed: false, turns_used: 5 })); + + const output = await captureStderr(async () => { await collector.finalize(); }); + + expect(output).toContain('vs previous'); + expect(output).toContain('REGRESSION'); + expect(output).toContain('1 regressed'); + expect(output).not.toContain('NO BASELINE'); + }); + test('empty collector writes valid file', async () => { const collector = new EvalCollector('llm-judge', tmpDir); const filepath = await collector.finalize(); @@ -131,6 +182,59 @@ describe('EvalCollector', () => { }); }); +// --- GSTACK_EVAL_DIR + shard slug tests --- + +describe('EvalCollector eval-dir resolution', () => { + const savedEnv = process.env.GSTACK_EVAL_DIR; + + afterEach(() => { + if (savedEnv === undefined) delete process.env.GSTACK_EVAL_DIR; + else process.env.GSTACK_EVAL_DIR = savedEnv; + }); + + test('honors GSTACK_EVAL_DIR set after import — no --preload needed', async () => { + // The default eval dir must resolve lazily at construction, not at module + // load: the sharded runner sets GSTACK_EVAL_DIR in each shard child's env + // and shard tests import this module long before any collector exists. + const envDir = path.join(tmpDir, 'env-dir'); + process.env.GSTACK_EVAL_DIR = envDir; + const collector = new EvalCollector('e2e'); + collector.addTest(makeEntry()); + await captureStderr(async () => { await collector.finalize(); }); + expect(fs.readdirSync(envDir).filter(f => !f.startsWith('_partial'))).toHaveLength(1); + }); + + test('explicit constructor arg beats GSTACK_EVAL_DIR', async () => { + process.env.GSTACK_EVAL_DIR = path.join(tmpDir, 'env-dir'); + const explicit = path.join(tmpDir, 'explicit'); + const collector = new EvalCollector('e2e', explicit); + collector.addTest(makeEntry()); + await captureStderr(async () => { await collector.finalize(); }); + expect(fs.existsSync(path.join(tmpDir, 'env-dir'))).toBe(false); + expect(fs.readdirSync(explicit).length).toBeGreaterThan(0); + }); + + test('writes the shard slug when the eval dir is a shards/ subdir', async () => { + const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + const collector = new EvalCollector('e2e', shardDir); + collector.addTest(makeEntry()); + await captureStderr(async () => { await collector.finalize(); }); + + const partial = JSON.parse(fs.readFileSync(path.join(shardDir, '_partial-e2e.json'), 'utf-8')); + expect(partial.shard).toBe('skill-e2e-qa'); + const final = fs.readdirSync(shardDir).find(f => !f.startsWith('_partial'))!; + expect(JSON.parse(fs.readFileSync(path.join(shardDir, final), 'utf-8')).shard).toBe('skill-e2e-qa'); + }); + + test('writes no shard slug for a flat eval dir', async () => { + const collector = new EvalCollector('e2e', tmpDir); + collector.addTest(makeEntry()); + await captureStderr(async () => { await collector.finalize(); }); + const final = fs.readdirSync(tmpDir).find(f => f.endsWith('.json') && !f.startsWith('_partial'))!; + expect(JSON.parse(fs.readFileSync(path.join(tmpDir, final), 'utf-8')).shard).toBeUndefined(); + }); +}); + // --- judgePassed tests --- describe('judgePassed', () => { @@ -259,6 +363,34 @@ describe('findPreviousRun', () => { expect(result).toBeNull(); // only file is excluded }); + test('never returns the in-progress accumulator as a baseline', () => { + // The current run's own partial carries the current tier + branch and the + // freshest timestamp. If it were a candidate, every run would compare + // against itself and report "no regressions" forever. + fs.writeFileSync( + path.join(tmpDir, '_partial-e2e.json'), + JSON.stringify(makeResult({ branch: 'main', timestamp: '2026-03-14T10:00:00Z', _partial: true })), + ); + + const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json')); + expect(result).toBeNull(); + }); + + test('prefers a completed run over a newer in-progress accumulator', () => { + fs.writeFileSync( + path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'), + JSON.stringify(makeResult({ branch: 'main', timestamp: '2026-03-12T10:00:00Z' })), + ); + // Newer, same tier + branch, but in-progress — must lose to the older completed run. + fs.writeFileSync( + path.join(tmpDir, '_partial-e2e.json'), + JSON.stringify(makeResult({ branch: 'main', timestamp: '2026-03-14T10:00:00Z', _partial: true })), + ); + + const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json')); + expect(result).toContain('0.3.5-main-e2e'); + }); + test('filters by tier', () => { fs.writeFileSync( path.join(tmpDir, '0.3.6-main-llm-judge-20260314-100000.json'), @@ -268,6 +400,143 @@ describe('findPreviousRun', () => { const result = findPreviousRun(tmpDir, 'e2e', 'main', 'current.json'); expect(result).toBeNull(); // only llm-judge file, looking for e2e }); + + test('a shard run prefers its own shard history over newer other-shard or flat priors', () => { + const mine = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + const other = path.join(tmpDir, 'shards', 'codex-e2e'); + fs.mkdirSync(mine, { recursive: true }); + fs.mkdirSync(other, { recursive: true }); + // Same-shard prior — oldest of the three, must still win. + fs.writeFileSync( + path.join(mine, '0.3.4-main-e2e-20260311-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-11T10:00:00Z' })), + ); + fs.writeFileSync( + path.join(other, '0.3.5-main-e2e-20260312-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })), + ); + fs.writeFileSync( + path.join(tmpDir, '0.3.6-main-e2e-20260313-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-13T10:00:00Z' })), + ); + + const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(mine, 'current.json')); + expect(result).toContain(path.join('shards', 'skill-e2e-qa')); + }); + + test('a flat run prefers flat history over a newer shard prior', () => { + const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + fs.mkdirSync(shardDir, { recursive: true }); + fs.writeFileSync( + path.join(shardDir, '0.3.6-main-e2e-20260314-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-14T10:00:00Z' })), + ); + fs.writeFileSync( + path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })), + ); + + const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json')); + expect(result).toContain('0.3.5-main-e2e'); + }); + + test('falls back to a shard prior when the flat dir has no candidate', () => { + const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + fs.mkdirSync(shardDir, { recursive: true }); + fs.writeFileSync( + path.join(shardDir, '0.3.6-main-e2e-20260314-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-14T10:00:00Z' })), + ); + + const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json')); + expect(result).toContain(path.join('shards', 'skill-e2e-qa')); + }); +}); + +// --- isPartialEval tests --- + +describe('isPartialEval', () => { + test('flag set but file renamed — still partial', () => { + expect(isPartialEval({ _partial: true }, 'renamed-to-look-final.json')).toBe(true); + }); + + test('name matches but no flag — still partial', () => { + expect(isPartialEval({}, '_partial-e2e.json')).toBe(true); + expect(isPartialEval(null, path.join('/some/dir', '_partial-e2e.json'))).toBe(true); + }); + + test('finalized run is not partial', () => { + expect(isPartialEval(makeResult(), '0.3.6-main-e2e-20260314-100000.json')).toBe(false); + }); +}); + +// --- listEvalJsonFiles / findLatestFinalizedRun tests --- + +describe('findLatestFinalizedRun', () => { + test('listEvalJsonFiles recurses exactly one shards/*/ level', () => { + fs.writeFileSync(path.join(tmpDir, 'flat.json'), '{}'); + const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + fs.mkdirSync(shardDir, { recursive: true }); + fs.writeFileSync(path.join(shardDir, 'sharded.json'), '{}'); + // Nested one level deeper than the contract — must NOT be picked up. + const tooDeep = path.join(shardDir, 'shards', 'nested'); + fs.mkdirSync(tooDeep, { recursive: true }); + fs.writeFileSync(path.join(tooDeep, 'too-deep.json'), '{}'); + fs.writeFileSync(path.join(tmpDir, 'not-json.txt'), ''); + + const files = listEvalJsonFiles(tmpDir).map(f => path.basename(f)).sort(); + expect(files).toEqual(['flat.json', 'sharded.json']); + }); + + test('finds the newest finalized run across flat dir and shard subdirs', () => { + fs.writeFileSync( + path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })), + ); + const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + fs.mkdirSync(shardDir, { recursive: true }); + fs.writeFileSync( + path.join(shardDir, '0.3.6-main-e2e-20260314-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-14T10:00:00Z' })), + ); + + const latest = findLatestFinalizedRun(tmpDir, 'e2e'); + expect(latest?.filepath).toContain('skill-e2e-qa'); + expect(latest?.result.timestamp).toBe('2026-03-14T10:00:00Z'); + }); + + test('skips partials by flag and by name, filters by tier', () => { + // Newest by timestamp, but partial by flag under a shard dir. + const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa'); + fs.mkdirSync(shardDir, { recursive: true }); + fs.writeFileSync( + path.join(shardDir, 'flagged.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-16T10:00:00Z', _partial: true })), + ); + // Partial by name only. + fs.writeFileSync( + path.join(tmpDir, '_partial-e2e.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-15T10:00:00Z' })), + ); + // Wrong tier. + fs.writeFileSync( + path.join(tmpDir, '0.3.6-main-llm-judge-20260317-100000.json'), + JSON.stringify(makeResult({ tier: 'llm-judge', timestamp: '2026-03-17T10:00:00Z' })), + ); + // The genuine baseline. + fs.writeFileSync( + path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'), + JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })), + ); + + const latest = findLatestFinalizedRun(tmpDir, 'e2e'); + expect(latest?.result.timestamp).toBe('2026-03-12T10:00:00Z'); + }); + + test('returns null for missing dir or no finalized runs', () => { + expect(findLatestFinalizedRun('/nonexistent/path', 'e2e')).toBeNull(); + expect(findLatestFinalizedRun(tmpDir, 'e2e')).toBeNull(); + }); }); // --- compareEvalResults tests --- @@ -524,6 +793,29 @@ describe('generateCommentary', () => { expect(notes.some(n => n.includes('No regressions'))).toBe(true); }); + test('says NO BASELINE instead of "stable" when nothing matched the prior run', () => { + // A baseline file existed but shares no test names (renamed/retired suite), + // so zero tests were actually compared. Claiming stability here is a lie. + const c: ComparisonResult = { + before_file: 'a.json', after_file: 'b.json', + before_branch: 'main', after_branch: 'main', + before_timestamp: '', after_timestamp: '', + deltas: [ + { name: 'a', before: { passed: false, cost_usd: 0 }, after: { passed: true, cost_usd: 0.10 }, status_change: 'unchanged' }, + { name: 'b', before: { passed: false, cost_usd: 0 }, after: { passed: true, cost_usd: 0.10 }, status_change: 'unchanged' }, + { name: 'c', before: { passed: false, cost_usd: 0 }, after: { passed: true, cost_usd: 0.10 }, status_change: 'unchanged' }, + ], + total_cost_delta: 0.30, total_duration_delta: 0, + improved: 0, regressed: 0, unchanged: 3, + tool_count_before: 0, tool_count_after: 0, + matched: 0, + }; + + const notes = generateCommentary(c); + expect(notes.some(n => n.includes('NO BASELINE'))).toBe(true); + expect(notes.some(n => n.includes('Stable run'))).toBe(false); + }); + test('returns empty for stable run with no significant changes', () => { const c: ComparisonResult = { before_file: 'a.json', after_file: 'b.json', diff --git a/test/helpers/eval-store.ts b/test/helpers/eval-store.ts index 9a801ae1c..3f1c8e24c 100644 --- a/test/helpers/eval-store.ts +++ b/test/helpers/eval-store.ts @@ -39,7 +39,16 @@ export function getProjectEvalDir(): string { return LEGACY_EVAL_DIR; } -const DEFAULT_EVAL_DIR = getProjectEvalDir(); +/** + * Lazy + memoized so importing this module never spawns the gstack-slug + * subprocess. Callers that pass an explicit dir or set GSTACK_EVAL_DIR + * (the sharded paid runner does, per shard) never pay for slug detection. + */ +let memoizedDefaultEvalDir: string | null = null; +function defaultEvalDir(): string { + if (memoizedDefaultEvalDir === null) memoizedDefaultEvalDir = getProjectEvalDir(); + return memoizedDefaultEvalDir; +} // --- Interfaces --- @@ -104,6 +113,8 @@ export interface EvalResult { total_duration_ms: number; wall_clock_ms?: number; // wall-clock from collector creation to finalization (shows parallelism) tests: EvalTestEntry[]; + /** Shard slug when the run was collected under /shards//. */ + shard?: string; _partial?: boolean; // true for incremental saves, absent in final } @@ -131,10 +142,94 @@ export interface ComparisonResult { unchanged: number; tool_count_before: number; tool_count_after: number; + /** After-tests that had a same-named entry in the before run. 0 = nothing was + * actually compared, so no stability claim is warranted. */ + matched?: number; } // --- Shared helpers --- +/** + * Is this eval file an in-progress accumulator rather than a finalized run? + * + * True on either signal: the `_partial` flag inside the JSON (the authoritative + * role marker) OR a filename starting with `_partial` (catches accumulators + * whose body predates the flag, and flagged files that were renamed keep being + * caught by the flag). Every baseline lookup must exclude these — an + * accumulator carries the current run's tier, branch, and freshest timestamp, + * so treating it as a baseline makes the run compare against itself. + */ +export function isPartialEval(data: unknown, filename: string): boolean { + if (path.basename(filename).startsWith('_partial')) return true; + return Boolean((data as { _partial?: unknown } | null)?._partial); +} + +/** + * List eval JSON files in `evalDir` plus one level of `/shards//` + * subdirectories (where the sharded paid runner points each shard's collector). + * Returns absolute paths. Missing dirs yield []. + */ +export function listEvalJsonFiles(evalDir: string): string[] { + const jsonIn = (dir: string): string[] => { + let names: string[]; + try { + names = fs.readdirSync(dir); + } catch { + return []; + } + return names.filter(f => f.endsWith('.json')).map(f => path.join(dir, f)); + }; + + const files = jsonIn(evalDir); + const shardsRoot = path.join(evalDir, 'shards'); + let shardDirs: fs.Dirent[]; + try { + shardDirs = fs.readdirSync(shardsRoot, { withFileTypes: true }); + } catch { + return files; + } + for (const entry of shardDirs) { + if (!entry.isDirectory()) continue; + files.push(...jsonIn(path.join(shardsRoot, entry.name))); + } + return files; +} + +/** + * Shard slug for an eval dir: when the dir is directly under a `shards/` + * directory (the sharded paid runner's per-shard GSTACK_EVAL_DIR layout), + * the dir name is the slug; otherwise null. + */ +export function shardSlugOfEvalDir(evalDir: string): string | null { + const normalized = path.resolve(evalDir); + return path.basename(path.dirname(normalized)) === 'shards' ? path.basename(normalized) : null; +} + +/** + * Find the most recent finalized (non-partial) eval file for a tier, scanning + * `evalDir` and one level of `shards//` subdirs. Shared by the budget + * regression gate and any tooling that needs "the latest real run". + */ +export function findLatestFinalizedRun( + evalDir: string, + tier: 'e2e' | 'llm-judge', +): { filepath: string; result: EvalResult } | null { + let latest: { filepath: string; result: EvalResult; timestamp: string } | null = null; + for (const filepath of listEvalJsonFiles(evalDir)) { + let data: EvalResult; + try { + data = JSON.parse(fs.readFileSync(filepath, 'utf-8')) as EvalResult; + } catch { continue; } + if (isPartialEval(data, filepath)) continue; + if (data.tier !== tier) continue; + const timestamp = data.timestamp ?? ''; + if (!latest || timestamp.localeCompare(latest.timestamp) > 0) { + latest = { filepath, result: data, timestamp }; + } + } + return latest ? { filepath: latest.filepath, result: latest.result } : null; +} + /** * Determine if a planted-bug eval passed based on judge results vs ground truth thresholds. * Centralizes the pass/fail logic so all planted-bug tests use the same criteria. @@ -171,8 +266,16 @@ export function extractToolSummary(transcript: any[]): Record { } /** - * Find the most recent prior eval file for comparison. - * Prefers same branch, falls back to any branch. + * Find the most recent prior COMPLETED eval file for comparison. + * Scans the eval dir plus one level of `shards//` subdirs. Prefers + * same shard slug (a shard's own history over another shard's or the flat + * dir's), then same branch, then falls back to anything. + * + * In-progress accumulators (`_partial: true`, written by savePartial after every + * test) are never candidates: the current run's own partial carries the current + * tier + branch and the freshest timestamp, so including it made every run + * compare against itself and report "no regressions" unconditionally. The + * exclusion is by role (the `_partial` flag), not by filename. */ export function findPreviousRun( evalDir: string, @@ -180,24 +283,22 @@ export function findPreviousRun( branch: string, excludeFile: string, ): string | null { - let files: string[]; - try { - files = fs.readdirSync(evalDir).filter(f => f.endsWith('.json')); - } catch { - return null; // dir doesn't exist - } - // Parse top-level fields from each file (cheap — no full tests array needed) - const entries: Array<{ file: string; branch: string; timestamp: string }> = []; - for (const file of files) { - if (file === path.basename(excludeFile)) continue; - const fullPath = path.join(evalDir, file); + const entries: Array<{ file: string; branch: string; timestamp: string; shard: string | null }> = []; + for (const fullPath of listEvalJsonFiles(evalDir)) { + if (path.resolve(fullPath) === path.resolve(excludeFile)) continue; try { const raw = fs.readFileSync(fullPath, 'utf-8'); // Quick parse — only grab the fields we need const data = JSON.parse(raw); + if (isPartialEval(data, fullPath)) continue; // in-progress run, not a baseline if (data.tier !== tier) continue; - entries.push({ file: fullPath, branch: data.branch || '', timestamp: data.timestamp || '' }); + entries.push({ + file: fullPath, + branch: data.branch || '', + timestamp: data.timestamp || '', + shard: data.shard || shardSlugOfEvalDir(path.dirname(fullPath)), + }); } catch { continue; } } @@ -206,11 +307,17 @@ export function findPreviousRun( // Sort by timestamp descending entries.sort((a, b) => b.timestamp.localeCompare(a.timestamp)); - // Prefer same branch - const sameBranch = entries.find(e => e.branch === branch); - if (sameBranch) return sameBranch.file; - - // Fallback: any branch + // Prefer same shard slug (null = the flat dir), then same branch, then any. + const targetShard = shardSlugOfEvalDir(path.dirname(excludeFile)); + const preferences: Array<(e: typeof entries[number]) => boolean> = [ + e => e.shard === targetShard && e.branch === branch, + e => e.shard === targetShard, + e => e.branch === branch, + ]; + for (const matches of preferences) { + const hit = entries.find(matches); + if (hit) return hit.file; + } return entries[0].file; } @@ -226,6 +333,7 @@ export function compareEvalResults( const deltas: TestDelta[] = []; let improved = 0, regressed = 0, unchanged = 0; let toolCountBefore = 0, toolCountAfter = 0; + let matched = 0; // Index before tests by name const beforeMap = new Map(); @@ -246,6 +354,7 @@ export function compareEvalResults( let statusChange: TestDelta['status_change'] = 'unchanged'; if (beforeTest) { + matched++; if (!beforeTest.passed && afterTest.passed) { statusChange = 'improved'; improved++; } else if (beforeTest.passed && !afterTest.passed) { statusChange = 'regressed'; regressed++; } else { unchanged++; } @@ -314,6 +423,7 @@ export function compareEvalResults( unchanged, tool_count_before: toolCountBefore, tool_count_after: toolCountAfter, + matched, }; } @@ -512,7 +622,17 @@ export function generateCommentary(c: ComparisonResult): string[] { } } - // 4. Overall summary + // 4. No baseline — say so. A run with nothing to compare against must never + // read as "stable"; silence or a false all-clear is worse than no output. + if (c.matched === 0 && c.deltas.length > 0) { + notes.push( + `NO BASELINE: none of these ${c.deltas.length} test(s) appear in ${path.basename(c.before_file)}. ` + + 'Nothing was compared, so this run says nothing about regressions.', + ); + return notes; + } + + // 5. Overall summary if (c.deltas.length >= 3 && regressions.length === 0) { const overallParts: string[] = []; @@ -649,11 +769,13 @@ export class EvalCollector { private tests: EvalTestEntry[] = []; private finalized = false; private evalDir: string; + private shard: string | null; private createdAt = Date.now(); constructor(tier: 'e2e' | 'llm-judge', evalDir?: string) { this.tier = tier; - this.evalDir = evalDir || DEFAULT_EVAL_DIR; + this.evalDir = evalDir || process.env.GSTACK_EVAL_DIR || defaultEvalDir(); + this.shard = shardSlugOfEvalDir(this.evalDir); } addTest(entry: EvalTestEntry): void { @@ -684,6 +806,7 @@ export class EvalCollector { total_cost_usd: Math.round(totalCost * 100) / 100, total_duration_ms: totalDuration, tests: this.tests, + ...(this.shard ? { shard: this.shard } : {}), _partial: true, }; @@ -721,6 +844,7 @@ export class EvalCollector { total_duration_ms: totalDuration, wall_clock_ms: Date.now() - this.createdAt, tests: this.tests, + ...(this.shard ? { shard: this.shard } : {}), }; // Write eval file @@ -742,7 +866,11 @@ export class EvalCollector { const comparison = compareEvalResults(prevResult, result, prevFile, filepath); process.stderr.write(formatComparison(comparison) + '\n'); } else { - process.stderr.write('\nFirst run — no comparison available.\n'); + process.stderr.write( + `\nNO BASELINE: no completed prior ${this.tier} run found in ${this.evalDir}` + + ' (the in-progress accumulator is not a baseline). Nothing compared —' + + ' this run says nothing about regressions.\n', + ); } } catch (err: any) { process.stderr.write(`\nCompare error: ${err.message}\n`); diff --git a/test/helpers/gemini-session-runner.ts b/test/helpers/gemini-session-runner.ts index 3b58c79ca..2b080265d 100644 --- a/test/helpers/gemini-session-runner.ts +++ b/test/helpers/gemini-session-runner.ts @@ -8,8 +8,8 @@ * Key differences from Codex session-runner: * - Uses `gemini -p` instead of `codex exec` * - Output is NDJSON with event types: init, message, tool_use, tool_result, result - * - Uses `--output-format stream-json --yolo --skip-trust` instead of `--json -s read-only` - * (`--skip-trust` required for headless/untrusted cwds; see gemini trusted-folders docs) + * - Uses `--output-format stream-json --yolo` instead of `--json -s read-only` + * (`--skip-trust` was removed in gemini-cli 0.34; folder trust is settings-driven now) * - No temp HOME needed — Gemini discovers skills from `.agents/skills/` in cwd * - Message events are streamed with `delta: true` — must concatenate */ @@ -121,10 +121,11 @@ export async function runGeminiSkill(opts: { }; } - // Build gemini command - // --skip-trust: headless/CI and temp cwds aren't in ~/.gemini/trustedFolders.json; - // without it gemini exits FatalUntrustedWorkspaceError before any model call. - const args = ['-p', prompt, '--output-format', 'stream-json', '--yolo', '--skip-trust']; + // Build gemini command. + // --skip-trust was REMOVED in gemini-cli 0.34 ("Unknown arguments: + // skip-trust"); folder trust moved to settings and no longer needs a flag + // for headless runs. --yolo still auto-approves tool actions. + const args = ['-p', prompt, '--output-format', 'stream-json', '--yolo']; // Spawn gemini — uses real HOME for auth (~/.gemini; HOME is allowlisted), // cwd for skill discovery. Hermetic scrub with gemini's auth surface @@ -198,6 +199,31 @@ export async function runGeminiSkill(opts: { process.stderr.write(` [gemini stderr] ${stderr.trim().slice(0, 200)}\n`); } + // Environment-unusable classification: these are Google-side conditions no + // test assertion can act on — the deprecated individual code-assist auth + // path ("migrate to the Antigravity suite") and argv drift on older/newer + // CLIs. Return the same SKIP shape as binary-not-found so callers report + // SKIPPED instead of a false FAIL. + const unusableMarkers = [ + 'no longer supported for Gemini Code Assist', + 'antigravity', + 'Unknown arguments: skip-trust', + ]; + if (exitCode !== 0 && parsed.tokens === 0) { + const marker = unusableMarkers.find((m) => stderr.toLowerCase().includes(m.toLowerCase())); + if (marker) { + return { + output: `SKIP: gemini CLI unusable (${marker})`, + toolCalls: [], + tokens: 0, + exitCode: -1, + durationMs, + sessionId: null, + rawLines: collectedLines, + }; + } + } + return { output: parsed.output, toolCalls: parsed.toolCalls, diff --git a/test/helpers/hermetic-env.ts b/test/helpers/hermetic-env.ts index def6dd9bc..4fb17e734 100644 --- a/test/helpers/hermetic-env.ts +++ b/test/helpers/hermetic-env.ts @@ -36,7 +36,8 @@ import * as fs from 'fs'; import * as path from 'path'; import * as os from 'os'; import { promotedEnv } from '../../lib/conductor-env-shim'; -import { isProcessAlive } from '../../browse/src/error-handling'; +import { isProcessAlive, safeUnlink } from '../../browse/src/error-handling'; +import { skillCensus, frontmatterName } from './skill-census'; /** Exact env names a hermetic child keeps. Everything not listed (or matched * by a prefix rule below) is dropped. */ @@ -225,6 +226,73 @@ export function getHermeticDirs(): HermeticDirs { return cachedDirs; } +let cachedSkillsConfigDir: string | null = null; + +/** + * A hermetic CLAUDE_CONFIG_DIR with the repo's shipped skills REGISTERED in + * user scope, mirroring ./setup's registration exactly: each discovered skill + * gets a REAL directory `/skills//` containing a + * SYMLINK to that skill's SKILL.md (absolute path), plus a `sections/` + * symlink when the skill has one. registryName is the frontmatter `name:` + * (dir-name fallback), NO gstack- prefix; the root SKILL.md router registers + * as `_gstack-command`. skillCensus().registryEntries is the authoritative + * set of what must appear here. + * + * The default hermetic dir deliberately seeds no skills — correct for + * children that install their own or probe setup behavior — but a PTY test + * that TYPES `/office-hours` needs the slash command to exist, or claude + * rejects it as Unknown command before any model turn and the gate measures + * nothing. Separate dir, same runRoot: opt-in per session, never contaminates + * the default-config children, and the existing exit teardown + pid-aware GC + * cover it. Ends in `/.claude` for the same plan-path anchoring reason as + * HermeticDirs.configDir. + * + * Two intentional non-hermetic edges: + * - Seeding reads the LIVE repo tree BY DESIGN — the skills ARE the subject + * under test; a snapshot would measure stale copies. + * - HOME is not hermeticized, so the ~64 absolute + * `~/.claude/skills/gstack/...` preamble references inside each SKILL.md + * still resolve to the operator install (same limitation as CI). + */ +export function hermeticSkillsConfigDir(): string { + if (cachedSkillsConfigDir) return cachedSkillsConfigDir; + const { runRoot } = getHermeticDirs(); + const configDir = path.join(runRoot, 'with-skills', '.claude'); + const skillsDir = path.join(configDir, 'skills'); + fs.mkdirSync(skillsDir, { recursive: true }); + fs.writeFileSync( + path.join(configDir, '.claude.json'), + JSON.stringify(buildSeedConfig({ + apiKey: process.env.ANTHROPIC_API_KEY ?? process.env.GSTACK_ANTHROPIC_API_KEY, + trustedDirs: [repoRoot()], + }), null, 2), + ); + const root = repoRoot(); + for (const rel of skillCensus(root).physicalSkillFiles) { + const skillMd = path.join(root, rel); + const skillDir = path.dirname(rel); + const registryName = rel === 'SKILL.md' + ? '_gstack-command' + : frontmatterName(skillMd) || skillDir; + const target = path.join(skillsDir, registryName); + // Idempotent overwrite mirrors setup's re-link: connect-chrome (a dir + // symlink to open-gstack-browser) shares its target's frontmatter name, + // so the two walk entries collapse to one registry dir. + fs.mkdirSync(target, { recursive: true }); + safeUnlink(path.join(target, 'SKILL.md')); + fs.symlinkSync(skillMd, path.join(target, 'SKILL.md')); + if (rel !== 'SKILL.md') { + const sections = path.join(root, skillDir, 'sections'); + if (fs.existsSync(sections)) { + safeUnlink(path.join(target, 'sections')); + fs.symlinkSync(sections, path.join(target, 'sections')); + } + } + } + cachedSkillsConfigDir = configDir; + return configDir; +} + /** A dir younger than this is never GC'd even if its pid looks dead — guards * against PID reuse deleting a freshly-created dir whose original pid exited * and was recycled to an unrelated live process between create and GC. */ diff --git a/test/helpers/paid-test-set.ts b/test/helpers/paid-test-set.ts new file mode 100644 index 000000000..a4faf0b75 --- /dev/null +++ b/test/helpers/paid-test-set.ts @@ -0,0 +1,25 @@ +/** + * The ONE definition of which test files are paid (API spend, external + * services, e2e harnesses). package.json's test:gate/test:evals globs, the + * free-suite exclusion in scripts/test-free-shards.ts, and the sharded paid + * runner in scripts/test-paid-shards.ts all derive from this list — a file + * added to one and not the others either burns money in the free suite or + * silently never runs in the paid tier. + */ + +import { matchGlob } from './touchfiles'; + +/** The exact globs package.json's `test:gate` passes to `bun test`. */ +export const PAID_TEST_GLOBS = [ + 'test/skill-llm-eval.test.ts', + 'test/skill-e2e-*.test.ts', + 'test/skill-routing-e2e.test.ts', + 'test/codex-e2e.test.ts', + 'test/gemini-e2e.test.ts', +] as const; + +/** True when a repo-relative path (either slash style) is a paid test file. */ +export function isPaidTestFile(relativePath: string): boolean { + const normalized = relativePath.replace(/\\/g, '/'); + return PAID_TEST_GLOBS.some((glob) => matchGlob(normalized, glob)); +} diff --git a/test/helpers/providers/gemini.ts b/test/helpers/providers/gemini.ts index b53d725d1..e5d967f85 100644 --- a/test/helpers/providers/gemini.ts +++ b/test/helpers/providers/gemini.ts @@ -103,10 +103,10 @@ export function resultFromGeminiStream( * Headless flags always passed: * --output-format stream-json — NDJSON events (message/tool_use/result) * --yolo — auto-approve tools (non-interactive) - * --skip-trust — trust cwd for this session; required when - * workdir is a temp/untrusted folder (benchmarks - * use mkdtemp). Without it headless gemini exits - * before calling the model. + * + * --skip-trust is gone: gemini-cli 0.34 removed the flag ("Unknown arguments: + * skip-trust") — folder trust is settings-driven now and headless runs no + * longer need a flag for temp workdirs. */ export class GeminiAdapter implements ProviderAdapter { readonly name = 'gemini'; @@ -136,10 +136,9 @@ export class GeminiAdapter implements ProviderAdapter { async run(opts: RunOpts): Promise { const start = Date.now(); // Default to --yolo (non-interactive) and stream-json output so we can parse - // tokens + tool calls. --skip-trust is required for headless/temp workdirs - // (gemini CLI otherwise exits: "not running in a trusted directory"). - // Callers can override via extraArgs. - const args = ['-p', opts.prompt, '--output-format', 'stream-json', '--yolo', '--skip-trust']; + // tokens + tool calls. Callers can override via extraArgs. (--skip-trust was + // removed in gemini-cli 0.34; passing it errors at argv parse.) + const args = ['-p', opts.prompt, '--output-format', 'stream-json', '--yolo']; if (opts.model) args.push('--model', opts.model); if (opts.extraArgs) args.push(...opts.extraArgs); diff --git a/test/helpers/skill-census.ts b/test/helpers/skill-census.ts new file mode 100644 index 000000000..378201041 --- /dev/null +++ b/test/helpers/skill-census.ts @@ -0,0 +1,96 @@ +/** + * Shared skill census — the ONE place that counts skills (C11). + * + * Three consumers (hermetic seeding, context-bill ground truth, catalog-budget + * test) each need a DIFFERENT count of "the skills", and hand-rolled walks + * encode the wrong one somewhere. The counts diverge because of two facts: + * + * 1. `connect-chrome/` is a directory SYMLINK to `open-gstack-browser/`. + * `Dirent.isDirectory()` is false for it (scripts/discover-skills.ts + * skips it) while setup's trailing-slash shell glob follows it. + * 2. The root `SKILL.md` is a router, registered by `setup` under the + * alias name `_gstack-command`, not as an authored skill. + * + * So: use `physicalSkillFiles` when you mean "every SKILL.md a filesystem + * walk can reach" (context-bill's walker), `authoredSkills` when you mean + * "distinct skills a human maintains" (catalog budget), and + * `registryEntries` when you mean "what a host discovers after ./setup" + * (hermetic seeding must mirror this exactly). + */ + +import * as fs from 'fs'; +import * as path from 'path'; + +const SKIP = new Set(['node_modules', '.git', 'dist']); + +export interface SkillCensus { + /** Every reachable `/SKILL.md` (symlinked dirs INCLUDED) plus the + * root `SKILL.md` router. Relative paths from root. */ + physicalSkillFiles: string[]; + /** Distinct authored skills: symlink-deduped by realpath, root router + * EXCLUDED. Each entry is the canonical directory name. */ + authoredSkills: string[]; + /** What `setup` registers into `~/.claude/skills/` with prefixing off: + * one entry per unique frontmatter `name:` (falling back to dir name), + * plus the `_gstack-command` root alias when the root router exists. + * Symlinked dirs collapse here because they share a frontmatter name. */ + registryEntries: string[]; +} + +/** First `name:` from SKILL.md frontmatter, mirroring `setup`'s + * `grep -m1 '^name:'` (whitespace stripped). Empty string if absent. */ +export function frontmatterName(skillMdPath: string): string { + let body: string; + try { + body = fs.readFileSync(skillMdPath, 'utf-8'); + } catch { + return ''; + } + const m = body.match(/^name:\s*(.+)$/m); + return m ? m[1].replace(/\s+/g, '') : ''; +} + +export function skillCensus(root: string): SkillCensus { + const physicalSkillFiles: string[] = []; + const authoredByRealpath = new Map(); + const registrySet = new Set(); + + if (fs.existsSync(path.join(root, 'SKILL.md'))) { + physicalSkillFiles.push('SKILL.md'); + registrySet.add('_gstack-command'); + } + + for (const entry of fs.readdirSync(root, { withFileTypes: true })) { + if (entry.name.startsWith('.') || SKIP.has(entry.name)) continue; + const dirPath = path.join(root, entry.name); + // Follow directory symlinks like setup's shell glob does; Dirent alone + // reports symlinks as non-directories. + let isDir = entry.isDirectory(); + if (!isDir && entry.isSymbolicLink()) { + try { + isDir = fs.statSync(dirPath).isDirectory(); + } catch { + continue; // dangling symlink + } + } + if (!isDir) continue; + + const skillMd = path.join(dirPath, 'SKILL.md'); + if (!fs.existsSync(skillMd)) continue; + + physicalSkillFiles.push(`${entry.name}/SKILL.md`); + + const real = fs.realpathSync(dirPath); + if (!authoredByRealpath.has(real)) { + authoredByRealpath.set(real, path.basename(real)); + } + + registrySet.add(frontmatterName(skillMd) || entry.name); + } + + return { + physicalSkillFiles: physicalSkillFiles.sort(), + authoredSkills: [...authoredByRealpath.values()].sort(), + registryEntries: [...registrySet].sort(), + }; +} diff --git a/test/hermetic-skills-seeding.test.ts b/test/hermetic-skills-seeding.test.ts new file mode 100644 index 000000000..43b294db9 --- /dev/null +++ b/test/hermetic-skills-seeding.test.ts @@ -0,0 +1,84 @@ +/** + * Unit tests for hermeticSkillsConfigDir() — the opt-in hermetic config dir + * that registers the repo's shipped skills for PTY slash-command children. + * Free tier — no API calls; exercises the real seeder against the live repo + * tree (that's the seeder's contract: the skills ARE the subject under test). + * + * Pins four contracts: + * 1. The seeded dir is a valid CLAUDE_CONFIG_DIR (.claude.json present, + * /.claude suffix, under the hermetic runRoot). + * 2. Registration mirrors ./setup exactly: one entry per + * skillCensus().registryEntries, each a REAL dir with a SKILL.md symlink + * resolving to a real file (plus sections/ when the skill has one). + * 3. connect-chrome (dir symlink) collapses into open-gstack-browser — no + * duplicate, no connect-chrome entry. + * 4. Per-process idempotence: the second call returns the cached dir. + */ + +import { describe, test, expect } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; +import { + hermeticSkillsConfigDir, + getHermeticDirs, + buildSeedConfig, +} from './helpers/hermetic-env'; +import { skillCensus } from './helpers/skill-census'; + +const ROOT = path.resolve(__dirname, '..'); +const configDir = hermeticSkillsConfigDir(); +const skillsDir = path.join(configDir, 'skills'); + +describe('hermeticSkillsConfigDir', () => { + test('seeded dir contains .claude.json and ends in /.claude under runRoot', () => { + expect(fs.existsSync(path.join(configDir, '.claude.json'))).toBe(true); + expect(path.basename(configDir)).toBe('.claude'); + expect(configDir.startsWith(getHermeticDirs().runRoot + path.sep)).toBe(true); + }); + + test('one registry entry per skillCensus registryEntries, nothing extra', () => { + const seeded = fs.readdirSync(skillsDir).sort(); + expect(seeded).toEqual(skillCensus(ROOT).registryEntries); + }); + + test('every SKILL.md is a symlink resolving to a real file', () => { + for (const entry of fs.readdirSync(skillsDir)) { + const link = path.join(skillsDir, entry, 'SKILL.md'); + expect(fs.lstatSync(link).isSymbolicLink()).toBe(true); + expect(fs.statSync(link).isFile()).toBe(true); // follows the link + } + }); + + test('sections/ symlink registered for skills that ship one', () => { + // ship/ is a carved skill with a sections/ dir — the registered entry + // must expose it or runtime "Read sections/.md" 404s. + expect(fs.existsSync(path.join(ROOT, 'ship', 'sections'))).toBe(true); + const link = path.join(skillsDir, 'ship', 'sections'); + expect(fs.lstatSync(link).isSymbolicLink()).toBe(true); + expect(fs.statSync(link).isDirectory()).toBe(true); + }); + + test('connect-chrome collapses into a single open-gstack-browser entry', () => { + const seeded = fs.readdirSync(skillsDir); + expect(seeded.filter((n) => n === 'open-gstack-browser')).toHaveLength(1); + expect(seeded).not.toContain('connect-chrome'); + }); + + test('root router registered as _gstack-command pointing at the root SKILL.md', () => { + const link = path.join(skillsDir, '_gstack-command', 'SKILL.md'); + expect(fs.realpathSync(link)).toBe(fs.realpathSync(path.join(ROOT, 'SKILL.md'))); + }); + + test('second call returns the cached dir', () => { + expect(hermeticSkillsConfigDir()).toBe(configDir); + }); + + test('buildSeedConfig with undefined apiKey omits customApiKeyResponses', () => { + // The seeder passes process.env keys straight through; when the operator + // has no key exported the seed must stay valid (child fails auth later, + // not here). + const seed = buildSeedConfig({ apiKey: undefined, trustedDirs: [ROOT] }); + expect(seed).not.toHaveProperty('customApiKeyResponses'); + expect(seed.hasCompletedOnboarding).toBe(true); + }); +}); diff --git a/test/hermetic-wiring.test.ts b/test/hermetic-wiring.test.ts index 08528586d..6ebbccbce 100644 --- a/test/hermetic-wiring.test.ts +++ b/test/hermetic-wiring.test.ts @@ -16,6 +16,8 @@ import { describe, test, expect } from 'bun:test'; import * as fs from 'fs'; import * as path from 'path'; +import * as os from 'os'; +import { getHermeticDirs, hermeticSkillsConfigDir } from './helpers/hermetic-env'; const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); @@ -110,4 +112,23 @@ describe('hermetic wiring tripwire', () => { offenders.join(', '), ).toEqual([]); }); + + test('skill seeding stays under runRoot and reads the live repo tree, never operator ~/.claude', () => { + // hermeticSkillsConfigDir() is a BLESSED non-hermetic edge: it registers + // the LIVE repo tree's skills (the skills are the subject under test). + // What it must never do is hand children the operator's ~/.claude — the + // seeded CLAUDE_CONFIG_DIR lives under the hermetic runRoot, and every + // registered symlink resolves into the repo checkout. + const configDir = hermeticSkillsConfigDir(); + const { runRoot } = getHermeticDirs(); + const operatorClaude = path.join(os.homedir(), '.claude') + path.sep; + expect(configDir.startsWith(runRoot + path.sep)).toBe(true); + expect(configDir.startsWith(operatorClaude)).toBe(false); + const skillsDir = path.join(configDir, 'skills'); + for (const entry of fs.readdirSync(skillsDir)) { + const target = fs.readlinkSync(path.join(skillsDir, entry, 'SKILL.md')); + expect(target.startsWith(operatorClaude), `${entry}: symlink escapes to ${target}`).toBe(false); + expect(fs.realpathSync(target).startsWith(fs.realpathSync(ROOT) + path.sep), `${entry}: symlink outside repo: ${target}`).toBe(true); + } + }); }); diff --git a/test/paid-shards.test.ts b/test/paid-shards.test.ts new file mode 100644 index 000000000..2f58bef6b --- /dev/null +++ b/test/paid-shards.test.ts @@ -0,0 +1,117 @@ +/** + * Pins the paid-tier sharded runner (scripts/test-paid-shards.ts). + * + * Two properties matter, and both are why `test:gate` has never finished a run: + * 1. Enumeration + sharding — every file `test:gate`'s globs expand to gets + * its own process, and tier exclusion only ever fires on explicit evidence. + * 2. A spinning shard is killed externally and the run CONTINUES. The fake + * command here is a real busy loop, so an in-process timer could not save + * it — exactly the failure mode `sample` caught on the wedged run. + */ + +import { describe, test, expect } from 'bun:test'; +import { + PAID_TEST_GLOBS, + classifyPaidTestFile, + collectPaidTestFiles, + isPaidTestFile, + planPaidShards, + runPaidShards, + summarize, + type ShardOutcome, +} from '../scripts/test-paid-shards'; + +describe('paid test enumeration', () => { + test('matches the globs package.json test:gate expands', () => { + expect(isPaidTestFile('test/skill-e2e-qa-workflow.test.ts')).toBe(true); + expect(isPaidTestFile('test/skill-llm-eval.test.ts')).toBe(true); + expect(isPaidTestFile('test/codex-e2e.test.ts')).toBe(true); + expect(isPaidTestFile('test/skill-e2e-triage-audit.test.ts')).toBe(true); + // Outside the globs: no dash, extra suffix, or a free test. + expect(isPaidTestFile('test/skill-e2e.test.ts')).toBe(false); + expect(isPaidTestFile('test/codex-e2e-recommendation-substance.test.ts')).toBe(false); + expect(isPaidTestFile('test/paid-shards.test.ts')).toBe(false); + }); + + test('discovers files and gives each one its own shard', () => { + const files = collectPaidTestFiles(); + expect(files.length).toBeGreaterThan(0); + expect(files.every(isPaidTestFile)).toBe(true); + expect(PAID_TEST_GLOBS.length).toBe(5); + + const shards = planPaidShards(files); + expect(shards.flat().sort()).toEqual([...files].sort()); + expect(shards.every((shard) => shard.length === 1)).toBe(true); + }); +}); + +describe('tier classification', () => { + test('excludes only on an explicit other-tier guard', () => { + const gateGuard = "const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';"; + const periodicGuard = "const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'periodic';"; + + expect(classifyPaidTestFile(gateGuard, 'gate').included).toBe(true); + expect(classifyPaidTestFile(periodicGuard, 'gate').included).toBe(false); + expect(classifyPaidTestFile(gateGuard, 'periodic').included).toBe(false); + expect(classifyPaidTestFile(periodicGuard, 'periodic').included).toBe(true); + }); + + test('keeps files whose tier is decided per-test at runtime', () => { + // Naming an E2E_TIERS key is not evidence — 'retro' appears in the + // LLM-judge file, which test:gate does run. + const noGuard = "runSkillTest('retro', async () => {});"; + expect(classifyPaidTestFile(noGuard, 'gate').included).toBe(true); + expect(classifyPaidTestFile(noGuard, 'periodic').included).toBe(true); + expect(classifyPaidTestFile('', 'gate').included).toBe(true); + }); +}); + +describe('shard execution', () => { + const BUSY_LOOP = 'const end = Date.now() + 600000; while (Date.now() < end) {}'; + + const commandFor = (files: string[]) => { + if (files[0] === 'spin') return { command: process.execPath, args: ['-e', BUSY_LOOP] }; + if (files[0] === 'fail') return { command: process.execPath, args: ['-e', 'process.exit(3)'] }; + return { command: process.execPath, args: ['-e', 'console.log("ok")'] }; + }; + + test('a spinning shard times out, is killed, and the run continues', async () => { + const lines: string[] = []; + const summary = await runPaidShards([['spin'], ['fail'], ['pass']], { + timeoutMs: 1_200, + jobs: 1, + commandFor, + log: (line) => lines.push(line), + }); + + const byName = (name: string) => summary.outcomes.find((o) => o.files[0] === name) as ShardOutcome; + expect(byName('spin').status).toBe('timed-out'); + expect(byName('fail').status).toBe('failed'); + expect(byName('pass').status).toBe('passed'); + + // The run never aborted: every shard reports, none is 'never-started'. + expect(summary).toMatchObject({ + total: 3, executed: 3, passed: 1, failed: 1, timedOut: 1, neverStarted: 0, + }); + + // The spinner was killed at the deadline, not left to burn a core. + expect(byName('spin').elapsedMs).toBeLessThan(30_000); + expect(byName('spin').groupPid).toBeGreaterThan(0); + if (process.platform !== 'win32') { + expect(() => process.kill(byName('spin').groupPid as number, 0)).toThrow(); + } + + // Heartbeat: a START and a terminal line per shard, with elapsed seconds. + expect(lines.filter((l) => l.includes(' START ')).length).toBe(3); + expect(lines.some((l) => /TIMED-OUT in \d+s/.test(l))).toBe(true); + expect(lines.some((l) => /PASSED in \d+s/.test(l))).toBe(true); + }, 30_000); + + test('summarize reports shards that never ran', () => { + const summary = summarize([ + { shard: 1, files: ['a'], status: 'passed', exitCode: 0, elapsedMs: 1, groupPid: 1 }, + { shard: 2, files: ['b'], status: 'never-started', exitCode: null, elapsedMs: 0, groupPid: null }, + ]); + expect(summary).toMatchObject({ total: 2, executed: 1, passed: 1, neverStarted: 1 }); + }); +}); diff --git a/test/pty-askuserquestion-single-line.test.ts b/test/pty-askuserquestion-single-line.test.ts new file mode 100644 index 000000000..90da7466e --- /dev/null +++ b/test/pty-askuserquestion-single-line.test.ts @@ -0,0 +1,67 @@ +/** + * Pins single-logical-line AskUserQuestion detection (the plan-design-with-ui + * gate-timeout class). When the PTY reflows a boxed AskUserQuestion, ALL + * options land on ONE logical line after stripAnsi — the per-line option + * parser then finds only option 1 and the >= 2 check fails forever while the + * (correct) question sits on screen. Both fixture strings below are condensed + * from REAL observed failure buffers of + * test/skill-e2e-plan-design-with-ui.test.ts. + */ + +import { describe, expect, it } from 'bun:test'; +import { + isNumberedOptionListVisible, + parseNumberedOptions, + stripPtyResidue, +} from './helpers/claude-pty-runner'; + +// Condensed from the 2026-08-13 failure buffer: whole AskUserQuestion on one +// logical line — dividers, cursor option, options 2-5, footer. Note the +// missing spaces ("2.Planordesigndoc") from stripped cursor-positioning +// escapes. +const SINGLE_LINE_QUESTION = + '────────────Planning: /tmp/x/.claude/plans/soft-questing-jellyfish.md──────────' + + ' ☐ Review target What should I review? ' + + '❯1.Branch diff (current WIP) Review the design implications of what changed. ' + + 'Recommendation: A when a branch diff exists.2.Planordesigndoc Paste or point me to a plan file. ' + + '3. Spcific page, file, r pah Name a specific file. 4. Type something.' + + '────────────5.ChataboutthisEnter to select · ↑/↓ o navigate · Escto cancel'; + +// The same AskUserQuestion with DEC cursor-visibility residue + spinner +// frames interleaved, as captured before stripPtyResidue existed. +const RESIDUE_QUESTION = + '[?25l✻Sprouting…[?25h[?25l✶[?25h' + SINGLE_LINE_QUESTION.slice(0, 200) + + '[?25l·still thinking[?25h' + SINGLE_LINE_QUESTION.slice(200); + +describe('single-logical-line AskUserQuestion detection', () => { + it('isNumberedOptionListVisible matches the reflowed one-line AskUserQuestion', () => { + expect(isNumberedOptionListVisible(SINGLE_LINE_QUESTION)).toBe(true); + }); + + it('parseNumberedOptions finds the full ascending option run on one line', () => { + const options = parseNumberedOptions(SINGLE_LINE_QUESTION); + expect(options.length).toBeGreaterThanOrEqual(2); + expect(options[0]).toEqual({ index: 1, label: expect.stringContaining('Branch diff') }); + expect(options[1]?.index).toBe(2); + }); + + it('survives DEC residue + spinner interleave', () => { + expect(isNumberedOptionListVisible(RESIDUE_QUESTION)).toBe(true); + expect(parseNumberedOptions(RESIDUE_QUESTION).length).toBeGreaterThanOrEqual(2); + }); + + it('stripPtyResidue removes cursor-visibility fragments only', () => { + expect(stripPtyResidue('[?25la[?25hb')).toBe('ab'); + expect(stripPtyResidue('keep [?250] this')).toBe('keep [?250] this'); + }); + + it('still rejects prose numbered lists (no cursor sigil)', () => { + expect(parseNumberedOptions('steps: 1. Read the file 2. Edit it 3. Done')).toEqual([]); + }); + + it('still parses classic multi-line AskUserQuestions', () => { + const multiLine = 'What should I do?\n❯ 1. First option\n 2. Second option\n 3. Third option\n'; + const options = parseNumberedOptions(multiLine); + expect(options.map((o) => o.index)).toEqual([1, 2, 3]); + }); +}); diff --git a/test/pty-skill-seeding-wiring.test.ts b/test/pty-skill-seeding-wiring.test.ts new file mode 100644 index 000000000..01f5999ce --- /dev/null +++ b/test/pty-skill-seeding-wiring.test.ts @@ -0,0 +1,77 @@ +/** + * Static-grep tripwire for PTY slash-command skill seeding. Free tier — no API. + * + * The default hermetic config dir registers NO skills, so a PTY test that + * TYPES a /skill slash command against it gets "Unknown command" before any + * model turn — the test still runs, still spends money, and measures nothing. + * Every test file that sends a slash command over the PTY must therefore + * either route through a runPlanSkill* helper (which opts in for you) or pass + * `seedSkills: true` in its own launchClaudePty options. + * + * Pattern mirrors test/hermetic-wiring.test.ts: read sources as text, assert + * invariants on their contents. Brittle by design — changing the seeding + * wiring must force the author to look here. + */ + +import { describe, test, expect } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; + +const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); + +/** A PTY send whose payload starts with a slash command (`/name`, optionally + * followed by `\r`, whitespace, or the closing quote). A second slash right + * after the name (a file path like '/tmp/x') does NOT match. */ +const SLASH_SEND = /\.send\(\s*(['"`])\/[a-z][a-z0-9-]*(\\r|\s|\1)/; + +const RUN_PLAN_HELPER = /\brunPlanSkill(Observation|Counting|FloorCheck)\s*\(/; + +function testFiles(dir: string): string[] { + const out: string[] = []; + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) out.push(...testFiles(full)); + else if (entry.name.endsWith('.test.ts')) out.push(full); + } + return out; +} + +describe('PTY skill-seeding tripwire', () => { + test('every slash-command PTY test seeds skills (helper or seedSkills: true)', () => { + const offenders: string[] = []; + for (const full of testFiles(path.join(ROOT, 'test'))) { + if (path.basename(full) === 'pty-skill-seeding-wiring.test.ts') continue; + const src = fs.readFileSync(full, 'utf-8'); + const lines = src.split('\n'); + const sendLine = lines.findIndex((l) => SLASH_SEND.test(l)); + if (sendLine === -1) continue; + if (RUN_PLAN_HELPER.test(src)) continue; + if (src.includes('seedSkills: true')) continue; + offenders.push(`${path.relative(ROOT, full)}:${sendLine + 1}`); + } + expect( + offenders, + 'These tests type a /skill slash command into a hermetic PTY child that has ' + + 'no skills registered — claude rejects it as Unknown command and the test ' + + 'measures nothing. Pass seedSkills: true to launchClaudePty (or route ' + + 'through a runPlanSkill* helper): ' + offenders.join(', '), + ).toEqual([]); + }); + + test('the runPlanSkill* helpers all opt in via seedSkills: true', () => { + // The helper family types slash commands on behalf of ~20 test files; + // dropping the opt-in there silently un-measures all of them at once. + const src = fs.readFileSync(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'), 'utf-8'); + const optIns = src.match(/seedSkills: true/g) ?? []; + expect(optIns.length).toBeGreaterThanOrEqual(3); + }); + + test('launchClaudePty wires seedSkills to hermeticSkillsConfigDir()', () => { + const src = fs.readFileSync(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'), 'utf-8'); + // Gated on hermetic (EVALS_HERMETIC=0 must keep the operator config) and + // on the per-test env override (explicit CLAUDE_CONFIG_DIR wins). + const wired = + /if\s*\(opts\.seedSkills && hermetic && !opts\.env\?\.CLAUDE_CONFIG_DIR\)\s*\{\s*\n\s*childEnv\.CLAUDE_CONFIG_DIR = hermeticSkillsConfigDir\(\);/.test(src); + expect(wired, 'launchClaudePty must set CLAUDE_CONFIG_DIR from hermeticSkillsConfigDir() when seedSkills && hermetic && no per-test override').toBe(true); + }); +}); diff --git a/test/skill-budget-regression.test.ts b/test/skill-budget-regression.test.ts index 85391bfc2..318875cd9 100644 --- a/test/skill-budget-regression.test.ts +++ b/test/skill-budget-regression.test.ts @@ -27,10 +27,10 @@ import { describe, test } from 'bun:test'; import { spawnSync } from 'child_process'; import * as fs from 'fs'; -import * as path from 'path'; import { getProjectEvalDir, findPreviousRun, + findLatestFinalizedRun, compareEvalResults, assertNoBudgetRegression, type EvalResult, @@ -72,42 +72,9 @@ function currentGitBranch(): string { } } -interface LatestRun { - filepath: string; - result: EvalResult; -} - -/** Find the most recent finalized (non-_partial) eval file for a tier. */ -function findLatestRun(evalDir: string, tier: 'e2e' | 'llm-judge'): LatestRun | null { - let entries: string[]; - try { - entries = fs.readdirSync(evalDir); - } catch { - return null; - } - const candidates: Array<{ filepath: string; timestamp: string }> = []; - for (const f of entries) { - if (!f.endsWith('.json')) continue; - if (f.startsWith('_partial')) continue; - const fullPath = path.join(evalDir, f); - try { - const data = JSON.parse(fs.readFileSync(fullPath, 'utf-8')) as EvalResult; - if (data.tier !== tier) continue; - candidates.push({ filepath: fullPath, timestamp: data.timestamp ?? '' }); - } catch { /* ignore corrupt */ } - } - if (candidates.length === 0) return null; - candidates.sort((a, b) => b.timestamp.localeCompare(a.timestamp)); - const top = candidates[0]!; - return { - filepath: top.filepath, - result: JSON.parse(fs.readFileSync(top.filepath, 'utf-8')) as EvalResult, - }; -} - function checkTier(tier: 'e2e' | 'llm-judge'): void { const evalDir = getProjectEvalDir(); - const latest = findLatestRun(evalDir, tier); + const latest = findLatestFinalizedRun(evalDir, tier); if (!latest) { // eslint-disable-next-line no-console console.log(`[budget-regression:${tier}] no current run in ${evalDir} — skipping`); @@ -165,7 +132,7 @@ function checkTier(tier: 'e2e' | 'llm-judge'): void { /** Enforce a hard dollar cap on per-run eval cost. */ function checkHardCap(tier: 'e2e' | 'llm-judge'): void { const evalDir = getProjectEvalDir(); - const latest = findLatestRun(evalDir, tier); + const latest = findLatestFinalizedRun(evalDir, tier); if (!latest) return; const cap = TIER_CAPS[tier]; const cost = latest.result.total_cost_usd; diff --git a/test/skill-census.test.ts b/test/skill-census.test.ts new file mode 100644 index 000000000..b9bb21578 --- /dev/null +++ b/test/skill-census.test.ts @@ -0,0 +1,69 @@ +/** + * Pins the three-count contract of test/helpers/skill-census.ts (C11). + * + * No hardcoded totals here — the catalog-budget test owns the ratchet. + * This file pins the STRUCTURAL relationships that make the three counts + * mean different things, using the live repo as the fixture. + */ + +import { describe, expect, it } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; +import { frontmatterName, skillCensus } from './helpers/skill-census'; + +const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..'); +const census = skillCensus(ROOT); + +describe('skillCensus', () => { + it('physicalSkillFiles includes the root router and the symlinked dir', () => { + expect(census.physicalSkillFiles).toContain('SKILL.md'); + expect(census.physicalSkillFiles).toContain('connect-chrome/SKILL.md'); + expect(census.physicalSkillFiles).toContain('open-gstack-browser/SKILL.md'); + }); + + it('authoredSkills dedupes the connect-chrome symlink and excludes the root router', () => { + expect(census.authoredSkills).toContain('open-gstack-browser'); + expect(census.authoredSkills).not.toContain('connect-chrome'); + // Root router is not an authored skill; its dir entry would be '' anyway. + for (const name of census.authoredSkills) expect(name.length).toBeGreaterThan(0); + }); + + it('registryEntries carries the root alias and collapses shared frontmatter names', () => { + expect(census.registryEntries).toContain('_gstack-command'); + expect( + census.registryEntries.filter((n) => n === 'open-gstack-browser'), + ).toHaveLength(1); + }); + + it('count relationships hold: physical = authored + root + symlink dups', () => { + const symlinkDups = census.physicalSkillFiles.length - 1 - census.authoredSkills.length; + expect(symlinkDups).toBeGreaterThanOrEqual(1); // connect-chrome today + // Registry = unique frontmatter names + root alias. It can only collapse + // entries relative to physical, never invent them. + expect(census.registryEntries.length).toBeLessThanOrEqual(census.physicalSkillFiles.length); + expect(census.registryEntries.length).toBeGreaterThan(census.authoredSkills.length - 1); + }); + + it('frontmatterName mirrors setup: first ^name: line, whitespace stripped', () => { + const qa = frontmatterName(path.join(ROOT, 'qa', 'SKILL.md')); + expect(qa).toBe('qa'); + const alias = frontmatterName(path.join(ROOT, 'connect-chrome', 'SKILL.md')); + expect(alias).toBe('open-gstack-browser'); + expect(frontmatterName(path.join(ROOT, 'no-such-dir', 'SKILL.md'))).toBe(''); + }); + + it('every registry entry a host would see resolves back to a physical SKILL.md', () => { + const names = new Set( + census.physicalSkillFiles + .filter((p) => p !== 'SKILL.md') + .map((p) => frontmatterName(path.join(ROOT, p)) || path.dirname(p)), + ); + for (const entry of census.registryEntries) { + if (entry === '_gstack-command') { + expect(fs.existsSync(path.join(ROOT, 'SKILL.md'))).toBe(true); + } else { + expect(names.has(entry)).toBe(true); + } + } + }); +}); diff --git a/test/skill-e2e-autoplan-chain.test.ts b/test/skill-e2e-autoplan-chain.test.ts index b5e3ce740..645e55bf9 100644 --- a/test/skill-e2e-autoplan-chain.test.ts +++ b/test/skill-e2e-autoplan-chain.test.ts @@ -71,6 +71,7 @@ describeE2E('/autoplan chain ordering (periodic)', () => { permissionMode: 'plan', cwd: tempDir, timeoutMs: 1_080_000, // 18 min, slightly above test budget + seedSkills: true, }); const hits: PhaseHit[] = []; diff --git a/test/skill-e2e-benchmark-providers.test.ts b/test/skill-e2e-benchmark-providers.test.ts index 98c2f3e03..10395dea5 100644 --- a/test/skill-e2e-benchmark-providers.test.ts +++ b/test/skill-e2e-benchmark-providers.test.ts @@ -30,7 +30,15 @@ import * as os from 'os'; // --- Prerequisites / gating --- const evalsEnabled = !!process.env.EVALS; -const describeIfEvals = evalsEnabled ? describe : describe.skip; +// External-service tests are periodic-tier (CLAUDE.md tiering rule 3) — +// the header above says so, but without a whole-file guard the sharded gate +// runner still selects this file into gate. The positive form below is the +// canonical guard shape classifyPaidTestFile greps for. +const tierOk = process.env.EVALS_TIER === 'periodic'; +const describeIfEvals = evalsEnabled && tierOk ? describe : describe.skip; +if (evalsEnabled && !tierOk) { + process.stderr.write('\nbenchmark-providers: SKIPPED — external-service test, periodic tier only\n'); +} const PROMPT = 'Reply with exactly this text and nothing else: ok'; @@ -127,6 +135,15 @@ describeIfEvals('multi-provider benchmark adapters (live)', () => { } const result = await gemini.run({ prompt: PROMPT, workdir, timeoutMs: 120_000 }); if (result.error) { + // auth / rate_limit are ENVIRONMENT conditions the test can't act on + // (e.g. Google deprecated the individual code-assist auth path — the + // adapter classifies "no longer supported" as auth). A live smoke + // reports them as a skip, not a false adapter failure. timeout/unknown + // still fail: those are the drift classes this test exists to catch. + if (result.error.code === 'auth' || result.error.code === 'rate_limit') { + process.stderr.write(`\ngemini live smoke: SKIPPED — ${result.error.code}: ${result.error.reason.slice(0, 160)}\n`); + return; + } throw new Error(`gemini errored: ${result.error.code} — ${result.error.reason}`); } // Adapter must never report empty-success (#2159). After content/stats diff --git a/test/skill-e2e-hermetic-canary.test.ts b/test/skill-e2e-hermetic-canary.test.ts index 1356b7dce..06f1dc302 100644 --- a/test/skill-e2e-hermetic-canary.test.ts +++ b/test/skill-e2e-hermetic-canary.test.ts @@ -84,9 +84,13 @@ describeIfSelected('hermetic isolation canaries', ['hermetic-canary', 'hermetic- try { const result = await runSkillTest({ + // ${VAR:-} expansion, not bare $VAR: when scrubbing WORKS the planted + // vars are unset, and under a nounset shell (set -u in the operator's + // shell snapshot) a bare expansion of an unset var errors the whole + // command — making the canary fail exactly when isolation succeeds. prompt: 'Run exactly this bash command and then stop: ' + - 'echo "CFG=$CLAUDE_CONFIG_DIR"; echo "GH=$GSTACK_HOME"; ' + - 'echo "CW=$CONDUCTOR_WORKSPACE_PATH"; echo "GP=$GBRAIN_POISON_PROBE"', + 'echo "CFG=${CLAUDE_CONFIG_DIR:-}"; echo "GH=${GSTACK_HOME:-}"; ' + + 'echo "CW=${CONDUCTOR_WORKSPACE_PATH:-}"; echo "GP=${GBRAIN_POISON_PROBE:-}"', workingDirectory: workDir, maxTurns: 3, allowedTools: ['Bash'], diff --git a/test/skill-e2e-plan-ceo-mode-routing.test.ts b/test/skill-e2e-plan-ceo-mode-routing.test.ts index 0199413b8..0772e7a11 100644 --- a/test/skill-e2e-plan-ceo-mode-routing.test.ts +++ b/test/skill-e2e-plan-ceo-mode-routing.test.ts @@ -152,6 +152,7 @@ describeE2E('/plan-ceo-review mode routing (gate)', () => { const session = await launchClaudePty({ permissionMode: 'plan', timeoutMs: 540_000, + seedSkills: true, }); try { await Bun.sleep(8000); diff --git a/test/skill-e2e-plan-design-with-ui.test.ts b/test/skill-e2e-plan-design-with-ui.test.ts index 622bd9382..904fe96ba 100644 --- a/test/skill-e2e-plan-design-with-ui.test.ts +++ b/test/skill-e2e-plan-design-with-ui.test.ts @@ -44,7 +44,8 @@ describeE2E('/plan-design-review with UI scope (gate)', () => { const session = await launchClaudePty({ permissionMode: 'plan', cwd: ROOT, - timeoutMs: 480_000, + timeoutMs: 720_000, + seedSkills: true, }); let outcome: 'real_question' | 'plan_ready' | 'timeout' | 'exited' = 'timeout'; @@ -70,7 +71,11 @@ describeE2E('/plan-design-review with UI scope (gate)', () => { `Reference plan file: ${fixtureRelPath}\r` ); - const budgetMs = 360_000; + // 600s, not 360s: the skill preamble (update-check, session bookkeeping, + // learnings) plus extended model thinking can take ~6 minutes before the + // scope-gate AskUserQuestion renders — a 360s budget expired seconds + // before the (correct) AUQ appeared in the observed failure transcript. + const budgetMs = 600_000; const start = Date.now(); let lastPermSig = ''; while (Date.now() - start < budgetMs) { @@ -145,6 +150,6 @@ describeE2E('/plan-design-review with UI scope (gate)', () => { ); } }, - 540_000, + 780_000, ); }); diff --git a/test/skill-e2e-ship-idempotency.test.ts b/test/skill-e2e-ship-idempotency.test.ts index daed1f1d7..e4388a7d0 100644 --- a/test/skill-e2e-ship-idempotency.test.ts +++ b/test/skill-e2e-ship-idempotency.test.ts @@ -161,6 +161,7 @@ describeE2E('/ship idempotency E2E (periodic, real-PTY)', () => { timeoutMs: 720_000, // Disable network-y pieces so the agent can't reach actual github. env: { GH_TOKEN: 'mock-not-real', NO_COLOR: '1' }, + seedSkills: true, }); let outcome: 'detected' | 'plan_ready' | 'attempted_mutation' | 'timeout' | 'exited' = 'timeout'; diff --git a/test/strict-output.test.ts b/test/strict-output.test.ts new file mode 100644 index 000000000..52f79fd48 --- /dev/null +++ b/test/strict-output.test.ts @@ -0,0 +1,61 @@ +/** + * Pins scripts/test-strict-output.ts — the verdict-integrity layer of the + * sharded paid runner. Its whole reason to exist is refusing to trust a zero + * exit when failures were printed OR fewer files ran than planned; paid-shards' + * fake commands never emit real Bun result lines, so without this file that + * core was exercised nowhere and a regex regression would silently revert the + * paid tier to trusting exit codes. + */ + +import { describe, expect, it } from 'bun:test'; +import { BunTestOutputClassifier, strictTestExitCode } from '../scripts/test-strict-output'; + +describe('strictTestExitCode', () => { + it('trusts a clean zero exit when the expected file count ran', () => { + const summary = { failedTests: 0, unhandledBetweenTests: 0, terminalFileCounts: [1] }; + expect(strictTestExitCode(0, summary, 1)).toBe(0); + }); + + it('refuses a zero exit when fewer files ran than expected (invisible non-execution)', () => { + const summary = { failedTests: 0, unhandledBetweenTests: 0, terminalFileCounts: [1] }; + expect(strictTestExitCode(0, summary, 2)).toBe(1); + }); + + it('refuses a zero exit when failure lines were printed', () => { + const summary = { failedTests: 1, unhandledBetweenTests: 0, terminalFileCounts: [1] }; + expect(strictTestExitCode(0, summary, 1)).toBe(1); + }); + + it('refuses a zero exit on an unhandled error between tests', () => { + const summary = { failedTests: 0, unhandledBetweenTests: 1, terminalFileCounts: [1] }; + expect(strictTestExitCode(0, summary, 1)).toBe(1); + }); + + it('propagates a non-zero child exit regardless of expectedFiles', () => { + const summary = { failedTests: 0, unhandledBetweenTests: 0, terminalFileCounts: [1] }; + expect(strictTestExitCode(1, summary, 1)).toBe(1); + }); +}); + +describe('BunTestOutputClassifier', () => { + it('counts a (fail) line split across write chunks', () => { + const c = new BunTestOutputClassifier(); + c.write('(fail) my te'); + c.write('st [3.42ms]\nRan 4 tests across 1 files. [2.10s]\n'); + const summary = c.end(); + expect(summary.failedTests).toBe(1); + expect(summary.terminalFileCounts).toEqual([1]); + // exit 0 + a printed failure must not be trusted + expect(strictTestExitCode(0, summary, 1)).toBe(1); + }); + + it('records the terminal file count from the summary line', () => { + const c = new BunTestOutputClassifier(); + c.write('Ran 0 tests across 1 files. [0.01s]\n'); + const summary = c.end(); + expect(summary.terminalFileCounts).toEqual([1]); + // a fully diff-skipped single-file shard (0 tests, 1 file loaded) still + // passes: 1 file ran, which is what was expected + expect(strictTestExitCode(0, summary, 1)).toBe(0); + }); +});