mirror of https://github.com/garrytan/gstack.git
docs: sync every doc surface with the v1.63 release
/document-release audit (4-lane, all claims verified against branch code): - README: gstack-egress + gstack-context-bill rows in the standalone-binaries table; Privacy & Telemetry gains the receipted-egress bullet (attempted- egress framing per the shipped threat model). - ARCHITECTURE: /health is liveness-only, POST /extension-token endpoint row + bootstrap mechanics paragraph; new Egress receipt ledger subsection under Security model; eval persistence covers the sharded runner, GSTACK_EVAL_DIR, and the finalized-run baseline rule. - CLAUDE.md: sharded test scripts in Commands; sharded semantics in the detached-evals section; PTY skill seeding in the hermetic section; egress invariant block beside the other server-egress invariants; catalog-budget ceiling beside the 160KB token ceiling; project-tree entries for lib/egress-receipt.ts, lib/context-bill.ts, scripts/test-paid-shards.ts. - CONTRIBUTING: seedSkills + live-tree seeding in the hermetic paragraph; sharded runner in detached runs; catalog-budget in the Tier 1 list. - BROWSER: extension token bootstrap section, tunnel egress receipts section, identity-pin migration note in manual install. - REMOTE_BROWSER_ACCESS: tunnel-start receipt bullet in the security model. - gbrain docs: /sync-gbrain + brain-sync egress-receipt behavior documented; dead consumer-token instructions removed (consumer machinery deleted this release); new fail-closed refusal added to the error catalog. - CHANGELOG: measured-vs-ceiling catalog numbers, contributor notes for the external-service tier move and the PTY single-line AskUserQuestion parser, release date. - TODOS: /health token-distribution TODO resolved by this release, removed; port-wave follow-up sections re-labeled to the shipped version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
parent
562273d793
commit
e7110aac93
|
|
@ -91,14 +91,15 @@ When a user runs `pair-agent --client`, the daemon starts an ngrok tunnel so a r
|
|||
|
||||
The fix is **two HTTP listeners**, not one:
|
||||
|
||||
- **Local listener** (`127.0.0.1:LOCAL_PORT`) — always bound. Serves bootstrap (`/health` with token delivery), `/cookie-picker`, `/inspector/*`, `/welcome`, `/refs`, the sidebar-agent API, and the full command surface. Never forwarded.
|
||||
- **Local listener** (`127.0.0.1:LOCAL_PORT`) — always bound. Serves token bootstrap (`POST /extension-token`, released only to the pinned extension identity), `/health` (liveness/status only — never a token), `/cookie-picker`, `/inspector/*`, `/welcome`, `/refs`, the sidebar-agent API, and the full command surface. Never forwarded.
|
||||
- **Tunnel listener** (`127.0.0.1:TUNNEL_PORT`) — bound lazily on `/tunnel/start`, torn down on `/tunnel/stop`. Serves a locked allowlist: `/connect` (pairing ceremony, unauth + rate-limited), `/command` (scoped tokens only, further restricted to a browser-driving command allowlist), and `/sidebar-chat`. Everything else 404s.
|
||||
|
||||
ngrok forwards only the tunnel port. The security property comes from **physical port separation**: a tunnel caller cannot reach `/health` or `/cookie-picker` because those paths don't exist on that TCP socket. Header inference (check `x-forwarded-for`, check origin) is unreliable (ngrok header behavior changes; local proxies can add these headers); socket separation isn't.
|
||||
|
||||
| Endpoint | Local listener | Tunnel listener | Notes |
|
||||
|---|---|---|---|
|
||||
| `GET /health` | public (no token unless headed/extension) | 404 | Token bootstrap for extension happens locally only |
|
||||
| `GET /health` | public (liveness/status only — never a token) | 404 | Token bootstrap moved to `POST /extension-token` (v1.63) |
|
||||
| `POST /extension-token` | pinned Origin (`chrome-extension://<GSTACK_EXTENSION_ID>`) + loopback Host | 404 | The only endpoint that hands out the root token |
|
||||
| `GET /connect` | public (`{alive:true}`) | public (`{alive:true}`) | Probe path for tunnel liveness |
|
||||
| `POST /connect` | public (rate-limited 300/min) | public (rate-limited) | Setup-key exchange for pair-agent |
|
||||
| `POST /command` | auth (Bearer root OR scoped) | auth (scoped only, allowlisted commands) | Root token on tunnel = 403 |
|
||||
|
|
@ -114,6 +115,8 @@ ngrok forwards only the tunnel port. The security property comes from **physical
|
|||
| `GET /inspector/events` | Bearer OR HttpOnly `gstack_sse` cookie | 404 | SSE. Same cookie as /activity/stream |
|
||||
| `POST /sse-session` | auth (Bearer) | 404 | Mints the view-only 30-min SSE session cookie |
|
||||
|
||||
**Extension token bootstrap (v1.63.0.0).** `GET /health` never carries a token in any mode — it is liveness/status only. The sidebar extension obtains the root token via `POST /extension-token`, which releases it only when the caller's Origin is exactly `chrome-extension://<GSTACK_EXTENSION_ID>` (pinned by the `key` field in `extension/manifest.json`; reproduce the derivation with `bun browse/scripts/extension-id.ts`) and the Host header parses to a loopback hostname — parsed with `new URL()`, never compared raw, because Host carries the port. Web pages cannot forge a `chrome-extension://` Origin, and the endpoint is never added to the tunnel allowlist, so the tunnel surface 404s it by default-deny.
|
||||
|
||||
**Tunnel surface denial logs.** Every rejection on the tunnel listener (`path_not_on_tunnel`, `root_token_on_tunnel`, `missing_scoped_token`, `disallowed_command:*`) is recorded asynchronously to `~/.gstack/security/attempts.jsonl` with timestamp, source IP (from `x-forwarded-for`), path, and method. Rate-capped at 60 writes/min globally to prevent log-flood DoS. Shares the attempt log with the prompt-injection scanner.
|
||||
|
||||
**SSE session cookies.** EventSource can't send Authorization headers, so the extension POSTs `/sse-session` once at bootstrap with the root Bearer and receives a 30-minute view-only cookie (`gstack_sse`, HttpOnly, SameSite=Strict). The cookie is valid ONLY for `/activity/stream` and `/inspector/events` — it is NOT a scoped token and cannot be used on `/command`. Scope isolation is enforced by the module boundary: `sse-session-cookie.ts` has no imports from `token-registry.ts`.
|
||||
|
|
@ -144,6 +147,14 @@ Cookies are the most sensitive data gstack handles. The design:
|
|||
|
||||
The browser registry (Comet, Chrome, Arc, Brave, Edge) is hardcoded. Database paths are constructed from known constants, never from user input. Keychain access uses `Bun.spawn()` with explicit argument arrays, not shell string interpolation.
|
||||
|
||||
### Egress receipt ledger (v1.63.0.0)
|
||||
|
||||
Every gstack-initiated off-machine send writes a hash-chained, tamper-evident receipt to `~/.gstack/security/egress.jsonl` BEFORE the send — `writeReceipt` in `lib/egress-receipt.ts` for TypeScript callers, `_receipted_curl` / `_receipted_git` from `bin/gstack-egress-lib.sh` for shell scripts. Receipts hash the request body; they never store it.
|
||||
|
||||
Failure polarity is per-class and pinned by tests. Sensitive sinks are fail-closed: brain-sync pushes, memory-ingest, gbrain-sync, telemetry, ngrok tunnel starts, mcp-verify, and supabase-provision refuse to send if the receipt can't be written (each refusal prints problem + cause + fix). User-facing sinks fail open with a stderr warning: the design binary's OpenAI calls, update-check, the read-only dashboards, and git-class receipts. The zero-exception scanner in `test/egress-receipt-wiring.test.ts` fails CI when any new off-machine sink ships unwired.
|
||||
|
||||
Inspect the ledger with `bin/gstack-egress`: `list` (what gstack attempted to send), `verify` (recompute the chain, exit 3 on tamper), `grants` (every consent in force). Threat model: the ledger is forensic observability of ATTEMPTED egress — it records what gstack tried to send so accidents are auditable; it is not an exfiltration control.
|
||||
|
||||
### Unicode sanitization at server egress (v1.38.0.0)
|
||||
|
||||
Page content harvested by CDP can contain lone UTF-16 surrogate halves (orphaned high or low surrogates from broken JavaScript string handling on the page). When those reach `JSON.stringify`, Bun emits them as `\uD800`-style escape sequences that the downstream consumer's `JSON.parse` accepts, but the Anthropic API rejects with a 400 — turning a single weird page into a session-killing error. Defense is single-point, applied at every server egress that ships page-derived strings.
|
||||
|
|
@ -414,7 +425,7 @@ The `EvalCollector` accumulates test results and writes them in two ways:
|
|||
1. **Incremental:** `savePartial()` writes `_partial-e2e.json` after each test (atomic: write `.tmp`, `fs.renameSync`). Survives kills.
|
||||
2. **Final:** `finalize()` writes a timestamped eval file (e.g. `e2e-20260314-143022.json`). The partial file is never cleaned up — it persists alongside the final file for observability.
|
||||
|
||||
`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack-dev/evals/`.
|
||||
`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack-dev/evals/`. Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `<evalDir>/shards/<slug>/` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards/<slug>/` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline.
|
||||
|
||||
### Test tiers
|
||||
|
||||
|
|
|
|||
26
BROWSER.md
26
BROWSER.md
|
|
@ -705,6 +705,10 @@ Or do it manually: `chrome://extensions` → toggle Developer mode → Load
|
|||
unpacked → navigate to `~/.claude/skills/gstack/extension` → pin the
|
||||
extension → enter the port from `$B status`.
|
||||
|
||||
v1.63 pinned the extension identity via the manifest `key` field, so existing
|
||||
unpacked installs get a new extension ID and panel-local state (saved port)
|
||||
resets once — a one-time in-product notice explains this.
|
||||
|
||||
---
|
||||
|
||||
## Pair-agent
|
||||
|
|
@ -758,6 +762,15 @@ remote agent that tries them gets a 403 plus a fresh entry in the denial log.
|
|||
+ domain only (no raw IP, no full request body), rotates at 10MB with 5
|
||||
generations. Per-device salt at `~/.gstack/security/device-salt` (mode 0600).
|
||||
|
||||
### Tunnel egress receipts (v1.63+)
|
||||
|
||||
Every tunnel session open writes a hash-chained egress receipt (sink
|
||||
`browse-tunnel`) to `~/.gstack/security/egress.jsonl` BEFORE ngrok forwards
|
||||
anything. Fail-closed: if the receipt can't be written, the tunnel listener
|
||||
is torn down and the start is refused. Inspect the ledger with
|
||||
`bin/gstack-egress list` and verify chain integrity with
|
||||
`bin/gstack-egress verify` (exit 3 on tamper).
|
||||
|
||||
See [`docs/REMOTE_BROWSER_ACCESS.md`](docs/REMOTE_BROWSER_ACCESS.md) for the
|
||||
full operator guide.
|
||||
|
||||
|
|
@ -800,6 +813,19 @@ The Terminal pane uses a separate session cookie, `gstack_pty`, minted via
|
|||
PTY, can't dispatch arbitrary `/command` calls. `/health` endpoint MUST NOT
|
||||
surface this token.
|
||||
|
||||
### Extension token bootstrap (v1.63+)
|
||||
|
||||
`GET /health` is liveness/status only — it never carries a token, in any
|
||||
mode. The Side Panel extension bootstraps the root token via
|
||||
`POST /extension-token` on the local listener. The server releases the
|
||||
token only when the caller's Origin is exactly
|
||||
`chrome-extension://<GSTACK_EXTENSION_ID>` — the `key` field in
|
||||
`extension/manifest.json` pins the extension ID (`GSTACK_EXTENSION_ID` in
|
||||
`browse/src/server.ts`; derivation reproducible via
|
||||
`bun browse/scripts/extension-id.ts`) — AND the parsed Host hostname is
|
||||
loopback. Anything else gets a detail-free 403. The endpoint is never
|
||||
added to `TUNNEL_PATHS`, so the tunnel surface 404s it by default-deny.
|
||||
|
||||
### Token registry
|
||||
|
||||
`browse/src/token-registry.ts` handles mint/validate/revoke for all three
|
||||
|
|
|
|||
14
CHANGELOG.md
14
CHANGELOG.md
|
|
@ -1,6 +1,6 @@
|
|||
# Changelog
|
||||
|
||||
## [1.63.0.0] - 2026-08-12
|
||||
## [1.63.0.0] - 2026-08-13
|
||||
|
||||
**Everything gstack sends off your machine now leaves a receipt you can read.**
|
||||
**And the eval harness stopped grading itself a passing grade.**
|
||||
|
|
@ -25,7 +25,7 @@ Source: the assembled branch (`git log 1.62.0.0..HEAD`), the free suite
|
|||
| gstack-owned off-machine sinks with a receipt | 0 | every enumerated sink | tripwire-enforced, zero exceptions |
|
||||
| Eval "no regressions" lines that were self-comparisons | every one | 0 | the harness compared runs against their own in-progress accumulator |
|
||||
| Paid gate runner isolation | one process, one hung file kills the tier | one process per file, group-SIGKILL on stall | + never-started accounting |
|
||||
| Discovery catalog budget | unenforced | pinned at 1,105 token-equivalents | ratchet-protocol on every skill add |
|
||||
| Discovery catalog budget | unenforced | 1,105 token-equivalents measured, 1,150 ceiling | ratchet-protocol on every skill add |
|
||||
| Browser `/health` endpoint | served the root auth token to any localhost caller in headed mode | serves no token in any mode | token bootstrap moved to a pinned-origin POST |
|
||||
|
||||
The eval-store line is the one that matters most for anyone hacking on gstack:
|
||||
|
|
@ -95,6 +95,16 @@ bug fix and the port shortlist were selected and hardened for upstream.
|
|||
the seeder, context-bill, and the catalog gate.
|
||||
- `CLAUDE.md`'s compiled-binaries note corrected: the `browse/dist` binaries have
|
||||
been untracked since v0.11.16.0, so they no longer appear in `git status`.
|
||||
- External-service E2E tests (Codex, Gemini, benchmark providers) are declared
|
||||
periodic-tier with the canonical whole-file guard, so the merge-blocking gate
|
||||
never waits on a third-party CLI. The Codex runner passes
|
||||
`--skip-git-repo-check` (now required in non-git working dirs) and the Gemini
|
||||
runner classifies an unusable CLI (removed flags, retired auth paths) as a
|
||||
skip instead of a false failure.
|
||||
- The PTY test runner parses AskUserQuestion prompts that reflow onto a single
|
||||
logical line and strips DEC cursor-visibility residue, pinned by
|
||||
`test/pty-askuserquestion-single-line.test.ts` — the failure class that
|
||||
previously ate a gate test's whole time budget.
|
||||
- New follow-ups filed in `TODOS.md`: egress ledger rotation (chain-genesis
|
||||
records), a launch-nonce token bootstrap, and eval-watch shard-awareness.
|
||||
|
||||
|
|
|
|||
58
CLAUDE.md
58
CLAUDE.md
|
|
@ -9,6 +9,8 @@ bun run test:evals # run paid evals: LLM judge + E2E (diff-based, ~$4/run max)
|
|||
bun run test:evals:all # run ALL paid evals regardless of diff
|
||||
bun run test:gate # run gate-tier tests only (CI default, blocks merge)
|
||||
bun run test:periodic # run periodic-tier tests only (weekly cron / manual)
|
||||
bun run test:gate:sharded # gate tier via the sharded paid runner (one Bun process per test file)
|
||||
bun run test:periodic:sharded # periodic tier via the sharded paid runner (implies EVALS_ALL=1)
|
||||
bun run test:e2e # run E2E tests only (diff-based, ~$3.85/run max)
|
||||
bun run test:e2e:all # run ALL E2E tests regardless of diff
|
||||
bun run eval:select # show which tests would run based on current diff
|
||||
|
|
@ -48,13 +50,21 @@ MCP servers / skills), a temp `GSTACK_HOME`, and `--strict-mcp-config`.
|
|||
Local eval signal matches CI. Debug against real operator state with
|
||||
`EVALS_HERMETIC=0` (restores the legacy env AND drops the strict-MCP flag).
|
||||
Per-test `env:` overrides merge last, so deliberate contamination
|
||||
(`CONDUCTOR_WORKSPACE_PATH`, per-test `GSTACK_HOME`) keeps working. Wiring
|
||||
is pinned by `test/hermetic-wiring.test.ts` (static tripwire) and two
|
||||
gate-tier canaries in `test/skill-e2e-hermetic-canary.test.ts`.
|
||||
(`CONDUCTOR_WORKSPACE_PATH`, per-test `GSTACK_HOME`) keeps working. The
|
||||
hermetic config dir seeds NO skills by default; a PTY test that types a
|
||||
`/skill` slash command must pass `seedSkills: true` to the PTY runner, which
|
||||
points the child's `CLAUDE_CONFIG_DIR` at `hermeticSkillsConfigDir()` — a
|
||||
seeded registry that symlinks the LIVE working tree's SKILL.md files (by
|
||||
design: the skills ARE the subject under test; a snapshot would measure stale
|
||||
copies). Wiring is pinned by `test/hermetic-wiring.test.ts` (static tripwire),
|
||||
two gate-tier canaries in `test/skill-e2e-hermetic-canary.test.ts`, and the
|
||||
seeding tripwires in `test/hermetic-skills-seeding.test.ts` /
|
||||
`test/pty-skill-seeding-wiring.test.ts`.
|
||||
|
||||
E2E tests stream progress in real-time (tool-by-tool via `--output-format stream-json
|
||||
--verbose`). Results are persisted to `~/.gstack-dev/evals/` with auto-comparison
|
||||
against the previous run.
|
||||
against the previous finalized run (in-flight `_partial` files are never used as
|
||||
a baseline, so a run can't compare against itself).
|
||||
|
||||
**Diff-based test selection:** `test:evals` and `test:e2e` auto-select tests based
|
||||
on `git diff` against the base branch. Each test declares its file dependencies in
|
||||
|
|
@ -110,6 +120,7 @@ gstack/
|
|||
│ ├── host-adapters/ # Host-specific adapters (OpenClaw tool mapping)
|
||||
│ ├── resolvers/ # Template resolver modules (preamble, design, review, gbrain, etc.)
|
||||
│ ├── skill-check.ts # Health dashboard
|
||||
│ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard)
|
||||
│ └── dev-skill.ts # Watch mode
|
||||
├── test/ # Skill validation + eval tests
|
||||
│ ├── helpers/ # skill-parser.ts, session-runner.ts, llm-judge.ts, eval-store.ts
|
||||
|
|
@ -147,7 +158,7 @@ gstack/
|
|||
│ ├── test/ # Integration tests
|
||||
│ └── dist/ # Compiled binary
|
||||
├── extension/ # Chrome extension (side panel + activity feed + CSS inspector)
|
||||
├── lib/ # Shared libraries (worktree.ts)
|
||||
├── lib/ # Shared libraries (worktree.ts, egress-receipt.ts, context-bill.ts, redact-engine.ts)
|
||||
├── docs/designs/ # Design documents
|
||||
├── setup-deploy/ # /setup-deploy skill (one-time deploy config)
|
||||
├── .github/ # CI workflows + Docker image
|
||||
|
|
@ -184,6 +195,16 @@ behavior). If you blow past 40K, the right fix is usually: (1) look at WHAT grew
|
|||
or as a reference doc, (3) only compress carefully-tuned prose as a last resort —
|
||||
cuts to the coverage audit, review army, or voice directive have real quality cost.
|
||||
|
||||
A second, harder ceiling guards the DISCOVERY surface: `test/catalog-budget.test.ts`
|
||||
caps the aggregate frontmatter `name` + `description` across all skills at 1,150
|
||||
token-equivalents (260-byte per-skill sub-cap), counted through the shared census
|
||||
in `test/helpers/skill-census.ts`. This one is enforced, not a warning — every
|
||||
host loads the full catalog every session, so growth here taxes every
|
||||
conversation. The failure message carries the re-measure + ratchet protocol.
|
||||
`bin/gstack-context-bill` shows the full token bill-of-materials for a skills
|
||||
tree (always-on vs per-invocation, `--diff`, `--budget`; `--exact` opts into the
|
||||
real tokenizer and POSTs file text to api.anthropic.com with an egress receipt).
|
||||
|
||||
**Merge conflicts on SKILL.md files:** NEVER resolve conflicts on generated SKILL.md
|
||||
files by accepting either side. Instead: (1) resolve conflicts on the `.tmpl` templates
|
||||
and `scripts/gen-skill-docs.ts` (the sources of truth), (2) run `bun run gen:skill-docs`
|
||||
|
|
@ -319,6 +340,21 @@ response in `server.ts`, read
|
|||
`browse/test/server-sanitize-surrogates.test.ts` pins the wiring with invariant
|
||||
tests, so bypasses fail CI.
|
||||
|
||||
**Egress receipts at every off-machine sink** (v1.63.0.0+). Every gstack-initiated
|
||||
send off the machine MUST write a hash-chained receipt to
|
||||
`~/.gstack/security/egress.jsonl` BEFORE the send: TypeScript callers use
|
||||
`writeReceipt` from `lib/egress-receipt.ts`; shell scripts source
|
||||
`bin/gstack-egress-lib.sh` and use `_receipted_curl` / `_receipted_git`. Failure
|
||||
polarity is per-class: fail-closed for sensitive sinks (brain-sync, memory-ingest,
|
||||
gbrain-sync, telemetry, ngrok tunnels, mcp-verify, supabase-provision), fail-open
|
||||
+ stderr warning for user-facing ones (design OpenAI calls, update-check,
|
||||
dashboards, git-class ops). The zero-exception scanner in
|
||||
`test/egress-receipt-wiring.test.ts` fails CI on any unreceipted `curl` /
|
||||
`git push` / `fetch` to a non-loopback host — if you add a new off-machine sink,
|
||||
wire it through the helpers and add it to the enumerated sink list. Inspect with
|
||||
`bin/gstack-egress` (`list` | `verify`, exit 3 on tamper | `grants`). Threat
|
||||
model: forensic observability of ATTEMPTED egress, not an exfiltration control.
|
||||
|
||||
**SSE endpoint helper** (v1.51.0.0+). New SSE endpoints in `server.ts` MUST route
|
||||
through `createSseEndpoint(req, config)` from `browse/src/sse-helpers.ts`. The
|
||||
helper owns the cleanup contract (abort + enqueue-throw + heartbeat-throw, all
|
||||
|
|
@ -870,7 +906,17 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
|||
machine-wide `gstack-evals` lock (concurrent worktrees serialize instead of
|
||||
saturating the shared model API), a per-tier watchdog, and a **run-scoped** log
|
||||
under `~/.gstack-dev/eval-runs/` (no shared-`/tmp` collision). Each prints its
|
||||
log path. Or call `gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
|
||||
log path. `eval:bg:gate` / `eval:bg:periodic` run their tier through the
|
||||
sharded paid runner (`scripts/test-paid-shards.ts`, also exposed as
|
||||
`test:gate:sharded` / `test:periodic:sharded`): one Bun process per test
|
||||
file, an external wall-clock timeout that kills the shard's process GROUP
|
||||
(stray `claude`/`codex` grandchildren included), a per-shard
|
||||
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
||||
constructor, and an aggregate that separates failed vs timed-out vs
|
||||
never-started shards — the detach timeouts (25200s gate / 28800s periodic)
|
||||
are sized against worst-case shard wall clock. `eval:list` / `eval:compare` /
|
||||
`eval:summary` read the shard dirs too. Or call
|
||||
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
|
||||
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
|
||||
pass keys in argv).
|
||||
- Then **poll the printed logfile** with a death-aware watcher: break on the
|
||||
|
|
|
|||
|
|
@ -160,6 +160,7 @@ Runs automatically with `bun test`. No API keys needed.
|
|||
- **Skill validation tests** (`test/skill-validation.test.ts`) — Validates that SKILL.md files reference only real commands and flags, and that command descriptions meet quality thresholds.
|
||||
- **Generator tests** (`test/gen-skill-docs.test.ts`) — Tests the template system: verifies placeholders resolve correctly, output includes value hints for flags (e.g. `-d <N>` not just `-d`), enriched descriptions for key commands (e.g. `is` lists valid states, `press` lists key examples).
|
||||
- **Tier-alignment invariant** (`test/e2e-tier-alignment.test.ts`) — For every self-gated `test/skill-e2e-*.test.ts` named in a touchfiles dep list, the file's `EVALS_TIER` self-gate must match its declared tier in `E2E_TIERS`. Kills the "inert demotion" class where a test is re-tiered in `touchfiles.ts` but the file still gates on the old tier and keeps running in the wrong lane. Unmapped or mixed-tier files are reported, never silently skipped.
|
||||
- **Catalog budget** (`test/catalog-budget.test.ts`) — Caps the aggregate discovery surface: the sum of every skill's frontmatter `name` + `description` (what every host loads at discovery, every session) must stay under 1,150 token-equivalents, with a 260-byte per-skill cap. Counting goes through the shared census in `test/helpers/skill-census.ts` (physical files vs authored skills vs registry entries — three deliberately different counts). Adding a skill? The failure message carries the re-measure + ratchet protocol.
|
||||
|
||||
### Tier 2: E2E via `claude -p` (~$3.85/run)
|
||||
|
||||
|
|
@ -184,10 +185,16 @@ seeded `CLAUDE_CONFIG_DIR`, a temp `GSTACK_HOME`, and `--strict-mcp-config`. You
|
|||
operator `~/.claude` config, MCP servers (gbrain, Conductor), skills, `~/.gstack`
|
||||
decision logs, and `CONDUCTOR_*` env never leak into the child, so local eval
|
||||
signal matches CI instead of disagreeing for reasons unrelated to the code under
|
||||
test. Set `EVALS_HERMETIC=0` to debug against your real operator state (this also
|
||||
test. The hermetic `CLAUDE_CONFIG_DIR` seeds no skills by default; a PTY test
|
||||
that types a `/skill` slash command passes `seedSkills: true` to the PTY runner,
|
||||
which swaps in `hermeticSkillsConfigDir()` — a seeded skill registry that
|
||||
symlinks the LIVE working tree's SKILL.md files (by design: the skills are the
|
||||
subject under test, so a snapshot would measure stale copies). Set
|
||||
`EVALS_HERMETIC=0` to debug against your real operator state (this also
|
||||
drops `--strict-mcp-config`). The wiring is pinned by `test/hermetic-wiring.test.ts`
|
||||
(a free static tripwire) and two gate-tier isolation canaries in
|
||||
`test/skill-e2e-hermetic-canary.test.ts`.
|
||||
(a free static tripwire), two gate-tier isolation canaries in
|
||||
`test/skill-e2e-hermetic-canary.test.ts`, and the skill-seeding tripwires in
|
||||
`test/hermetic-skills-seeding.test.ts` / `test/pty-skill-seeding-wiring.test.ts`.
|
||||
|
||||
### E2E observability
|
||||
|
||||
|
|
@ -227,8 +234,16 @@ bun run eval:bg:gate # detached gate-tier suite
|
|||
bun run eval:bg:periodic # detached periodic-tier suite
|
||||
```
|
||||
|
||||
Each prints its log path. Humans running `bun run test:evals` foreground in their
|
||||
own terminal don't need this — Ctrl-C is intended there.
|
||||
Each prints its log path. The gate and periodic variants run their tier through
|
||||
the sharded paid runner (`scripts/test-paid-shards.ts`, also available directly
|
||||
as `bun run test:gate:sharded` / `bun run test:periodic:sharded`): one Bun
|
||||
process per test file, an external wall-clock timeout that kills the shard's
|
||||
whole process group (stray `claude`/`codex` grandchildren included), a per-shard
|
||||
eval dir (`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/`), and an aggregate that
|
||||
distinguishes failed vs timed-out vs never-started shards. `eval:list`,
|
||||
`eval:compare`, and `eval:summary` are shard-aware. Humans running
|
||||
`bun run test:evals` foreground in their own terminal don't need this — Ctrl-C
|
||||
is intended there.
|
||||
|
||||
**Eval comparison commentary:** `eval:compare` generates natural-language Takeaway sections interpreting what changed between runs — flagging regressions, noting improvements, calling out efficiency gains (fewer turns, faster, cheaper), and producing an overall summary. This is driven by `generateCommentary()` in `eval-store.ts`.
|
||||
|
||||
|
|
|
|||
|
|
@ -235,7 +235,7 @@ Each skill feeds into the next. `/office-hours` writes a design doc that `/plan-
|
|||
| `/ios-qa` | **iOS Live-Device QA (v1.43.0.0+)** — drive a real iPhone over USB CoreDevice via an embedded `StateServer` in the app. Read Swift source, codegen typed `@Observable` accessors, run the agent loop. Optional `--tailnet` flag exposes the device to OpenClaw or any HTTP-capable agent on your Tailscale tailnet so remote agents can run iOS QA without ever touching the hardware. Capability-tier allowlist (observe/interact/mutate/restore), per-device session lock, audit log. |
|
||||
| `/ios-fix`, `/ios-design-review`, `/ios-clean`, `/ios-sync` | iOS bug-fix loop, designer's-eye HIG audit, debug-bridge cleanup, and accessor resync. See `docs/skills.md`. End-to-end walkthrough: [docs/howto-ios-testing-with-gstack.md](docs/howto-ios-testing-with-gstack.md). |
|
||||
|
||||
### New binaries (v0.19)
|
||||
### Standalone binaries
|
||||
|
||||
Beyond the slash-command skills, gstack ships standalone CLIs for workflows that don't belong inside a session:
|
||||
|
||||
|
|
@ -243,6 +243,8 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that
|
|||
|---------|-------------|
|
||||
| `gstack-model-benchmark` | **Cross-model benchmark** — run the same prompt through Claude, GPT (via Codex CLI), and Gemini; compare latency, tokens, cost, and (optionally) LLM-judge quality score. Auth detected per provider, unavailable providers skip cleanly. Output as table, JSON, or markdown. `--dry-run` validates flags + auth without spending API calls. |
|
||||
| `gstack-taste-update` | **Design taste learning** — writes approvals and rejections from `/design-shotgun` into a persistent per-project taste profile. Decays 5%/week. Feeds back into future variant generation so the system learns what you actually pick. |
|
||||
| `gstack-egress` | **Egress receipt auditor** — every gstack-initiated off-machine send writes a tamper-evident, hash-chained receipt to `~/.gstack/security/egress.jsonl` before the send. `list` shows what gstack attempted to send and to which host, `grants` shows every consent grant in force plus the exact command that revokes it, `verify` recomputes the hash chain and exits 3 on tamper. |
|
||||
| `gstack-context-bill` | **Token bill-of-materials** — read-only, offline audit of what an installed skills tree costs in tokens: always-on frontmatter every session pays vs per-invocation SKILL.md + forced references. `--diff` compares two trees, `--budget` enforces a ceiling, `--exact` opts into Anthropic `count_tokens` (sends file text off-machine; writes an egress receipt first, degrades to the offline estimate if the receipt can't be written). |
|
||||
| `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](docs/howto-ios-testing-with-gstack.md). |
|
||||
| `gstack-ios-qa-mint` | **iOS allowlist manager** — owner-grant CLI for the tailnet allowlist. `grant`/`revoke`/`list` against `~/.gstack/ios-qa-allowlist.json` (mode 0600). Remote agents never auto-allowlist; this is the explicit-intent path. |
|
||||
| `gstack-ios-qa-regen` | **iOS bridge regenerator** — deterministically installs the canonical DebugBridge package, generates typed state accessors, and records the installed gstack version. Safe to rerun after source changes or upgrades. |
|
||||
|
|
@ -450,6 +452,7 @@ gstack includes **opt-in** usage telemetry to help improve the project. Here's e
|
|||
- **What's sent (if you opt in):** skill name, duration, success/fail, gstack version, OS. That's it.
|
||||
- **What's never sent:** code, file paths, repo names, branch names, prompts, or any user-generated content.
|
||||
- **Change anytime:** `gstack-config set telemetry off` disables everything instantly.
|
||||
- **Every off-machine send is receipted.** Any gstack-initiated network send — telemetry included — writes a hash-chained, tamper-evident receipt to `~/.gstack/security/egress.jsonl` before the send. Audit with `gstack-egress list`, verify the chain with `gstack-egress verify` (exit 3 on tamper), see every consent in force with `gstack-egress grants`. The ledger records attempted sends so accidents are auditable — it's an audit trail, not a network firewall.
|
||||
|
||||
Data is stored in [Supabase](https://supabase.com) (open source Firebase alternative). The schema is in [`supabase/migrations/`](supabase/migrations/) — you can verify exactly what's collected. The Supabase publishable key in the repo is a public key (like a Firebase API key) — row-level security policies deny all direct access. Telemetry flows through validated edge functions that enforce schema checks, event type allowlists, and field length limits.
|
||||
|
||||
|
|
|
|||
40
TODOS.md
40
TODOS.md
|
|
@ -875,32 +875,6 @@ plus a TTL so abandoned PTYs eventually exit.
|
|||
|
||||
---
|
||||
|
||||
### v1.1+: Audit `/health` token distribution
|
||||
|
||||
**What:** Codex's outside-voice review on cc-pty-import flagged that
|
||||
`/health` already surfaces `AUTH_TOKEN` to any localhost caller in headed
|
||||
mode (`server.ts:1657`). That's a pre-existing soft leak — anything
|
||||
running on localhost gets the root token by hitting `/health`.
|
||||
|
||||
**Why:** cc-pty-import sidesteps it by NOT putting the PTY token there
|
||||
(uses an HttpOnly cookie path instead). But the underlying leak is still
|
||||
shippable surface. A second extension or a localhost web app could
|
||||
currently scrape `AUTH_TOKEN` and hit any browse-server endpoint.
|
||||
|
||||
**Pros:** Closes a real privilege-escalation path on multi-extension
|
||||
machines. **Cons:** Either we tighten the gate (Origin must be OUR
|
||||
extension id, not just any chrome-extension://) or we move bootstrap
|
||||
discovery off `/health` entirely. Either has migration cost for tests
|
||||
and the existing extension.
|
||||
|
||||
**Context:** codex finding #2 on cc-pty-import plan-eng review. Not in
|
||||
scope of that PR; deliberately deferred to keep PTY-import small.
|
||||
|
||||
**Priority:** P2.
|
||||
**Effort:** M.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
## P2: Per-finding AskUserQuestion count assertion for /plan-ceo-review
|
||||
|
|
@ -2655,7 +2629,7 @@ five green files at the tail of a release. Zero user-facing value; pure DRY.
|
|||
|
||||
**Effort:** S (human ~3h, CC ~20min). **Depends on:** None.
|
||||
|
||||
## Egress-receipt follow-ups (filed via /plan-eng-review + /codex on the v1.62 port wave)
|
||||
## Egress-receipt follow-ups (filed via /plan-eng-review + /codex on the v1.63 port wave)
|
||||
|
||||
### P2: egress ledger rotation with chain-genesis records
|
||||
|
||||
|
|
@ -2664,7 +2638,7 @@ five green files at the tail of a release. Zero user-facing value; pure DRY.
|
|||
each new generation's FIRST record embeds the prior file's tail hash so
|
||||
`gstack-egress verify` can walk across generations.
|
||||
|
||||
**Why:** v1.62 ships WARN-at-25MB (visible growth) but nothing bounds the file.
|
||||
**Why:** v1.63 ships WARN-at-25MB (visible growth) but nothing bounds the file.
|
||||
Rotation was deliberately deferred: it changes the verify contract, and a wrong
|
||||
implementation makes healthy ledgers verify as "broken".
|
||||
|
||||
|
|
@ -2676,7 +2650,7 @@ implementation makes healthy ledgers verify as "broken".
|
|||
design sketch in its rotation TODO comment. Start from the `attempts.jsonl`
|
||||
rotation precedent.
|
||||
|
||||
**Effort:** S (human ~4h, CC ~25min). **Depends on:** v1.62 port wave landed.
|
||||
**Effort:** S (human ~4h, CC ~25min). **Depends on:** v1.63 port wave landed.
|
||||
|
||||
### P3: launch-nonce token bootstrap (local-process impersonation)
|
||||
|
||||
|
|
@ -2685,13 +2659,13 @@ mints a nonce at headed launch, seeds it into the extension (CDP
|
|||
`chrome.storage` injection or a launcher-written sidecar), and the endpoint
|
||||
requires it alongside the pinned origin.
|
||||
|
||||
**Why:** v1.62's pinned-origin check authenticates browser contexts; any local
|
||||
**Why:** v1.63's pinned-origin check authenticates browser contexts; any local
|
||||
PROCESS can still forge an Origin header with curl. That threat is explicitly
|
||||
outside the current model (any local process can hit the port anyway) — this
|
||||
TODO documents the deliberate boundary and the designed path across it.
|
||||
|
||||
**Pros:** Closes the local-process impersonation path (strongest of the three
|
||||
options evaluated in the v1.62 plan review).
|
||||
options evaluated in the v1.63 plan review).
|
||||
**Cons:** Largest bootstrap change; CDP seeding is fiddly across the three
|
||||
launch paths (`--load-extension`, baked-in Browser.app, real-Chrome fallback);
|
||||
low present-day value.
|
||||
|
|
@ -2708,7 +2682,7 @@ low present-day value.
|
|||
~line 17) about the sharded layout: watch `<evalDir>/shards/*/_partial-e2e.json`
|
||||
and aggregate live progress across shard subdirs.
|
||||
|
||||
**Why:** v1.62's sharded runner gives each shard its own eval subdir (so shards
|
||||
**Why:** v1.63's sharded runner gives each shard its own eval subdir (so shards
|
||||
baseline against their own priors); `findPreviousRun`, `eval-compare`,
|
||||
`eval-list`, and `eval-summary` were all made shard-aware, but the live watcher
|
||||
intentionally stayed flat — it shows nothing during sharded runs.
|
||||
|
|
@ -2721,7 +2695,7 @@ log already streams per-shard results).
|
|||
`scripts/test-paid-shards.ts` (slug = test filename); `listEvalJsonFiles` in
|
||||
`test/helpers/eval-store.ts` already enumerates the layout — reuse it.
|
||||
|
||||
**Effort:** S (human ~2h, CC ~15min). **Depends on:** v1.62 port wave landed.
|
||||
**Effort:** S (human ~2h, CC ~15min). **Depends on:** v1.63 port wave landed.
|
||||
|
||||
## v1.63 port-wave review follow-ups (deferred from /ship review army — non-blocking polish)
|
||||
|
||||
|
|
|
|||
|
|
@ -132,6 +132,8 @@ Storage: `~/.gstack/gbrain-repo-policy.json`, mode 0600, schema-versioned so fut
|
|||
|
||||
The skill runs three stages — code, memory, brain-sync — independently. A failure in one doesn't block the others. State persists to `~/.gstack/.gbrain-sync-state.json` so re-running picks up cleanly.
|
||||
|
||||
Stages that can send data off-machine (code sync into a possibly-remote gbrain DB, memory ingest, the brain-sync push) each write a tamper-evident receipt to the egress ledger (`~/.gstack/security/egress.jsonl`) before sending, fail-closed: if the receipt can't be written, the stage refuses with `EGRESS_RECEIPT_FAILED` instead of syncing unrecorded. Fix is usually `chmod -R u+w ~/.gstack/security`, then re-run. Inspect receipts with `gstack-egress list`.
|
||||
|
||||
**What it does on a fresh worktree:**
|
||||
|
||||
1. **Pre-flight.** Checks `gbrain_local_status` (the local engine's health). If the engine is `broken-db` or `broken-config`, the skill STOPs with a remediation menu — it refuses to silently degrade. If the local engine is missing and you're in remote-MCP mode (Path 4), the code stage SKIPs cleanly and only brain-sync runs.
|
||||
|
|
|
|||
|
|
@ -179,6 +179,7 @@ Each agent owns the tabs it creates. Rules:
|
|||
- **Path traversal guarded** on `/welcome` — `GSTACK_SLUG` must match `^[a-z0-9_-]+$` or falls back to the built-in template.
|
||||
- **SSRF guards** on `goto`, `download`, and scrape paths — validates URL target against a localhost/private-range blocklist.
|
||||
- **Tunnel surface denial logging.** Every rejection on the tunnel listener (`path_not_on_tunnel`, `root_token_on_tunnel`, `missing_scoped_token`, `disallowed_command:*`) is appended to `~/.gstack/security/attempts.jsonl` with timestamp, source IP, path, method. Rate-capped at 60 writes/min.
|
||||
- **Egress receipt on tunnel start (v1.63+).** Every tunnel session open writes a hash-chained receipt (sink `browse-tunnel`) to `~/.gstack/security/egress.jsonl` BEFORE ngrok forwards anything. Fail-closed: if the receipt can't be written, the tunnel refuses to start. Audit with `bin/gstack-egress list` / `bin/gstack-egress verify`.
|
||||
- All agent activity is logged with attribution (clientId).
|
||||
|
||||
**Known non-goal (tracked as #1136):** on Windows, the cookie-import-browser path launches Chrome with `--remote-debugging-port=<random>`. With App-Bound Encryption v20, a same-user local process can connect to that port and exfiltrate decrypted v20 cookies — an elevation path relative to reading the SQLite DB directly. Fix direction is `--remote-debugging-pipe` instead of TCP.
|
||||
|
|
|
|||
|
|
@ -92,6 +92,29 @@ your local commit still exists — the next skill run will retry the push.
|
|||
|
||||
---
|
||||
|
||||
## `gstack: brain-sync push NOT sent — the egress receipt could not be written`
|
||||
|
||||
**Problem.** The push was refused before anything left your machine. Every
|
||||
brain-sync push writes a tamper-evident receipt to the egress ledger
|
||||
(`~/.gstack/security/egress.jsonl`) before sending, fail-closed. The
|
||||
receipt could not be written, so nothing was sent, no local commit was
|
||||
made, and the queue is preserved — the next run retries the whole drain.
|
||||
`gstack-brain-sync --status` shows `EGRESS_RECEIPT_FAILED` as the failure
|
||||
detail.
|
||||
|
||||
**Cause.** `~/.gstack/security/` is missing or unwritable, the disk is
|
||||
full, or `GSTACK_HOME` points at a read-only location.
|
||||
|
||||
**Fix.**
|
||||
```bash
|
||||
chmod -R u+w ~/.gstack/security
|
||||
```
|
||||
Then run any skill (or `gstack-brain-sync --once`) to retry. Inspect the
|
||||
ledger with `gstack-egress list`; verify its hash chain with
|
||||
`gstack-egress verify`.
|
||||
|
||||
---
|
||||
|
||||
## `gstack-brain-init: ~/.gstack/.git is already a git repo pointing at <url>`
|
||||
|
||||
**Problem.** You tried to init with a remote URL that doesn't match the
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ GBrain.
|
|||
By design, these stay local even when sync is on:
|
||||
|
||||
- Credentials: `.auth.json`, `auth-token.json`, `sidebar-sessions/`,
|
||||
`security/device-salt`, consumer tokens in `config.yaml`
|
||||
`security/device-salt`
|
||||
- Machine-specific state: Chromium profiles, ONNX model weights,
|
||||
caches, eval-cache, CDP-profile, one-time prompt markers
|
||||
(`.welcome-seen`, `.telemetry-prompted`, `.vendoring-warned-*`, etc.)
|
||||
|
|
@ -80,9 +80,7 @@ On machine B:
|
|||
```
|
||||
3. Run `gstack-brain-restore`. That clones the repo, rehydrates your
|
||||
learnings/plans/retros, and re-registers the git merge drivers.
|
||||
4. Re-enter consumer tokens (they're machine-local and NOT synced —
|
||||
`gstack-config set gbrain_token <your-token>`).
|
||||
5. Next skill: your yesterday-on-machine-A learning surfaces. That's the
|
||||
4. Next skill: your yesterday-on-machine-A learning surfaces. That's the
|
||||
magical moment.
|
||||
|
||||
## Status, health, and queue depth
|
||||
|
|
@ -141,6 +139,12 @@ To remediate:
|
|||
There's a defense-in-depth hook at `~/.gstack/.git/hooks/pre-commit` that
|
||||
runs the same scan if you manually `git commit` against the repo.
|
||||
|
||||
Separately (v1.63.0.0+), every push writes a tamper-evident receipt to the
|
||||
egress ledger (`~/.gstack/security/egress.jsonl`) *before* anything is
|
||||
sent, fail-closed: if the receipt can't be written, the push is refused
|
||||
and the queue is preserved. Inspect the ledger with `gstack-egress list`
|
||||
and verify its hash chain with `gstack-egress verify`.
|
||||
|
||||
## Two-machine conflicts
|
||||
|
||||
If you write on machine A and machine B the same day, both will push
|
||||
|
|
|
|||
Loading…
Reference in New Issue