mirror of https://github.com/garrytan/gstack.git
3 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Garry Tan
|
1d41ee3ab3
|
v1.63.0.0 feat: GStack 2 fork port wave — egress receipts, context-bill, sharded gate, /health fix (#2541)
* test(helpers): shared skill-census helper with three explicit counts
physicalSkillFiles (symlinked dirs included, root router included),
authoredSkills (realpath-deduped, router excluded), registryEntries
(what ./setup registers: unique frontmatter names + _gstack-command).
One counting authority for the hermetic seeder, context-bill ground
truth, and the catalog-budget test — connect-chrome's dir symlink and
the root router otherwise produce three subtly different hand-rolled
censuses. Ported-wave foundation (C11).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(evals): stop the harness grading itself
findPreviousRun excluded only the file being written, by name, so every
suite compared against _partial-e2e.json — the current run's own
accumulator, relabelled with the current tier just before each flush.
That is why every block read '+$0.00, +0s, Stable run, no regressions.'
This harness has never been able to detect a regression, and reassuring
output that cannot fail is worse than none. In-progress runs are now
excluded by role, and a run with nothing to compare against says NO
BASELINE instead of claiming stability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit f3140b5245221fff7fb9411c7ec07c2ca11587b5)
* refactor(evals): shared partial-run predicate + finalized-run lookup
isPartialEval(data, filename) is the one place that decides what counts
as an in-progress accumulator (the _partial flag OR a _partial-prefixed
filename), and findLatestFinalizedRun(evalDir, tier) is the one place
that finds the newest real run — scanning the eval dir plus one level of
shards/<slug>/ subdirs, where the sharded paid runner points each
shard's collector. skill-budget-regression.test.ts's hand-rolled
findLatestRun (flag-blind: a flagged-but-renamed accumulator passed its
name check) is replaced by the shared helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit b55fcf6966366fd21a8cdc46de61aab6e1b1d100)
* feat(evals): register shipped skills for hermetic PTY children
Hermetic children get a config dir that deliberately seeds no skills —
right for children that install their own, fatal for the PTY family that
TYPES /office-hours or /plan-ceo-review: claude rejects the command as
Unknown before any model turn, so the plan-family gate smokes measure
nothing. hermeticSkillsConfigDir() is a second, opt-in config dir under
the same runRoot that mirrors ./setup's registration exactly (real dir
per registry name, SKILL.md + sections/ symlinks, frontmatter-name
resolution, _gstack-command root alias), driven by the shared
skill-census so connect-chrome's dir symlink collapses the same way
setup's idempotent overwrite does.
Ported from fork commit 03c4eca2, tree walk rewritten for the upstream
layout (top-level <skill>/SKILL.md dirs, no skills/ tree). Unit tests
are new: seed shape, census parity, symlink resolution, connect-chrome
collapse, idempotence, no-API-key seed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 93dae6107b30ce453a07c2d342b60262bba6ce0b)
* feat(evals): seedSkills opt-in for PTY slash-command tests + tripwire
Wire ClaudePtyOptions.seedSkills through launchClaudePty: when set (and
hermetic, and no per-test CLAUDE_CONFIG_DIR override), the child gets
hermeticSkillsConfigDir() so typed /skill slash commands resolve instead
of dying as Unknown command before any model turn. Opted in at the three
runPlanSkill* helpers and the four direct-launch slash-command tests
(plan-design-with-ui, plan-ceo-mode-routing, autoplan-chain,
ship-idempotency).
New static tripwire (test/pty-skill-seeding-wiring.test.ts): any test
file that sends a slash command over the PTY must route through a
runPlanSkill* helper or pass seedSkills: true — an unseeded slash-command
test spends money and measures nothing. hermetic-wiring.test.ts now
blesses the repo-tree seeding path explicitly (config dir under runRoot,
symlinks into the repo checkout, never operator ~/.claude).
The CI "Register gstack skills for PTY smoke" step keeps a keep-me note:
container cross-mount symlinks defeat the TUI scanner and HOME is not
hermeticized, so the real-file copies there must survive this change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 63c52269daaffb833b3105ea9b4b99be6df8fec7)
* refactor(evals): single shared paid-test-set module
test/helpers/paid-test-set.ts is now the one definition of which test
files are paid (the exact globs package.json's test:gate expands).
scripts/test-free-shards.ts derives its free/paid exclusion from it
instead of a private regex list, dropping the dead
browse/test/security-review-fullstack.test.ts pattern (file no longer
exists). The sharded paid runner derives its enumeration from the same
module, so a file added to one list can no longer silently miss the
other.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit a7f36479a6a1f3656452370f5883371f3cb65623)
* feat(evals): env-driven lazy eval dir + shard-aware store and tooling
Importing eval-store no longer spawns the gstack-slug subprocess: the
module-level DEFAULT_EVAL_DIR constant is now a memoized defaultEvalDir()
resolved at collector construction. Resolution order: explicit
constructor arg, then GSTACK_EVAL_DIR, then slug detection — so the
sharded paid runner can point each shard child at its own
<evalDir>/shards/<slug>/ dir with plain env, no --preload.
Runs collected under a shards/ subdir record their slug in the eval
JSON (EvalResult.shard). findPreviousRun scans one shards/<slug>/ level
and prefers same-slug priors, so each shard baselines against its own
history instead of whichever shard flushed last. eval:list,
eval:summary, and eval:compare enumerate the same one level of shard
subdirs; eval:compare's no-arg mode also stops picking an in-progress
accumulator as the after-run.
eval-watch stays flat (documented follow-up): it tails a single dir for
live progress and gains nothing from per-shard baselines until the
runner emits a merged stream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e1f53f7d9c7fe6b65877d843f2e25bd2e2d12ffd)
* feat(evals): sharded paid tier runner
scripts/test-paid-shards.ts runs the gate/periodic tier one Bun process
per test file, with an EXTERNAL wall-clock timeout that SIGKILLs the
shard's detached process group and an aggregate that distinguishes
passed / failed / timed-out / never-started — partial execution can no
longer read as a pass. Bun's native --shard/--isolate covers none of
this: no process-group kill (hung claude/codex PTY grandchildren
survive in-process isolation), no never-started taxonomy, no per-shard
env. Each shard child gets GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/
(slug = test filename sans extension, stable across runs) so shard
baselines compare against their own prior runs.
Output classification lives in scripts/test-strict-output.ts (strict
exit-code derivation, incremental fail-line classifier, child signal
forwarding) so the runner and any future strict bun-test wrapper share
one implementation. Enumeration derives from the shared paid-test-set
module; tier exclusion fires only on an explicit whole-file
EVALS_TIER === '<other>' guard.
package.json gains test:gate:sharded / test:periodic:sharded, and
eval:bg:gate / eval:bg:periodic now run the sharded scripts with detach
timeouts sized to the worst case (gate: 49 shards x 30min / 4 jobs ~
6.2h -> 25200s; periodic: 59 -> 28800s).
test/paid-shards.test.ts pins enumeration, tier classification, and the
kill-and-continue property with a real busy-loop shard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5e76bd5931836257f896cedfe4e93912cb759c70)
* feat(security): hash-chained egress receipt ledger (core)
Port lib/egress-receipt from the v2 fork as TypeScript: writeReceipt
(sync, fail-closed via typed EGRESS_RECEIPT_FAILED), best-effort
writeOutcome, readLedger/listReceipts/verifyLedger, GSTACK_HOME ->
GSTACK_STATE_DIR -> ~/.gstack resolution, 0600 ledger under a 0700
security dir, and an mkdir spin lock (2.5s budget) with documented
>10s-mtime stale-lock reclaim.
Changes vs the fork:
- lastRawLine tail-reads the final 4KB instead of loading the whole
ledger, so appends stay O(1) as the file grows.
- WARN-at-size: past 25MB writeReceipt emits one self-explanatory
stderr warning per process (what the ledger is, how to inspect it,
rotation TODO); verifyLedger gains a sizeWarning field. Rotation
TODO carries the chain-genesis sketch (new generation's first record
embeds the prior file's tail hash).
bin/gstack-egress-receipt is a bun script bridging shell callers:
write|outcome subcommands, exit 3 + EGRESS_RECEIPT_FAILED on stderr on
failure; --no-payload records sha256:null for git-class ops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 619726a3d77d987a2e50151a5727b3faaaf5fc6a)
* chore(bin): delete dead brain-consumer/reader scripts
bin/gstack-brain-consumer and bin/gstack-brain-reader are byte-identical
dead scripts that POST the repo URL + a Bearer token to a /ingest-repo
endpoint gbrain removed (docs/gbrain-sync.md already documents the
removal in past tense). No live references remain; CHANGELOG mentions
are historical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 254ddc69fc5a0270fcc973e36b6a81766d835d2d)
* feat(security): shared shell receipt helpers
bin/gstack-egress-lib.sh (sourced library, gstack-gbrain-lib.sh
precedent) provides _receipted_curl and _receipted_git: write the
egress receipt BEFORE the send via gstack-egress-receipt, hand curl the
SAME payload file via --data-binary @file so the receipt hash matches
the wire bytes exactly, then append a best-effort outcome. Per-call
fail policy: 'closed' refuses the send (return 3, problem/cause/fix
message on stderr) and 'open' warns and proceeds. Payload temp files
are consumed immediately per call — no EXIT traps, since callers like
gstack-telemetry-sync own their own EXIT trap and a sourced trap would
clobber it.
Tested end-to-end against a local Bun.serve listener: receipt sha256
equals the sha256 of the bytes the listener received, fail-closed
refusal never touches the network and carries the problem/cause/fix
stderr shape, fail-open warns and proceeds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6d067dce2d4c8815dec98be551763c85a3671357)
* feat(security): receipt core shell sinks
Wire the three core bash egress sinks through gstack-egress-lib.sh:
- gstack-telemetry-sync: the batch POST now writes the payload to a
temp file, receipts those exact bytes fail-closed, and hands curl the
SAME file. On refusal nothing is sent and the cursor does not
advance, so the batch stays buffered for the next run. The HTTP
status is recorded as the receipt outcome.
- gstack-update-check: fail-open receipts (warn + proceed) on the
Supabase ping POST, both VERSION curls (via a local
_receipted_version_fetch helper that skips non-network schemes), and
git ls-remote. The ping receipt is written inside the backgrounded
subshell, so it can never block the script's exit.
- gstack-brain-sync: fail-closed git-class receipts. The push receipt
is written BEFORE the commit consumes the queue, so a refused receipt
leaves the queue intact and the next run retries the whole drain
(pinned by a new queue-intact-on-refusal test, including the
problem/cause/fix refusal message shape). The retry-path fetch and
retry push carry their own fail-closed receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 3c60f699acceaf1c92a218874711e05fc17dca5d)
* feat(security): receipt TS module sinks + tunnel
writeReceipt (fail-closed, sha256:null — a subprocess or SDK owns the
wire bytes) before every TS-module network-bearing operation:
- bin/gstack-gbrain-sync.ts: before the gbrain code walk that ships
repo content to the user's gbrain DB (may be remote Postgres). A
refused receipt fails the stage with status refused-egress-receipt.
- bin/gstack-memory-ingest.ts: before the gbrain batch import of
transcript pages. A refused receipt returns a system_error verdict
without spawning the import.
- browse/src/server.ts: before both ngrok.forward call sites (start-up
BROWSE_TUNNEL=1 path and the /tunnel/start endpoint). A receipt
failure lands in the existing catch that tears the tunnel listener
back down and refuses the start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5677d618a48fcd0ae2b068bf868781d90f809cb5)
* feat(design): receipted fetch for OpenAI calls
design/src/receipted-fetch.ts wraps every api.openai.com call: a
content-free egress receipt (sink design-openai, sha256 of the JSON
body — hash only, never the body) is written BEFORE the send. Polarity
is FAIL-OPEN: user-facing generation must not die because an audit log
hiccuped, so a receipt failure warns on stderr and the call proceeds.
Streams pass through untouched (response bodies returned as-is;
non-string request bodies receipted as sha256:null rather than drained
to hash).
All ten call sites converted with per-command payload classes:
generate, variants (injected fetchFn passes through), iterate (both
threaded and fresh paths), evolve (image + screenshot analysis), check,
diff, design-to-code, memory.
Unit-tested with injected fetch: receipt-before-send ordering, stream
passthrough, and fail-open on an unwritable ledger.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit c0e5ff6639414ac2fd98e8ac3affb51401746b55)
* feat(security): receipt admin scripts + user git-ops (zero exceptions)
Wire the remaining shell egress through gstack-egress-lib.sh:
- gstack-gbrain-mcp-verify: both JSON-RPC probe POSTs (initialize +
tools/list) receipted fail-closed via payload files (hash == wire
bytes). A refused receipt lands in the NETWORK class — no send.
- gstack-security-dashboard / gstack-community-dashboard: the
community-pulse GETs receipted fail-open (read-only stats must not
break over an audit hiccup).
- gstack-gbrain-supabase-provision: api_call receipted fail-closed.
Each retry attempt hands the helper a fresh copy of the body file
(the helper consumes its payload). The receipt hashes the request
body only — the PAT never reaches the ledger or any log. Refusal
exits 8 without retrying.
- git-class sha256:null receipts, fail-open: gstack-artifacts-init
(ls-remote, initial push, fetch/pull recovery, retry push),
gstack-brain-restore (staging clone, existing-repo fetch),
gstack-session-update (self-update pull).
gstack-team-init needs no wiring: every git clone in it is inside an
echoed instruction string, not an executed command.
The lib now self-locates with shell builtins only (no dirname), so
sourcing works under the whitelist-PATH test harnesses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit b8c5e2055b21ab72878b3e46f8047782ee65a11c)
* test(security): egress wiring tripwire + polarity contract
Static-grep tripwire pinning the egress-receipt wiring (threat model in
the header: the ledger is forensic observability of ATTEMPTED egress,
not an exfiltration control):
- Per-sink assertions: every wired TS module imports egress-receipt and
calls writeReceipt; every wired shell sink sources
gstack-egress-lib.sh with each network op under a receipt;
ngrok-proximity check for server.ts; every design api.openai.com call
routes through receiptedFetch.
- Absence assertions: the dead brain-consumer/reader scripts stay
deleted (lstat, so a dangling symlink also fails).
- Polarity table pinned as data (fail-closed: brain-sync,
memory-ingest, gbrain-sync, telemetry-sync, ngrok, mcp-verify,
supabase-provision; fail-open: design-openai, update-check,
dashboards, git-class user ops, context-bill --exact) plus per-file
polarity spot-checks.
- NEW-SINK SCANNER with zero KNOWN_UNWIRED: sweeps bin/, lib/,
scripts/, design/src, browse/src for curl, absolute-URL fetch(, and
git remote ops (never local rev-parse/get-url; heredoc bodies and
message strings excluded) and requires every hit to be receipted or
in a REASONED exemption list where each entry carries its why.
Preamble-generated skill prose documented out-of-scope in the header.
- Shebang tripwire: no bin/gstack-* file may carry a node shebang.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit ff69ceeafaf9c017d539b6ad77ff8f95b680b979)
* feat(cli): gstack-egress reader
bin/gstack-egress (bun) — the auditor's view of the receipts ledger:
- list: one row per receipt (what gstack ATTEMPTED to send), with
--since/--host/--sink filters and --json.
- verify: recompute the hash chain; exit 3 on tamper naming the first
broken line; prints the sizeWarning when the ledger passes 25MB.
- grants: what CAN leave, built on the upstream config keys only
(telemetry, artifacts_sync_mode, redact_repo_visibility,
redact_prepush_hook via gstack-config get) — each grant names its
file, key, and the exact revoke command.
CLI smoke tests spawn the real bin against a temp GSTACK_HOME,
including a broken-chain fixture asserting exit 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 9e24eca0f1069fea2ea69e7df4e9b256e93d59a3)
* feat(cli): context-bill — token bill-of-materials (stripped port)
lib/context-bill.ts, ported from the v2 fork and STRIPPED to the tiers
this repo's skills can exercise: ALWAYS-ON (per-skill frontmatter bytes
with dead-key and foreign-host-file flags), EAGER (SKILL.md + any
forced 'for every invocation' references), on-disk totals, --diff,
--budget, and --exact with the calibration table. The fork's
CONDITIONAL/TRANSITIVE/LAZY/FAST-PATH parsers understand only its
dispatcher layout and were dropped; the tier fields stay in the report
shape (empty/zero/null) so re-adding a parser is additive.
TOKEN_DIVISORS and their provenance docblock kept; --help notes
recalibration via --exact's calibration block.
Three upstream fixes over the fork:
(a) findSkillDirs treats the walk ROOT as a container — the repo root's
router SKILL.md is billed AND its children are walked (the fork
short-circuited and billed one skill); walkMd skips node_modules
and dot-directories.
(b) installed-tree layout: subdirs that are their own repo checkout
(a gstack/ clone inside ~/.claude/skills, detected by .git) are
skipped, and directory symlinks (connect-chrome) are followed with
a container-recursion cycle guard.
(c) ROUTER_KEYS widened to the upstream frontmatter contract {name,
description, version, allowed-tools, triggers, preamble-tier}.
--exact writes an egress receipt (sink context-bill-exact, host
api.anthropic.com) BEFORE any count_tokens POST; if the receipt cannot
be written the run degrades to the offline estimate with a warning —
nothing is sent unrecorded. bin/gstack-context-bill is the bun shim.
Tests: fixture-tree ledgers, the three fixes, --diff/--budget exit
codes, --exact with injected fetch (envelope subtraction, receipt
ordering, fail-open degradation), CLI smoke test, and ground truth
against THIS repo via test/helpers/skill-census.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 675c19876b87ec927b555f5f64c7f93130b3de90)
* test(catalog): aggregate discovery-surface budget with ratchet protocol
Every host loads every skill's frontmatter name + description at
discovery, every session. applyCatalogTrim in scripts/gen-skill-docs.ts
shapes each description and the 160KB per-file warn covers body size,
but nothing capped the aggregate frontmatter — the catalog could grow
one reasonable-looking description at a time. This test is that
enforcement layer.
Measures the catalog via test/helpers/skill-census.ts authoredSkills
(symlink-deduped, root router counted separately as the _gstack-command
alias line item): 53 skills + router = 4,420 bytes = 1,105
token-equivalents today, asserted <= 1,150 (~4% headroom). Per-skill
sub-cap of 260 bytes (largest today: design-consultation at 229), plus
a non-empty-description check.
Failure messages are self-service ratchets: they print the new total,
the delta, and the update protocol (bump the constant AND the
derivation comment in the same commit; trim instead of grow for
existing descriptions). Parser handles folded block scalars
(description: >-) for fork parity; import-free by design so it
survives generator refactors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit c106fb36f768181b80c257e5cff1cde4f435f9c0)
* fix(browse): extension token bootstrap moves to pinned-origin POST; /health carries no token
GET /health is now liveness/status only in every mode — both token
carve-outs (headed-mode disjunct AND chrome-extension:// Origin
disjunct) are removed. Token bootstrap is POST /extension-token on the
local listener: the Origin header must be exactly
chrome-extension://<GSTACK_EXTENSION_ID> and the Host header's hostname
must parse to 127.0.0.1 or localhost (parsed via new URL, never literal
equality — Host arrives as '127.0.0.1:34567'). Wrong origin/host → 403
with no detail. The tunnel surface 404s the endpoint (not in
TUNNEL_PATHS, verified by test).
The extension ID is pinned by a new "key" field (RSA public key) in
extension/manifest.json; browse/scripts/extension-id.ts reproduces the
ID derivation (first 16 bytes of SHA-256 of the DER public key, hex
mapped 0-9a-f → a-p). The private key is not committed anywhere —
unpacked/baked-in loads only need the public key.
Extension side: background.js bootstraps and refreshes the token via
POST /extension-token (403 → disconnected state); sidepanel.js direct
connect path does the same; sidepanel-terminal.js's dead /health token
fallback (read AUTH_TOKEN/authToken keys the server never sent,
hardcoded port) is replaced with the window.gstackAuthToken path.
MIGRATION NOTE: the manifest key pins the extension ID, so existing
installs' side-panel local state (saved port, snoozes) resets once —
explained in-product via a one-time notice (flag
gstack_id_migrated_v162). After upgrading the server, restart the
browser so the old service worker stops polling for a token GET /health
no longer serves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e9a0b6847a2d17fe6656a4686b4efd0c8380eb09)
* docs: correct stale compiled-binaries claim; file three egress/eval follow-ups
CLAUDE.md's compiled-binaries section claimed browse/dist binaries are
tracked by git and appear as modified in git status — false since
|
|
|
Garry Tan
|
46c1fae7f1
|
v1.54.0.0 feat: carve /ship into skeleton + on-demand sections (-59% always-loaded) (#1806)
* feat(test): transcript-section-logger + ship-action fingerprint (T10) Pure-analysis module over a SkillTestResult/NDJSON transcript: - extractSectionReads(): which sections/*.md a run opened (post-carve check) - extractShipActions(): observable action fingerprint (merge/test/bump/ changelog/commit/push/pr) that works on the MONOLITH too, so a baseline captured before the carve can detect a sectioned-ship regression - baseline read/write + compareShipActions() for baseline-first dogf(T10) Baseline-first answers the Codex outside-voice critique that a logger in the same PR as the carve is post-failure telemetry without a pre-carve reference. 11 unit tests, all green. Paid monolith baseline capture runs separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): section discovery + generation machinery (T9) - discover-skills.ts: discoverSectionTemplates() scans <skill>/sections/*.md.tmpl - gen-skill-docs.ts: extract resolvePlaceholders + applyHostRewrites + buildContext as shared helpers (processTemplate and the new processSectionTemplate both call them, so a sanitization/rewrite fix can't miss sections) [C1] - processSectionTemplate: body-fragment generation (no frontmatter/catalog/voice), parent-skill TemplateContext (skillName pinned to parent, not 'sections', so appliesTo gating + tier behave identically), per-host output routing - --host all now fails the build on ANY host failure, not just claude, so a stale external-host output can't slip the freshness gate [Codex outside-voice #9] Inert until a skill is carved (no sections/ dirs exist yet). Refactor is output-neutral: gen:skill-docs --dry-run --host all reports 0 STALE. 5 discovery unit tests + 389 gen-skill-docs tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(setup): install sections/ for cherry-pick targets (claude + kiro) (T9) Two install targets cherry-pick SKILL.md and would leave a carved skill's sections/ behind, 404ing a runtime 'Read sections/<name>.md': - link_claude_skill_dirs: link the sections/ subdir via _link_or_copy (windows gets a fresh copy on every ./setup) - kiro per-skill loop: sed-rewrite + copy each sections/* so paths resolve under ~/.kiro, not ~/.codex/~/.claude codex/factory/opencode link the whole generated dir, so sections ride free. Addresses Codex outside-voice #4/#6 (runtime pathing landmine). Inert until a skill is carved. Static-tripwire test + windows-fallback invariant green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ship): gstack-version-bump CLI — tested idempotency classify + write (T9) Hybrid CLI extraction (CM1): the deterministic core of ship Step 12 becomes a tested CLI instead of bash prose the agent re-derives each run. - classify: FRESH/ALREADY_BUMPED/DRIFT_STALE_PKG/DRIFT_UNEXPECTED from VERSION vs origin/<base>:VERSION vs package.json.version (pure reader) - write: validated dual-write to VERSION + package.json (FRESH bump) - repair: DRIFT_STALE_PKG sync, no re-bump Bump-LEVEL choice + queue collision stay agent judgment; slot pick stays bin/gstack-next-version. This removes the re-bump-a-shipped-branch footgun from skippable prose into code that can't be skipped or misread. 15 tests (exhaustive state matrix + write/repair fs + real-git classify). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(parity): sectioned-skill parity capability — guards the carve (T9) Carved skills (skeleton + sections/*.md) need parity checks that see relocated content, or moving a phrase into a section reads as 'lost': - readSkillForParity(): union skeleton + all sections/*.md - checkSkillParity sectioned mode: content checks against the union; minBytes/ maxSizeRatio against union bytes (total behavior preserved); maxSkeletonBytes asserts the always-loaded skeleton actually shrank. Lowering minBytes to fit a small skeleton would otherwise make the size floor toothless [Codex #12]. Built + tested BEFORE the carve so ship's invariant can flip to sectioned in the same commit it lands. Monolith path byte-identical (verified: pre-existing investigate 1.053 ratio drift fails the same with this change stashed). 7 sectioned-parity tests + existing parity tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ship): carve into skeleton + on-demand sections (Claude) (T9) ship/SKILL.md drops 167KB → 68.7KB (~59% of the always-loaded skill) by moving 8 prose-heavy steps into ship/sections/*.md, read on demand: tests, test-coverage, plan-completion, review-army, greptile, adversarial, changelog, pr-body. Step 12's version logic now calls the tested gstack-version-bump CLI instead of inline bash. Claude-first (S2): {{SECTION:id}} emits a STOP-Read pointer on Claude (skeleton + generated section files) and INLINES the content on every other host, so external hosts keep the full monolith — verified factory at 162KB with no sections dir. {{SECTION_INDEX:ship}} renders the situation→section table from the PASSIVE manifest (CM2 / v2_PLAN.md:663); required-reads live only in test fixtures. Multi-pass resolve expands inlined sections' own resolvers. Parity: ship invariant flipped to sectioned (union content checks + maxSkeletonBytes asserts the shrink). Carve-fallout fixed across gen-skill-docs/skill-validation/ golden/plan-completion/#1539/size-budget tests via skeleton+sections union reads. Free suite green except the pre-existing investigate parity drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ship): manifest-consistency + context-parity + requiredReads helper (T9) Free deterministic guards for the carve: - required-reads.ts + unit test: assertRequiredReads(run, requiredFiles) — the mechanical layer-5 check that the agent Read the sections its situation needs (required set comes from the fixture, not the passive manifest) - section-manifest-consistency: 3-tier orphan classification (generated orphan + hand-edited generated file → FAIL; manifest orphan → WARN per v2_PLAN.md) and pins the PASSIVE-manifest contract (no applies_when/required_for) - template-context-parity: generated sections have zero unresolved placeholders and gated resolvers (ADVERSARIAL_STEP/CONFIDENCE_CALIBRATION/CHANGELOG_WORKFLOW) rendered — proving sections resolve with the parent skillName, not 'sections' 16 tests, all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ship): section-loading E2E + idempotency CLI detection (T9) - skill-e2e-ship-section-loading.test.ts (new, periodic): runs real /ship in plan mode against a fresh version-changing fixture and asserts the agent Read the required sections (review-army + changelog). Runs against the INSTALLED skill (~/.claude/skills/gstack/ship), not repo paths, so install-layout 404s surface [Codex outside-voice #5]. Layer-5 mechanical guard against silent section-skip. - skill-e2e-ship-idempotency.test.ts: detection updated for the carve — Step 12 now runs gstack-version-bump classify (JSON "state":"ALREADY_BUMPED") instead of the inline bash echo (STATE: ALREADY_BUMPED). Accept both; add a gstack-version-bump-write re-bump regression signal. - touchfiles: register ship-section-loading (periodic) + extend idempotency deps with bin/gstack-version-bump + scripts/resolvers/sections.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ship): union-read redaction wiring test for the carve (T9) main's PR-body redaction-at-sink lives in sections/pr-body.md.tmpl after the carve, not the skeleton template. Read skeleton + section templates union so the redaction-wiring assertions follow the relocated content. 9/9 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v1.54.0.0 feat: carve /ship into skeleton + on-demand sections (-59% always-loaded) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
|
|
Garry Tan
|
dde55103fc
|
v1.15.0.0 feat: slim preamble + real-PTY plan-mode E2E harness (#1215)
* chore: add gstack skill routing rules to CLAUDE.md
Per routing-injection preamble — once-per-project addition that lets
agents auto-invoke the right gstack skill instead of answering generically.
* refactor: slim preamble resolvers + sidecar-symlink helper
Compress prose across 18 preamble resolvers — Voice, Writing Style,
AskUserQuestion Format, Completeness Principle, Confusion Protocol,
Context Health, Context Recovery, Continuous Checkpoint, Lake Intro,
Proactive Prompt, Routing Injection, Telemetry Prompt, Upgrade Check,
Vendoring Deprecation, Writing Style Migration, Brain Sync Block,
Completion Status, and Question Tuning. Same semantic contract, ~half
the bytes. Restored "Treat the skill file as executable instructions"
phrase in the plan-mode info section after diagnosing it as load-bearing.
Restored "Effort both-scales" rule in AskUserQuestion format.
Bonus: scripts/skill-check.ts gains isRepoRootSymlink() so dev installs
that mount the repo root at host/skills/gstack as a runtime sidecar
(e.g., codex's .agents/skills/gstack) get skipped instead of double-counted.
opus-4-7 model overlay gets a Fan-Out directive — explicit instruction
to launch parallel reads/checks before synthesis.
Net token impact across all generated SKILL.md files: ~140K tokens
removed across 47 outputs. Plan-* skills retain full preamble surface
(Brain Sync, Context Recovery, Routing Injection) — load-bearing
functionality that early slim attempts incorrectly cut.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: regenerate SKILL.md outputs after preamble slim
bun run gen:skill-docs --host all output. Mirrors the resolver changes
in the previous commit. 47 generated SKILL.md files plus 3 ship-skill
golden fixtures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(test): real-PTY harness for plan-mode E2E tests
Adds test/helpers/claude-pty-runner.ts. Spawns the actual claude binary
via Bun.spawn({terminal:}) (Bun 1.3.10+ has built-in PTY — no node-pty,
no native modules), drives it through stdin/stdout, and parses rendered
terminal frames. Pattern adapted from the cc-pty-import branch's
terminal-agent.ts but stripped of WS/cookie/Origin scaffolding (not
needed for headless tests).
Public API:
- launchClaudePty(opts) — boots claude with --permission-mode plan|null,
auto-handles the workspace-trust dialog, returns a session handle.
- session.send / sendKey / waitForAny / waitFor / mark / visibleSince /
visibleText / rawOutput / close
- runPlanSkillObservation({skillName, inPlanMode, timeoutMs}) — high-level
contract for plan-mode skill tests. Returns { outcome, summary, evidence,
elapsedMs }. outcome ∈ {asked, plan_ready, silent_write, exited, timeout}.
Replaces the SDK-based runPlanModeSkillTest from plan-mode-helpers.ts
which never worked. Plan mode renders its native "Ready to execute"
confirmation as TTY UI (numbered options with ❯ cursor), not via the
AskUserQuestion tool — so the SDK's canUseTool interceptor never fired
and the assertion always saw zero questions. Real PTY observes the
rendered output directly.
Deletes test/helpers/plan-mode-helpers.ts. No production callers remained.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: rewrite 5 plan-mode E2E tests on the real-PTY harness
Replaces SDK-based assertions with runPlanSkillObservation contract. Each
test launches real claude --permission-mode plan, invokes the skill, and
asserts the outcome reaches 'asked' or 'plan_ready' within a 300s budget
(no silent Write/Edit, no crash, no timeout).
Affected:
- test/skill-e2e-plan-ceo-plan-mode.test.ts
- test/skill-e2e-plan-eng-plan-mode.test.ts
- test/skill-e2e-plan-design-plan-mode.test.ts
- test/skill-e2e-plan-devex-plan-mode.test.ts
- test/skill-e2e-plan-mode-no-op.test.ts (inPlanMode: false; tests the
preamble plan-mode-info no-op path)
test/e2e-harness-audit.test.ts — recognize runPlanSkillObservation as a
valid coverage path alongside the legacy canUseTool / runPlanModeSkillTest.
test/helpers/touchfiles.ts — point the 5 plan-mode test selections and
the e2e-harness-audit selection at test/helpers/claude-pty-runner.ts
instead of the deleted plan-mode-helpers.ts.
Proof: bun test EVALS=1 EVALS_TIER=gate on these 5 files runs sequentially
in 790s and passes 5/5. Same tests were 0/5 on origin/main, on v1.0.0.0,
and on this branch with the SDK harness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: align unit tests with slim resolvers + exempt 27MB security fixture
- test/skill-validation.test.ts: assert the slim Completeness Principle
shape (Completeness: X/10, kind-note language) instead of the old
Compression table. Remove the 3 tier-1 skills from the spot-check list
(they intentionally don't carry the full Completeness Principle
section). Exempt browse/test/fixtures/security-bench-haiku-responses.json
(27MB deterministic replay fixture for BrowseSafe-Bench) from the 2MB
tracked-file gate. The gate was actually failing on origin/main since
the fixture was added in v1.6.4.0 — this is a side-fix to a real
regression.
- test/brain-sync.test.ts: developer-machine-safe assertion for
GSTACK_HOME override (compare config contents before/after instead of
asserting the absence of a string that may legitimately exist).
- test/gen-skill-docs.test.ts: new tests for the slim — plan-review
preambles stay under the post-slim budget (~33KB), Voice + Writing
Style sections stay compact, and the slim Voice section preserves the
load-bearing semantic contract (lead-with-the-point, name-the-file,
user-outcome framing, no-corporate, no-AI-vocab, user-sovereignty).
Update path-leakage scan to allow repo-root sidecar symlinks.
- test/writing-style-resolver.test.ts: assert the compact contract
(gloss-on-first-use, outcome-framing, user-impact, terse-mode override)
instead of the old 6-numbered-rules shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v1.13.1.0)
Slim preamble work + real-PTY plan-mode E2E harness on top of v1.13.0.0.
SKILL.md corpus -25.5% (3.08 MB → 2.30 MB, ~196K tokens). 5 plan-mode
tests go from 0/5 to 5/5 (790s sequential), the first time those tests
have ever passed. Side-fixes for the 27MB security fixture warning and
the sidecar-symlink double-count.
Reverts the Fan-Out directive accidentally restored to opus-4-7.md —
v1.10.1.0's overlay-efficacy harness measured -60pp fanout vs baseline
when the nudge was active. The intentional removal stays.
TODOS:
- Pre-existing test failures from v1.12.0.0 ship: RESOLVED on main + this branch
- security-bench-haiku-responses.json size gate: RESOLVED via warn-only + exemption
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(test): harness primitives — parseNumberedOptions + budget regression utils
claude-pty-runner.ts:
- parseNumberedOptions(visible) anchors on the latest "❯ 1." cursor and
returns {index, label}[]; tests that route on option labels can find
indices without hard-coding positions
- isPermissionDialogVisible(visible) detects file-grant + workspace-trust
+ bash-permission shapes (multiple regex variants)
- isNumberedOptionListVisible: replaced \b2\. word-boundary regex with
[^0-9]2\. — stripAnsi removes TTY cursor-positioning escapes that
collapse "Option 2." to "Option2.", and \b fails on word-to-word
eval-store.ts:
- findBudgetRegressions(comparison, opts?) — pure function returning
tests where tools or turns grew >cap× vs prior run; floors at 5 prior
tools / 3 prior turns to avoid noise on tiny numbers
- assertNoBudgetRegression() — wrapper that throws with full violation
list. Env override GSTACK_BUDGET_RATIO
helpers-unit.test.ts: 23 unit tests covering empty/sparse/wrap-around
buffers for parseNumberedOptions, plus regression-floor + env-override
cases for findBudgetRegressions/assertNoBudgetRegression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: register 6 real-PTY E2E touchfiles + UI-heavy plan fixture
touchfiles.ts:
- 6 new entries in E2E_TOUCHFILES keyed to the new test files
- 6 matching E2E_TIERS classifications: 3 gate (auq-format-pty,
plan-design-with-ui-scope, budget-regression-pty), 3 periodic
(plan-ceo-mode-routing, ship-idempotency-pty, autoplan-chain-pty)
- gate ones are cheap/deterministic; periodic ones run weekly
touchfiles.test.ts:
- update the "skill-specific change selects only that skill" count
from 15 → 18 (plan-ceo-review/SKILL.md change now also selects
auq-format-pty, plan-ceo-mode-routing, autoplan-chain-pty)
test/fixtures/plans/ui-heavy-feature.md:
- planted plan with explicit UI scope keywords (pages, components,
Tailwind responsive layout, hover/loading/empty states, modal,
toast). Used by plan-design-with-ui-scope and autoplan-chain tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(test): 3 gate-tier real-PTY E2E tests
skill-e2e-auq-format-compliance.test.ts (~$0.50/run, 90-130s):
- Asserts /plan-ceo-review's first AUQ contains all 7 mandated format
elements (ELI10, Recommendation, Pros/Cons with ✅/❌, Net,
(recommended) label). Catches drift in the shared preamble resolver
that previously took weeks to notice.
- Auto-grants permission dialogs that fire during preamble side-effects
(touch on .feature-prompted markers in fresh user environments).
- Verified PASS in 126s.
skill-e2e-plan-design-with-ui.test.ts (~$0.80/run, 50-90s):
- Counterpart to the existing no-UI early-exit test. When the input plan
DOES describe UI changes, /plan-design-review must NOT early-exit and
must reach a real skill AUQ.
- Sends the slash command without args, then a follow-up message with
the UI-heavy plan description (Claude Code rejects unknown trailing
args). Asserts evidence does NOT contain "no UI scope".
- Verified PASS in 54s.
skill-budget-regression.test.ts (free, gate):
- Library-only assertion. Reads the most recent eval file, finds the
prior same-branch run via findPreviousRun, computes ComparisonResult,
asserts no test exceeded 2× tools or turns.
- Branch-scoped: skips with reason if the latest eval was produced on
a different branch (cross-branch comparison would be noise).
- First-run grace (vacuous pass) when no prior data exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(test): 3 periodic-tier real-PTY E2E tests
skill-e2e-plan-ceo-mode-routing.test.ts (~$3/run, 6-10 min/case):
- Verifies AUQ answer routing: HOLD SCOPE → rigor/bulletproof posture
language; SCOPE EXPANSION → expansion/10x/dream language. Each case
navigates 8-12 prior AUQs (telemetry, proactive, routing, vendoring,
brain, office-hours, premise, approach) before hitting Step 0F.
- Periodic, not gate: navigation phase too slow for PR-blocking.
V2 expansion to 4 modes (SELECTIVE + REDUCTION) when nav is faster.
skill-e2e-ship-idempotency.test.ts (~$3/run, 5-10 min):
- Builds a real git fixture with VERSION 0.0.2 already bumped, matching
package.json, CHANGELOG entry, pushed to a local bare remote. Runs
/ship in plan mode and asserts STATE: ALREADY_BUMPED echoes from the
Step 12 idempotency check, OR plan_ready terminates without mutation.
- Snapshots VERSION + package.json + CHANGELOG entry count + commit
count + branch HEAD before/after; fails if any changed.
skill-e2e-autoplan-chain.test.ts (~$8/run, 12-18 min):
- Asserts /autoplan phases run sequentially: tees timestamps as each
"**Phase N complete.**" marker first appears. Phase 1 (CEO) must
precede Phase 3 (Eng); Phase 2 (Design) is optional but if it
appears, must sit between 1 and 3.
- Auto-grants permission dialogs that fire during phase transitions.
All three auto-handle permission dialogs (preamble side-effects on
fresh user envs without .feature-prompted-* markers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: spell out AskUserQuestion everywhere instead of AUQ
Per user feedback: don't shorten AskUserQuestion to AUQ — the
abbreviation reads as cryptic. Apply across all the new code from this
branch:
- Rename test/skill-e2e-auq-format-compliance.test.ts →
test/skill-e2e-ask-user-question-format-compliance.test.ts
- Touchfile entry auq-format-pty → ask-user-question-format-pty
(touchfiles.ts + matching assertion in touchfiles.test.ts)
- Function rename navigateToModeAuq → navigateToModeAskUserQuestion
- Variable auqVisible → askUserQuestionVisible
- Outcome literal 'real_auq' → 'real_question'
- All comments + JSDoc + CHANGELOG entry write AskUserQuestion in full
- "AUQs" plural → "AskUserQuestions"
No behavior change. 49/49 free tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: harden v1.15.0.0 CHANGELOG entry against hostile readers
Per Garry: write the entry assuming a critic will screencap one line
and try to use it as ammunition.
Reframed the v1.15.0.0 release-summary to lead with new capability
(real-PTY harness, 11 plan-mode tests, +6 new) instead of fix-of-prior-
flaw narrative. Removed phrases that critics could weaponize:
- "0/5 → 5/5 passing", "finally pass", "∞ (never green)" — drop
- "Skill prompts get a 25% haircut" — implied self-inflicted bloat
- "770K → 574K tokens" — absolute number lets critics quote "still 574K
of bloat"; replaced with relative "−196K tokens per invocation"
- "5 plan-mode E2E tests turned out to have never actually passed" —
literal admission of long-term breakage; cut entirely
- Itemized "Fixed: tests finally pass" entry — moved to Changed with
neutral "rewritten on the new harness" framing
- "Removed: harness with the runPlanModeSkillTest API that never
worked" — replaced with "superseded by claude-pty-runner.ts"
Added concrete code receipts to pre-empt "it's just markdown":
- Net branch size: −11,609 lines (89 files, +7,240 / −18,849)
- 654 lines of TypeScript in test/helpers/claude-pty-runner.ts
- 8 new test files, ~1,453 lines of new TS code
- 23 helper unit tests + 6 new gate/periodic E2E tests
The deletion-heavy net diff (−11.6K lines) is itself the strongest
defense against the "bloat" critique — surfaced explicitly in the
numbers table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|