Commit Graph

3 Commits

Author SHA1 Message Date
Garry Tan 960c3a8d6c
v1.60.2.0 fix: free-suite drift on dev machines (eval-list cwd, gemini regex, observability floor) (#2470)
* fix(test): eval-list CLI spawns from neutral cwd so slug detection can't dodge the fixture store

getProjectEvalDir() probes cwd-relative .claude/skills/gstack/bin/gstack-slug;
with cwd=ROOT on a dev machine the self-symlink makes it succeed, routing reads
to an empty project-scoped dir instead of the seeded legacy ~/.gstack-dev/evals.
Neutral cwd + absolute script path fails both probes deterministically — same
behavior as CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): case-insensitive remediation-hint match for reworded Gemini NOT-READY message

The message now leads with 'Export GEMINI_API_KEY...' (free-tier OAuth
deprecation); the old pattern only knew lowercase 'export'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): observability check 11 floor 6 -> 5 after shell-free spawn removed promptFile unlink

aa3bd6f0 deleted the prompt temp file (and its /* non-fatal */ marker) when it
dropped shell interpolation. The invariant — every runner I/O path wrapped
non-fatally — still holds at the 5 remaining sites, now named in the comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(todos): P1 — free-suite exit code masked by in-process force-exits

Five browse test files setTimeout(() => process.exit(0), 500) inside the shared
bun process; the suite can exit 0 before the summary with real failures masked.
Receipts + fix path filed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(release): v1.60.2.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-07 15:08:12 -07:00
Garry Tan 5aae3ce117
fix: never clean up observability artifacts — partial file persists after finalize
Removing the _partial-e2e.json deletion from finalize(). These are small files
on a local disk and their persistence is the whole point of observability.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 12:37:38 -05:00
Garry Tan 029a7c2a37
feat: eval-watch dashboard + observability unit tests (15 tests, 11 codepaths)
eval-watch: live terminal dashboard reads heartbeat + partial file every 1s,
shows completed/running tests, stale detection (>10min), --tail flag for
progress.log tail. Pure renderDashboard() function for testability.

observability.test.ts: unit tests for sanitizeTestName, heartbeat schema,
progress.log format, NDJSON file naming, savePartial() with _partial flag,
finalize() cleanup, diagnostic fields, watcher rendering, stale detection,
and non-fatal I/O guarantees.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 11:04:40 -05:00