mirror of https://github.com/garrytan/gstack.git
8d57df27a8
Same overlays, same prompts, same metrics, `model: 'claude-sonnet-4-6'`. Tests whether the overlays behave differently on a weaker Claude model where baseline behavior is shakier. Sonnet trials cost ~3-4x less than Opus so these 5 add ~$4.50 to a full run. Measurement result from the first paired run (100 trials total, ~$14.55): - **Sonnet + effort-match shows real overlay benefit.** With the overlay on, Sonnet takes 2.5 turns on a trivial `What's the version in config.json?` prompt. Without, it takes exactly 3.0 turns in all 10 trials. ~17% reduction, below the 20% pass threshold but the signal is clean: overlay-ON distribution [2,2,2,2,2,3,3,3,3,3] vs overlay-OFF [3,3,3,3,3,3,3,3,3,3]. - All other Sonnet dimensions flat (fanout, dedicated-tools, literal interpretation). Same as Opus on those axes. - Opus effort-match remains flat (2.60 vs 2.50, +4% slower with overlay). Implication: model-stratified. The overlay stack helps Sonnet on some axes where it does nothing on Opus. Wholesale removal would hurt Sonnet. Per-nudge per-model measurement is the right move going forward. |
||
|---|---|---|
| .. | ||
| golden | ||
| mode-posture | ||
| coverage-audit-fixture.ts | ||
| eval-baselines.json | ||
| golden-ship-claude.md | ||
| overlay-nudges.ts | ||
| qa-eval-checkout-ground-truth.json | ||
| qa-eval-ground-truth.json | ||
| qa-eval-spa-ground-truth.json | ||
| review-army-migration.sql | ||
| review-army-n-plus-one.rb | ||
| review-eval-design-slop.css | ||
| review-eval-design-slop.html | ||
| review-eval-enum-diff.rb | ||
| review-eval-enum.rb | ||
| review-eval-vuln.rb | ||