gstack/test
Garry Tan b7f6246061
test: add Codex eval for AskUserQuestion format compliance
Four-case periodic-tier eval mirrors test/skill-e2e-plan-format.test.ts
but drives the plan review skills via codex exec instead of claude -p.

Context: Codex under the gpt.md "No preamble / Prefer doing over listing"
overlay tends to skip the Simplify/ELI10 paragraph and the RECOMMENDATION
line on AskUserQuestion calls. Users have to manually re-prompt "ELI10
and don't forget to recommend" almost every time. This test pins the
behavior so regressions surface.

Cases:
- plan-ceo-review mode selection (kind-differentiated)
- plan-ceo-review approach menu (coverage-differentiated)
- plan-eng-review per-issue coverage decision
- plan-eng-review per-issue architectural choice (kind-differentiated)

Assertions on captured AskUserQuestion text:
- RECOMMENDATION: Choose present (all cases)
- Completeness: N/10 present on coverage, absent on kind
- "options differ in kind" note present on kind
- ELI10 length floor (>400 chars) — catches bare options-only output

Cost: ~\$2-4 per full run.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-22 21:34:11 -07:00
..
fixtures Merge remote-tracking branch 'origin/main' into garrytan/plan-review-regressions 2026-04-22 12:29:35 -07:00
helpers Merge remote-tracking branch 'origin/main' into garrytan/plan-review-regressions 2026-04-22 12:29:35 -07:00
analytics.test.ts
audit-compliance.test.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
benchmark-cli.test.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
benchmark-runner.test.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
builder-profile.test.ts
codex-e2e-plan-format.test.ts test: add Codex eval for AskUserQuestion format compliance 2026-04-22 21:34:11 -07:00
codex-e2e.test.ts
codex-hardening.test.ts
context-save-hardening.test.ts fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064) 2026-04-19 08:38:19 +08:00
diff-scope.test.ts
explain-level-config.test.ts
gemini-e2e.test.ts
gen-skill-docs.test.ts feat(v1.5.2.0): Opus 4.7 migration — model overlay, voice, routing (#1117) 2026-04-22 01:06:22 -07:00
global-discover.test.ts
gstack-developer-profile.test.ts
gstack-question-log.test.ts
gstack-question-preference.test.ts
hook-scripts.test.ts
host-config.test.ts
jargon-list.test.ts
learnings-injection.test.ts
learnings.test.ts
migration-checkpoint-ownership.test.ts fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064) 2026-04-19 08:38:19 +08:00
openclaw-native-skills.test.ts
plan-tune.test.ts
readme-throughput.test.ts
relink.test.ts
review-log.test.ts
setup-codesign.test.ts
ship-version-sync.test.ts
skill-collision-sentinel.test.ts fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064) 2026-04-19 08:38:19 +08:00
skill-e2e-autoplan-dual-voice.test.ts fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064) 2026-04-19 08:38:19 +08:00
skill-e2e-benchmark-providers.test.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
skill-e2e-bws.test.ts
skill-e2e-context-skills.test.ts fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064) 2026-04-19 08:38:19 +08:00
skill-e2e-cso.test.ts
skill-e2e-deploy.test.ts
skill-e2e-design.test.ts
skill-e2e-learnings.test.ts
skill-e2e-office-hours.test.ts feat: mode-posture energy fix for /plan-ceo-review and /office-hours (v1.1.2.0) (#1065) 2026-04-19 05:44:39 +08:00
skill-e2e-opus-47.test.ts feat(v1.5.2.0): Opus 4.7 migration — model overlay, voice, routing (#1117) 2026-04-22 01:06:22 -07:00
skill-e2e-plan-format.test.ts test: add AskUserQuestion format regression eval for plan reviews 2026-04-22 01:10:35 -07:00
skill-e2e-plan-tune.test.ts
skill-e2e-plan.test.ts feat: mode-posture energy fix for /plan-ceo-review and /office-hours (v1.1.2.0) (#1065) 2026-04-19 05:44:39 +08:00
skill-e2e-qa-bugs.test.ts
skill-e2e-qa-workflow.test.ts
skill-e2e-review-army.test.ts
skill-e2e-review.test.ts
skill-e2e-session-intelligence.test.ts fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064) 2026-04-19 08:38:19 +08:00
skill-e2e-sidebar.test.ts
skill-e2e-workflow.test.ts
skill-e2e.test.ts
skill-llm-eval.test.ts
skill-parser.test.ts
skill-routing-e2e.test.ts
skill-validation.test.ts feat(v1.5.2.0): Opus 4.7 migration — model overlay, voice, routing (#1117) 2026-04-22 01:06:22 -07:00
taste-engine.test.ts feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040) 2026-04-19 17:50:31 +08:00
team-mode.test.ts feat(v1.5.2.0): Opus 4.7 migration — model overlay, voice, routing (#1117) 2026-04-22 01:06:22 -07:00
telemetry.test.ts
timeline.test.ts
touchfiles.test.ts test: add AskUserQuestion format regression eval for plan reviews 2026-04-22 01:10:35 -07:00
uninstall.test.ts
upgrade-migration-v1.test.ts
v0-dormancy.test.ts
worktree.test.ts
writing-style-resolver.test.ts