mirror of https://github.com/garrytan/gstack.git
614354fc41
Adds the full spec Codex asked for: real provider adapters with auth detection, normalized RunResult, pricing tables, tool compatibility maps, parallel execution with error isolation, and table/JSON/markdown output. Judge stays on Anthropic SDK as the single stable source of quality scoring, gated behind --judge. Codex flagged the original plan as massively under-scoped — the existing runner is Claude-only and the judge is Anthropic-only. You can't benchmark GPT or Gemini without real provider infrastructure. This commit ships it. New architecture: test/helpers/providers/types.ts ProviderAdapter interface test/helpers/providers/claude.ts wraps `claude -p --output-format json` test/helpers/providers/gpt.ts wraps `codex exec --json` test/helpers/providers/gemini.ts wraps `gemini -p --output-format stream-json --yolo` test/helpers/pricing.ts per-model USD cost tables (quarterly) test/helpers/tool-map.ts which tools each CLI exposes test/helpers/benchmark-runner.ts orchestrator (Promise.allSettled) test/helpers/benchmark-judge.ts Anthropic SDK quality scorer bin/gstack-model-benchmark CLI entry test/benchmark-runner.test.ts 9 unit tests (cost math, formatters, tool-map) Per-provider error isolation: - auth → record reason, don't abort batch - timeout → record reason, don't abort batch - rate_limit → record reason, don't abort batch - binary_missing → record in available() check, skip if --skip-unavailable Pricing correction: cached input tokens are disjoint from uncached input tokens (Anthropic/OpenAI report them separately). Original math subtracted them, producing negative costs. Now adds cached at the 10% discount alongside the full uncached input cost. CLI: gstack-model-benchmark --prompt "..." --models claude,gpt,gemini gstack-model-benchmark ./prompt.txt --output json --judge gstack-model-benchmark ./prompt.txt --models claude --timeout-ms 60000 Output formats: table (default), json, markdown. Each shows model, latency, in→out tokens, cost, quality (when --judge used), tool calls, and any errors. Known limitations for v1: - Claude adapter approximates toolCalls as num_turns (stream-json would give exact counts; v2 can upgrade). - Live E2E tests (test/providers.e2e.test.ts) not included — they require CI secrets for all three providers. Unit tests cover the shape and math. - Provider CLIs sometimes return non-JSON error text to stdout; the parsers fall back to treating raw output as plain text in that case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| fixtures | ||
| helpers | ||
| analytics.test.ts | ||
| audit-compliance.test.ts | ||
| benchmark-runner.test.ts | ||
| builder-profile.test.ts | ||
| codex-e2e.test.ts | ||
| diff-scope.test.ts | ||
| gemini-e2e.test.ts | ||
| gen-skill-docs.test.ts | ||
| global-discover.test.ts | ||
| hook-scripts.test.ts | ||
| host-config.test.ts | ||
| learnings-injection.test.ts | ||
| learnings.test.ts | ||
| relink.test.ts | ||
| review-log.test.ts | ||
| skill-e2e-bws.test.ts | ||
| skill-e2e-cso.test.ts | ||
| skill-e2e-deploy.test.ts | ||
| skill-e2e-design.test.ts | ||
| skill-e2e-learnings.test.ts | ||
| skill-e2e-plan.test.ts | ||
| skill-e2e-qa-bugs.test.ts | ||
| skill-e2e-qa-workflow.test.ts | ||
| skill-e2e-review-army.test.ts | ||
| skill-e2e-review.test.ts | ||
| skill-e2e-session-intelligence.test.ts | ||
| skill-e2e-sidebar.test.ts | ||
| skill-e2e-workflow.test.ts | ||
| skill-e2e.test.ts | ||
| skill-llm-eval.test.ts | ||
| skill-parser.test.ts | ||
| skill-routing-e2e.test.ts | ||
| skill-validation.test.ts | ||
| team-mode.test.ts | ||
| telemetry.test.ts | ||
| timeline.test.ts | ||
| touchfiles.test.ts | ||
| uninstall.test.ts | ||
| worktree.test.ts | ||