Commit Graph

4 Commits

Author SHA1 Message Date
Teknium 6b81590c55
test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions
Systematic prune per AGENTS.md test policy, one pass over every major
test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli,
cron, tui_gateway, honcho/openviking, root-level):

- DELETE: source-reading tests (read_text/getsource on prod files),
  change-detector tests (exact catalog counts, model-name snapshots,
  config version literals), mock-echo tests (assert a mock returns what
  it was told), assertion-free/trivial tests, near-duplicate
  parametrizations (boundaries + one representative kept), async/sync
  twin duplicates, cosmetic within-file variations.
- KEEP (mandatory): security/redaction/approval guards, message-role
  alternation invariants, prompt-caching/deterministic-call-id
  invariants, issue-number regression tests (deduped), E2E tests.
- 6 test files deleted outright (script-style/no-assert or fully
  redundant); conftest.py, fakes/, fixtures/ untouched.
- tests/acp/conftest.py added: autouse fixture stubs the live
  models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server
  tests performed on every session create — test_server.py 147s → 3.4s,
  and the tests are now genuinely hermetic.
- Sleep-based slowness shrunk where safe (codex_ttfb_watchdog,
  compression_concurrent_fork, etc.); no wall-clock assertion tightened.

Verification: full hermetic suite via scripts/run_tests.sh —
2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall
(baseline: 583s wall, 13,564s subprocess CPU).
2026-07-29 13:10:23 -07:00
Kshitij Kapoor 4af484d3dd feat(openai): complete gpt-5.6 E2E — codex catalog + 272K compaction auto-raise
Close the remaining end-to-end gaps so the full gpt-5.6 family (sol/
terra/luna + their -pro high-effort modes, 6 slugs) works on every
surface a user can reach them through:

- agent/auxiliary_client.py: the Codex OAuth backend hard-caps context
  at 272K for gpt-5.6 exactly as it does for 5.4/5.5, but the default
  50% compaction trigger would summarize at ~136K and waste half the
  usable window. Extend the existing _is_codex_gpt54_or_gpt55 chokepoint
  (single enforced predicate feeding _compression_threshold_for_model)
  to match gpt-5.6* on the openai-codex route so those sessions get the
  same 0.85 auto-raise. Direct-API/OpenRouter routes (full 1.05M window)
  are unaffected; the historical codex_gpt55_autoraise opt-out still
  applies. The one-time notice banner is model-dynamic and already
  renders the correct slug/cap.
- hermes_cli/config.py, agent/agent_init.py: refresh the autoraise
  comments/notice to mention the 5.6 family.
- hermes_cli/codex_models.py: add the -pro variants to DEFAULT_CODEX_MODELS
  + forward-compat so ChatGPT-OAuth (openai-codex) Pro users see the full
  family in /model, not just the base tiers.

Supersedes the earlier commit's note that 5.6 was intentionally kept out
of the codex catalog: the slugs are confirmed routable (OpenRouter live
+ codex backend), so they belong there like every other codex-capable
gpt-5.x slug.

E2E verified across all 6 slugs: direct-API ctx 1.05M, codex ctx 272K,
pricing reachable from openai + openai-api routes, codex compaction
override 0.85 (and None on direct-API + when opted out), present in
openai-api picker + codex catalog, /model gpt resolves to sol on both
native routes. Guard tests added for the compaction route matrix.
2026-07-10 00:47:51 +05:30
Kshitij Kapoor a3828a94d0 feat(openai): cover gpt-5.6 -pro variants (PR #61587 complement)
PR #61587 adds sol-pro/terra-pro/luna-pro to the aggregator lists.
Complete those on the native surfaces the same way this PR completes
the base tiers:

- hermes_cli/models.py: -pro variants in _PROVIDER_MODELS[openai-api].
- agent/usage_pricing.py: alias ("openai", "gpt-5.6-*-pro") onto the
  base-tier PricingEntry rows — the -pro high-effort modes bill at the
  SAME per-token rates (verified against OpenRouter live pricing
  2026-07-09: identical prompt/completion prices for base and -pro);
  they cost more per task by consuming more tokens, not a higher rate.
- Context lengths need no new entries: "gpt-5.6-sol" et al. are
  substrings of their -pro variants and both lookup tables match
  longest-key-first (verified: sol-pro -> 1.05M direct / 272K codex).
- model_switch sort: -pro variants parse as suffix "sol-pro" (rank 1),
  so /model gpt still defaults to base sol — pinned by test.
- Not added to DEFAULT_CODEX_MODELS: only confirmed routable via API/
  OpenRouter so far; codex live discovery will surface them if ChatGPT
  exposes them, same policy as other unconfirmed codex slugs.

Tests: invariant tests extended (pro aliases share base entries, base
sol outranks sol-pro); 191 targeted tests pass.
2026-07-10 00:47:51 +05:30
Kshitij Kapoor db117af478 review fixes: openai-api pricing route normalization, GA pricing_version, invariant tests
Phase-2 review findings addressed:
- resolve_billing_route: normalize the "openai-api" picker slug to the
  "openai" billing provider — without this the ("openai", <model>)
  _OFFICIAL_DOCS_PRICING keys (incl. every pre-existing gpt-4o/gpt-4.1
  entry, not just 5.6) were unreachable when the provider is openai-api.
- pricing_version: drop the "preview" tag (GA 2026-07-09 at same rates).
- model_metadata comment: dict order is cosmetic — lookups length-sort
  keys at match time; the old comment implied a positional invariant.
- model_switch comment: note "sol" is a series codename, not a generic
  quality word.
- tests/hermes_cli/test_gpt56_registration.py: behavior contracts (no
  list snapshots) — sol > terra/luna > 5.5 sort invariant, pricing
  reachability from both openai and openai-api routes, cache-write
  1.25x / cache-read 0.10x input relation.
2026-07-10 00:47:51 +05:30