RomanNum3ral
/
honcho
mirror of https://github.com/plastic-labs/honcho.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
honcho/tests/live_llm
History
Aakash Kattelu a0cc938f4a
feat: add model config option for json_object mode (#820) 
* feat: add model config option for json_object mode

* fix: catch possible validation error from structured output

* fix(llm): harden structured_output_mode json_object path

Follow-up fixes to the json_object structured-output mode for
OpenAI-compatible providers without Structured Outputs support:

- runtime: carry structured_output_mode onto the per-attempt fallback
  config (select_model_config_for_attempt dropped it, silently sending
  json_schema to a provider that can't parse it)
- backend: return a graceful empty on a contentless json_object
  response instead of raising, matching the json_schema path, and
  preserve token usage by normalizing the response
- backend: narrow the parse-failure catch to BadRequestError only, so
  transient JSONDecodeError/ValidationError propagate to retry/fallback
  instead of being swallowed to empty on the first attempt
- config: reject structured_output_mode on non-openai transports
  (silent no-op otherwise); trim docs to the deriver, the only
  structured-output feature
- backend: validate clean JSON before repair, cache the schema
  instruction, and share json_object setup between complete()/stream()

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(llm): consolidate structured-output repair, drop dead seam

Fold the OpenAI backend's three structured-output repair sites
(LengthFinishReasonError, parsed=None, json_object) into the one shared
_parse_or_repair_structured_content helper, gated by an empty_on_missing
flag: json_object returns a graceful empty on a contentless response so a
loose provider can't crash the call, while json_schema raises so the
retry/fallback chain engages.

Delete the dead execute_structured_output_call seam and its only
collaborators (attempt_structured_output_repair, StructuredOutputFailurePolicy)
— it was never called and its single-shot validate/repair/empty model
conflicts with the retry behavior in honcho_llm_call.

No behavior change. Adds tests covering the json_schema parse fallbacks
(repair, refusal passthrough, no-content raise).

---------

Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 		2026-06-23 10:42:03 -04:00
..
README.md Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
__init__.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
conftest.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
model_matrix.py feat: add model config option for json_object mode (#820) 2026-06-23 10:42:03 -04:00
test_live_anthropic.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_live_gemini.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_live_openai.py feat: add model config option for json_object mode (#820) 2026-06-23 10:42:03 -04:00

README.md

Live LLM Tests

These tests call real provider APIs and are disabled by default.

Run them with:

uv run pytest tests/live_llm -n 0 --live-llm --no-header -q

Required API key env vars:

  • LLM_ANTHROPIC_API_KEY
  • LLM_OPENAI_API_KEY
  • LLM_GEMINI_API_KEY

Model-family env vars:

  • LIVE_LLM_ANTHROPIC_45_PLUS_MODELS
  • LIVE_LLM_OPENAI_GPT4_MODELS
  • LIVE_LLM_OPENAI_GPT5_MODELS
  • LIVE_LLM_OPENAI_OPENROUTER_NON_REASONING_MODELS (OpenAI-transport → OpenRouter-served non-reasoning models)
  • LIVE_LLM_GEMINI_25_MODELS
  • LIVE_LLM_GEMINI_30_MODELS
  • LIVE_LLM_GEMINI_31_MODELS

Each model env var accepts a comma-separated list of bare model ids or provider-qualified ids.

Examples:

export LIVE_LLM_ANTHROPIC_45_PLUS_MODELS="claude-sonnet-4-5,claude-sonnet-4-6"
export LIVE_LLM_OPENAI_GPT4_MODELS="gpt-4.1"
export LIVE_LLM_OPENAI_GPT5_MODELS="gpt-5,gpt-5.4,gpt-5.4-mini"
export LIVE_LLM_OPENAI_OPENROUTER_NON_REASONING_MODELS="inception/mercury-2"
export LIVE_LLM_GEMINI_25_MODELS="gemini-2.5-flash,gemini-2.5-pro"
export LIVE_LLM_GEMINI_30_MODELS="gemini-3-flash-preview"
export LIVE_LLM_GEMINI_31_MODELS="gemini-3.1-pro-preview"

OpenRouter-routed models require additional env for the proxy endpoint:

export OPENROUTER_API_KEY="sk-or-v1-..."
# Per-feature config example:
#   DERIVER_MODEL_CONFIG__TRANSPORT=openai
#   DERIVER_MODEL_CONFIG__MODEL=inception/mercury-2
#   DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=https://openrouter.ai/api/v1
#   DERIVER_MODEL_CONFIG__OVERRIDES__API_KEY_ENV=OPENROUTER_API_KEY

Coverage by provider:

  • Anthropic: structured output path, prompt caching metrics, thinking blocks, multi-turn tool replay
  • OpenAI GPT-4 class: structured outputs, prompt caching
  • OpenAI GPT-5 class (incl. gpt-5.x point-releases): structured outputs, prompt caching, reasoning_effort, max_completion_tokens routing
  • OpenAI transport → OpenRouter non-reasoning models (e.g. inception/mercury-2): non-chat / diffusion architectures must stay on max_tokens, no reasoning_effort, tool-calling parameter-schema compatibility is the canary for exotic OR-served providers
  • Gemini 2.5/3.0 classes: structured outputs, cached-content reuse, thought signatures, multi-turn tool replay
  • Gemini 3.1 class: thinking and tool replay coverage by default; structured-output/caching coverage should only be added once Google documents support for that path