RomanNum3ral
/
honcho
mirror of https://github.com/plastic-labs/honcho.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
honcho/tests/live_llm
History
Vansh Sharma 7b8c2917f9
fix(embedding): request float encoding_format on openai embedding calls (#938) 
* test(embedding): add reproducer for missing encoding_format on openai paths

The openai SDK defaults encoding_format to base64 when it is not passed. OpenAI-compatible providers that don't support base64 embeddings (e.g. OpenRouter with nvidia/nemotron-3-embed-1b:free) return HTTP 200 with empty data, and every embedding call fails with 'No embedding data received'.

* fix(embedding): request float encoding_format on openai embedding calls

The openai SDK defaults encoding_format to base64 when the caller does not pass one. OpenAI-compatible providers that don't support base64 embeddings (e.g. OpenRouter hosting nvidia/nemotron-3-embed-1b:free) answer HTTP 200 with empty embedding data, and every embedding call fails with 'No embedding data received', breaking conclusions, semantic search, and the deriver. Pass encoding_format='float' explicitly on both the single-query and batch call paths.

* test(embedding): cover openai-compatible providers in the live embedding matrix

The existing openai family runs against real OpenAI, which serves base64
embeddings happily, so the matrix passes with or without the #932 fix. Adds an
`openai_compatible_embedding` family (openai transport, third-party base_url)
so the matrix can reach a provider that rejects base64. Empty default_models
keeps it skipped unless LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS is set.

Also adds test_live_openai_float_encoding_matches_base64, which pins the other
direction: switching the wire format must not move vectors on real OpenAI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(embedding): keep an explicit embedding-count check on the openai paths

Passing `encoding_format` disables the openai SDK's own empty-data guard, so a
provider answering 200 with missing embeddings surfaced as `IndexError: list
index out of range` on the single path and `zip() argument 2 is shorter than
argument 1` on the batch path. The latter is also #745's signature, which would
have left it with two unrelated causes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(live-llm): correct the openai-compatible embedding matrix env docs

The documented default dimensions said 2048 after the family moved to 3072, and
LIVE_EMBEDDING_OPENAI_COMPATIBLE_SEND_DIMENSIONS was missing entirely. Also
points the example and the coverage note at a model that is actually reachable,
and records that OpenRouter load-balances, so the base64 failure is per-attempt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(embedding): resolve openai encoding_format by mode instead of pinning float

Requesting float unconditionally costs ~3.6x the response bytes of base64 and up
to +83% latency on a 500-item batch, which the default deployment on real OpenAI
pays for nothing: only third-party OpenAI-compatible providers reject base64.

Adds EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE, mirroring dimensions_mode.
`auto` keeps base64 when no base_url override is set or it points at
api.openai.com, and picks float elsewhere. The format is still always sent
explicitly, since the SDK otherwise injects base64 on its own.

Also corrects the _validate_embedding_count docstring, which said "fewer" where
the guard is an inequality.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(embedding): request base64 embeddings by omission, not by name

The openai SDK decodes a base64 response only when it injected the default
itself; naming any format makes it skip the decode and hand back the raw string,
which then fails the dimension check with "Expected 1536, got 8192". base64 mode
therefore has to omit the kwarg rather than pass it.

The unit fake returned float lists whatever was asked for, so it could not catch
this. It now mirrors the SDK and returns a base64 string for a named base64
request, which fails against the previous commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Aakash Kattelu <aakash@plasticlabs.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 		2026-08-12 13:24:38 -04:00
..
README.md fix(embedding): request float encoding_format on openai embedding calls (#938) 2026-08-12 13:24:38 -04:00
__init__.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
conftest.py fix(embedding): request float encoding_format on openai embedding calls (#938) 2026-08-12 13:24:38 -04:00
embedding_matrix.py fix(embedding): request float encoding_format on openai embedding calls (#938) 2026-08-12 13:24:38 -04:00
model_matrix.py feat: add model config option for json_object mode (#820) 2026-06-23 10:42:03 -04:00
test_live_anthropic.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_live_embeddings.py fix(embedding): request float encoding_format on openai embedding calls (#938) 2026-08-12 13:24:38 -04:00
test_live_gemini.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_live_openai.py feat(llm backend): enable combined tool calling + structured output in the LLM backend transport layer (#907) 2026-07-15 11:47:49 -04:00
test_live_structured_output_unions.py feat(dialectic): optional structured outputs with limited schema for Dialectic calls (#896) 2026-07-20 18:46:49 -04:00
test_live_timeouts.py fix(llm): support per-request provider timeouts (#832) 2026-08-04 12:43:00 -04:00
test_live_tools_structured_output.py feat(llm backend): enable combined tool calling + structured output in the LLM backend transport layer (#907) 2026-07-15 11:47:49 -04:00

README.md

Live LLM Tests

These tests call real provider APIs and are disabled by default.

Run them with:

uv run pytest tests/live_llm -n 0 --live-llm --no-header -q

Required API key env vars:

  • LLM_ANTHROPIC_API_KEY
  • LLM_OPENAI_API_KEY
  • LLM_GEMINI_API_KEY

Model-family env vars:

  • LIVE_LLM_ANTHROPIC_45_PLUS_MODELS
  • LIVE_LLM_OPENAI_GPT4_MODELS
  • LIVE_LLM_OPENAI_GPT5_MODELS
  • LIVE_LLM_OPENAI_OPENROUTER_NON_REASONING_MODELS (OpenAI-transport → OpenRouter-served non-reasoning models)
  • LIVE_LLM_GEMINI_25_MODELS
  • LIVE_LLM_GEMINI_30_MODELS
  • LIVE_LLM_GEMINI_31_MODELS

Embedding-model env vars:

  • LIVE_EMBEDDING_GEMINI_MODELS (default: gemini-embedding-001,gemini-embedding-2; add gemini-embedding-2-preview to cover the preview twin)
  • LIVE_EMBEDDING_OPENAI_MODELS (default: text-embedding-3-small)
  • LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS (no default → skipped) — OpenAI transport pointed at a third-party OpenAI-compatible provider. Also reads OPENROUTER_API_KEY, LIVE_EMBEDDING_OPENAI_COMPATIBLE_BASE_URL (default https://openrouter.ai/api/v1), LIVE_EMBEDDING_OPENAI_COMPATIBLE_DIMENSIONS (default 3072) and LIVE_EMBEDDING_OPENAI_COMPATIBLE_SEND_DIMENSIONS (default on; set to 0 for a provider that rejects OpenAI's dimensions param)
export OPENROUTER_API_KEY="sk-or-v1-..."
export LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS="google/gemini-embedding-001"

Each model env var accepts a comma-separated list of bare model ids or provider-qualified ids.

Examples:

export LIVE_LLM_ANTHROPIC_45_PLUS_MODELS="claude-sonnet-4-5,claude-sonnet-4-6"
export LIVE_LLM_OPENAI_GPT4_MODELS="gpt-4.1"
export LIVE_LLM_OPENAI_GPT5_MODELS="gpt-5,gpt-5.4,gpt-5.4-mini"
export LIVE_LLM_OPENAI_OPENROUTER_NON_REASONING_MODELS="inception/mercury-2"
export LIVE_LLM_GEMINI_25_MODELS="gemini-2.5-flash,gemini-2.5-pro"
export LIVE_LLM_GEMINI_30_MODELS="gemini-3-flash-preview"
export LIVE_LLM_GEMINI_31_MODELS="gemini-3.1-pro-preview"

OpenRouter-routed models require additional env for the proxy endpoint:

export OPENROUTER_API_KEY="sk-or-v1-..."
# Per-feature config example:
#   DERIVER_MODEL_CONFIG__TRANSPORT=openai
#   DERIVER_MODEL_CONFIG__MODEL=inception/mercury-2
#   DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=https://openrouter.ai/api/v1
#   DERIVER_MODEL_CONFIG__OVERRIDES__API_KEY_ENV=OPENROUTER_API_KEY

Coverage by provider:

  • Anthropic: structured output path, prompt caching metrics, thinking blocks, multi-turn tool replay
  • OpenAI GPT-4 class: structured outputs, prompt caching
  • OpenAI GPT-5 class (incl. gpt-5.x point-releases): structured outputs, prompt caching, reasoning_effort, max_completion_tokens routing
  • OpenAI transport → OpenRouter non-reasoning models (e.g. inception/mercury-2): non-chat / diffusion architectures must stay on max_tokens, no reasoning_effort, tool-calling parameter-schema compatibility is the canary for exotic OR-served providers
  • Gemini 2.5/3.0 classes: structured outputs, cached-content reuse, thought signatures, multi-turn tool replay
  • Gemini 3.1 class: thinking and tool replay coverage by default; structured-output/caching coverage should only be added once Google documents support for that path
  • Embeddings (test_live_embeddings.py): single embed, batched embed, batch-vs-single alignment, and chunk-to-id mapping for every configured embedding model. gemini-embedding-2* is the reason this exists — those models collapse a list of bare strings into one document (#745), and only a live call catches it
  • OpenAI-compatible embedding providers (e.g. OpenRouter's google/gemini-embedding-001): the #932 surface. Those providers reject a base64 embedding request outright (HTTP 400) or answer HTTP 200 with empty data, so the whole matrix fails without encoding_format="float". Real OpenAI accepts base64 happily, so only a third-party provider catches it. Note that OpenRouter load-balances across upstreams, so the base64 failure is per-attempt rather than guaranteed: a retry can land on an endpoint that accepts it. test_live_openai_float_encoding_matches_base64 covers the other side, that the float switch must not move vectors on real OpenAI