honcho/tests/llm
Niyaz Almufti a92fb1e078
Fix Gemini batch embedding for gemini-embedding-2* models (#745)
* Fix Gemini batch embedding for gemini-embedding-2* models

The google-genai SDK treats embed_content(contents=[list_of_strings]) as a
single multi-part document for gemini-embedding-2* models, silently returning
exactly 1 embedding regardless of input count. This caused a zip(...,
strict=True) ValueError in _process_batch.

Wrap each text in genai_types.Content(parts=[genai_types.Part(text=...)])
so the SDK treats each string as a separate content item. This matches the
workaround used by pydantic-ai (#4873) and graphiti (#1474).

Upstream SDK issue: googleapis/python-genai#2523
Fixes plastic-labs/honcho#744

* test(embedding): live embedding coverage for every Gemini and OpenAI model

Adds tests/live_llm/test_live_embeddings.py plus an env-driven embedding
matrix alongside the existing LLM one. Covers single embed, batched embed,
batch-vs-single alignment, chunk-to-id mapping, and the batch-split path.

Only a live call catches the gemini-embedding-2* collapse: the SDK folds a
list of bare strings into one document and returns a single embedding.
Reverting the Content wrapping fails all four batch tests for
gemini-embedding-2-preview and gemini-embedding-2 with the reported
`zip() argument 2 is shorter than argument 1`, while gemini-embedding-001
and text-embedding-3-small stay green.

Also makes the concatenation in the conclusions semantic-search validation
message explicit, so the repo-wide basedpyright pre-push hook passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(embedding): drop the preview twin from the default embedding matrix

gemini-embedding-2 is the GA release of gemini-embedding-2-preview and
behaves identically, so running both by default doubles the Gemini cost for
no extra coverage. The preview stays reachable through
LIVE_EMBEDDING_GEMINI_MODELS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Aakash Kattelu <aakash@plasticlabs.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 17:50:40 -04:00
..
test_backends fix(llm): support per-request provider timeouts (#832) 2026-08-04 12:43:00 -04:00
conftest.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_agent_tool_schemas.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_capture.py feat(telemetry): CloudEvents + Langfuse tracing as projections over a captured LLM stream (#845) 2026-07-02 16:49:53 -04:00
test_conversation.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_credentials.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_embedding_client.py Fix Gemini batch embedding for gemini-embedding-2* models (#745) 2026-08-11 17:50:40 -04:00
test_history_adapters.py Refactor clients.py to add modern features and more flexible configuration (#459) 2026-04-20 02:46:37 -04:00
test_langfuse_trace_annotation.py feat(telemetry): CloudEvents + Langfuse tracing as projections over a captured LLM stream (#845) 2026-07-02 16:49:53 -04:00
test_model_config.py feat: add model config option for json_object mode (#820) 2026-06-23 10:42:03 -04:00
test_registry.py fix(ci): clear basedpyright warnings from #903 tests; run static analysis on PRs (#975) 2026-08-04 17:03:10 -04:00
test_request_builder.py fix(llm): support per-request provider timeouts (#832) 2026-08-04 12:43:00 -04:00
test_telemetry_agent_iteration.py feat(telemetry): CloudEvents + Langfuse tracing as projections over a captured LLM stream (#845) 2026-07-02 16:49:53 -04:00
test_telemetry_agent_tool_call.py feat: add new cloudevents for api routes (#637) 2026-05-20 18:25:30 -04:00
test_telemetry_llm_call.py Track user and session ID on Langfuse traces (#814) 2026-06-23 00:03:21 -04:00
test_tool_loop_truncation.py feat: add new cloudevents for api routes (#637) 2026-05-20 18:25:30 -04:00