fix(embedding): request float encoding_format on openai embedding calls (#938)
* test(embedding): add reproducer for missing encoding_format on openai paths The openai SDK defaults encoding_format to base64 when it is not passed. OpenAI-compatible providers that don't support base64 embeddings (e.g. OpenRouter with nvidia/nemotron-3-embed-1b:free) return HTTP 200 with empty data, and every embedding call fails with 'No embedding data received'. * fix(embedding): request float encoding_format on openai embedding calls The openai SDK defaults encoding_format to base64 when the caller does not pass one. OpenAI-compatible providers that don't support base64 embeddings (e.g. OpenRouter hosting nvidia/nemotron-3-embed-1b:free) answer HTTP 200 with empty embedding data, and every embedding call fails with 'No embedding data received', breaking conclusions, semantic search, and the deriver. Pass encoding_format='float' explicitly on both the single-query and batch call paths. * test(embedding): cover openai-compatible providers in the live embedding matrix The existing openai family runs against real OpenAI, which serves base64 embeddings happily, so the matrix passes with or without the #932 fix. Adds an `openai_compatible_embedding` family (openai transport, third-party base_url) so the matrix can reach a provider that rejects base64. Empty default_models keeps it skipped unless LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS is set. Also adds test_live_openai_float_encoding_matches_base64, which pins the other direction: switching the wire format must not move vectors on real OpenAI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(embedding): keep an explicit embedding-count check on the openai paths Passing `encoding_format` disables the openai SDK's own empty-data guard, so a provider answering 200 with missing embeddings surfaced as `IndexError: list index out of range` on the single path and `zip() argument 2 is shorter than argument 1` on the batch path. The latter is also #745's signature, which would have left it with two unrelated causes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(live-llm): correct the openai-compatible embedding matrix env docs The documented default dimensions said 2048 after the family moved to 3072, and LIVE_EMBEDDING_OPENAI_COMPATIBLE_SEND_DIMENSIONS was missing entirely. Also points the example and the coverage note at a model that is actually reachable, and records that OpenRouter load-balances, so the base64 failure is per-attempt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(embedding): resolve openai encoding_format by mode instead of pinning float Requesting float unconditionally costs ~3.6x the response bytes of base64 and up to +83% latency on a 500-item batch, which the default deployment on real OpenAI pays for nothing: only third-party OpenAI-compatible providers reject base64. Adds EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE, mirroring dimensions_mode. `auto` keeps base64 when no base_url override is set or it points at api.openai.com, and picks float elsewhere. The format is still always sent explicitly, since the SDK otherwise injects base64 on its own. Also corrects the _validate_embedding_count docstring, which said "fewer" where the guard is an inequality. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(embedding): request base64 embeddings by omission, not by name The openai SDK decodes a base64 response only when it injected the default itself; naming any format makes it skip the decode and hand back the raw string, which then fails the dimension check with "Expected 1536, got 8192". base64 mode therefore has to omit the kwarg rather than pass it. The unit fake returned float lists whatever was asked for, so it could not catch this. It now mirrors the SDK and returns a base64 string for a named base64 request, which fails against the previous commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Aakash Kattelu <aakash@plasticlabs.ai> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Vansh Sharma
GitHub
parent
e6f436e69d
commit
7b8c2917f9
|
|
@ -285,6 +285,14 @@ Forwarding `dimensions=` to OpenAI-compatible providers is controlled by `EMBEDD
|
|||
- `always`: always forward, regardless of whether `EMBEDDING_VECTOR_DIMENSIONS` was set. Use for OpenAI-compatible self-hosted providers that require it. Do not pick `always` *just* for same-as-default truncation — `auto` handles that case correctly as long as you set `EMBEDDING_VECTOR_DIMENSIONS=1536` explicitly in your environment. `always` is the right answer when your config layer might strip explicit "default-valued" envs, or when you want defense-in-depth.
|
||||
- `never`: never forward. Explicit opt-out for providers that reject the parameter (e.g. `text-embedding-ada-002` if it slips past the known-rejecting allowlist).
|
||||
|
||||
The embedding wire format is controlled by `EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE`. The `openai` SDK sends `encoding_format=base64` when the caller passes nothing, and some OpenAI-compatible providers answer that with an error or with empty data, so Honcho always sends the format explicitly:
|
||||
|
||||
- `auto` (default): `base64` when no `EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL` is set or it points at `api.openai.com`, `float` otherwise. base64 is roughly 3.6x smaller on the wire than JSON floats, so this keeps the compact format for real OpenAI and only pays the larger payload where compatibility requires it.
|
||||
- `float`: always request floats. Use for a provider that rejects base64 but sits behind a host `auto` reads as OpenAI-compatible-but-capable.
|
||||
- `base64`: always request base64. Use for a proxy that fronts real OpenAI (Azure OpenAI, LiteLLM) where `auto` cannot tell from the host that base64 is safe, and you want the smaller payload.
|
||||
|
||||
Both formats decode to identical vectors, so switching modes does not require re-embedding.
|
||||
|
||||
#### Bootstrapping non-default dimensions
|
||||
|
||||
`EMBEDDING_VECTOR_DIMENSIONS` is treated as immutable for the life of a deployment. The pgvector schema is dim-pinned by Alembic at `1536` by default; if you want a different dim, you must ALTER the empty columns once at bootstrap time.
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import math
|
|||
import os
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Any, ClassVar, Literal, cast
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import tomllib
|
||||
from dotenv import load_dotenv
|
||||
|
|
@ -26,12 +27,17 @@ logger = logging.getLogger(__name__)
|
|||
ModelTransport = Literal["anthropic", "openai", "gemini"]
|
||||
EmbeddingTransport = Literal["openai", "gemini"]
|
||||
EmbeddingDimensionsMode = Literal["auto", "always", "never"]
|
||||
EmbeddingEncodingFormat = Literal["float", "base64"]
|
||||
EmbeddingEncodingFormatMode = Literal["auto", "float", "base64"]
|
||||
|
||||
# OpenAI-compatible models that reject the `dimensions=` request parameter.
|
||||
_EMBEDDING_KNOWN_REJECTING_MODELS: frozenset[str] = frozenset(
|
||||
{"text-embedding-ada-002"}
|
||||
)
|
||||
|
||||
# Hosts known to serve base64 embeddings, which are ~3.6x smaller on the wire.
|
||||
_EMBEDDING_BASE64_CAPABLE_HOSTS: frozenset[str] = frozenset({"api.openai.com"})
|
||||
|
||||
|
||||
def _default_embedding_model_for_transport(transport: EmbeddingTransport) -> str:
|
||||
if transport == "gemini":
|
||||
|
|
@ -386,6 +392,7 @@ class ConfiguredEmbeddingModelSettings(BaseModel):
|
|||
transport: EmbeddingTransport = "openai"
|
||||
overrides: ModelOverrideSettings = Field(default_factory=ModelOverrideSettings)
|
||||
dimensions_mode: EmbeddingDimensionsMode = "auto"
|
||||
encoding_format_mode: EmbeddingEncodingFormatMode = "auto"
|
||||
max_batch_size: Annotated[int, Field(gt=0)] | None = None
|
||||
|
||||
@model_validator(mode="before")
|
||||
|
|
@ -830,6 +837,22 @@ class EmbeddingSettings(HonchoSettings):
|
|||
return False
|
||||
return "VECTOR_DIMENSIONS" in self.model_fields_set
|
||||
|
||||
def resolve_encoding_format(self) -> EmbeddingEncodingFormat:
|
||||
"""Pick the ``encoding_format`` for OpenAI embedding calls.
|
||||
|
||||
``auto`` keeps the compact base64 wire format on hosts known to support
|
||||
it and falls back to float elsewhere, since OpenAI-compatible providers
|
||||
may answer a base64 request with an error or empty data.
|
||||
"""
|
||||
mode = self.MODEL_CONFIG.encoding_format_mode
|
||||
if mode != "auto":
|
||||
return mode
|
||||
base_url = self.MODEL_CONFIG.overrides.base_url
|
||||
if not base_url:
|
||||
return "base64"
|
||||
host = urlparse(base_url).hostname
|
||||
return "base64" if host in _EMBEDDING_BASE64_CAPABLE_HOSTS else "float"
|
||||
|
||||
|
||||
class DeriverSettings(HonchoSettings):
|
||||
model_config = SettingsConfigDict( # pyright: ignore
|
||||
|
|
|
|||
|
|
@ -12,7 +12,12 @@ from google.genai import types as genai_types
|
|||
from nanoid import generate as generate_nanoid
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from .config import EmbeddingModelConfig, resolve_embedding_model_config, settings
|
||||
from .config import (
|
||||
EmbeddingEncodingFormat,
|
||||
EmbeddingModelConfig,
|
||||
resolve_embedding_model_config,
|
||||
settings,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -173,11 +178,13 @@ class _EmbeddingClient:
|
|||
max_input_tokens: int,
|
||||
max_tokens_per_request: int,
|
||||
send_dimensions: bool,
|
||||
encoding_format: EmbeddingEncodingFormat = "float",
|
||||
):
|
||||
self.transport: str = config.transport
|
||||
self.model: str = config.model
|
||||
self.vector_dimensions: int = vector_dimensions
|
||||
self.send_dimensions: bool = send_dimensions
|
||||
self.encoding_format: EmbeddingEncodingFormat = encoding_format
|
||||
|
||||
if self.transport == "gemini":
|
||||
if not config.api_key:
|
||||
|
|
@ -226,6 +233,29 @@ class _EmbeddingClient:
|
|||
)
|
||||
return embedding
|
||||
|
||||
def _apply_encoding_format(self, openai_kwargs: dict[str, Any]) -> None:
|
||||
"""Set the embedding wire format on an openai request.
|
||||
|
||||
Base64 is requested by omission, not by name: the SDK injects
|
||||
`encoding_format=base64` when the caller passes nothing and decodes the
|
||||
response, but skips that decode for any format the caller names, handing
|
||||
back the raw base64 string.
|
||||
"""
|
||||
if self.encoding_format != "base64":
|
||||
openai_kwargs["encoding_format"] = self.encoding_format
|
||||
|
||||
def _validate_embedding_count(self, expected: int, received: int) -> None:
|
||||
"""Guard against a 200 response whose embedding count differs from inputs.
|
||||
|
||||
An explicit `encoding_format` disables the openai SDK's own empty-data
|
||||
check, so this has to live here.
|
||||
"""
|
||||
if received != expected:
|
||||
raise ValueError(
|
||||
f"Embedding count mismatch for {self.transport}:{self.model}. "
|
||||
+ f"Expected {expected}, got {received}."
|
||||
)
|
||||
|
||||
async def embed(self, query: str) -> list[float]:
|
||||
token_count = len(self.encoding.encode(query))
|
||||
|
||||
|
|
@ -264,9 +294,11 @@ class _EmbeddingClient:
|
|||
|
||||
async def _call_openai() -> list[float]:
|
||||
openai_kwargs: dict[str, Any] = {"model": self.model, "input": [query]}
|
||||
self._apply_encoding_format(openai_kwargs)
|
||||
if self.send_dimensions:
|
||||
openai_kwargs["dimensions"] = self.vector_dimensions
|
||||
response = await openai_client.embeddings.create(**openai_kwargs)
|
||||
self._validate_embedding_count(1, len(response.data))
|
||||
return self._validate_embedding_dimensions(response.data[0].embedding)
|
||||
|
||||
return await _emit_embedding_call(
|
||||
|
|
@ -473,9 +505,11 @@ class _EmbeddingClient:
|
|||
"model": self.model,
|
||||
"input": [item.text for item in batch],
|
||||
}
|
||||
self._apply_encoding_format(openai_kwargs)
|
||||
if self.send_dimensions:
|
||||
openai_kwargs["dimensions"] = self.vector_dimensions
|
||||
response = await self.client.embeddings.create(**openai_kwargs)
|
||||
self._validate_embedding_count(len(batch), len(response.data))
|
||||
for item, embedding_data in zip(batch, response.data, strict=True):
|
||||
result[item.text_id][item.chunk_index] = (
|
||||
self._validate_embedding_dimensions(embedding_data.embedding)
|
||||
|
|
@ -612,6 +646,7 @@ class EmbeddingClient:
|
|||
max_input_tokens=settings.EMBEDDING.MAX_INPUT_TOKENS,
|
||||
max_tokens_per_request=settings.EMBEDDING.MAX_TOKENS_PER_REQUEST,
|
||||
send_dimensions=settings.EMBEDDING.resolve_send_dimensions(),
|
||||
encoding_format=settings.EMBEDDING.resolve_encoding_format(),
|
||||
)
|
||||
self._instance_signature = signature
|
||||
logger.debug(
|
||||
|
|
@ -637,6 +672,7 @@ class EmbeddingClient:
|
|||
settings.EMBEDDING.MAX_INPUT_TOKENS,
|
||||
settings.EMBEDDING.MAX_TOKENS_PER_REQUEST,
|
||||
settings.EMBEDDING.resolve_send_dimensions(),
|
||||
settings.EMBEDDING.resolve_encoding_format(),
|
||||
)
|
||||
|
||||
async def embed(self, query: str) -> list[float]:
|
||||
|
|
|
|||
|
|
@ -28,6 +28,12 @@ Embedding-model env vars:
|
|||
|
||||
- `LIVE_EMBEDDING_GEMINI_MODELS` (default: `gemini-embedding-001,gemini-embedding-2`; add `gemini-embedding-2-preview` to cover the preview twin)
|
||||
- `LIVE_EMBEDDING_OPENAI_MODELS` (default: `text-embedding-3-small`)
|
||||
- `LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS` (no default → skipped) — OpenAI transport pointed at a third-party OpenAI-compatible provider. Also reads `OPENROUTER_API_KEY`, `LIVE_EMBEDDING_OPENAI_COMPATIBLE_BASE_URL` (default `https://openrouter.ai/api/v1`), `LIVE_EMBEDDING_OPENAI_COMPATIBLE_DIMENSIONS` (default `3072`) and `LIVE_EMBEDDING_OPENAI_COMPATIBLE_SEND_DIMENSIONS` (default on; set to `0` for a provider that rejects OpenAI's `dimensions` param)
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY="sk-or-v1-..."
|
||||
export LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS="google/gemini-embedding-001"
|
||||
```
|
||||
|
||||
Each model env var accepts a comma-separated list of bare model ids or provider-qualified ids.
|
||||
|
||||
|
|
@ -63,3 +69,4 @@ Coverage by provider:
|
|||
- Gemini 2.5/3.0 classes: structured outputs, cached-content reuse, thought signatures, multi-turn tool replay
|
||||
- Gemini 3.1 class: thinking and tool replay coverage by default; structured-output/caching coverage should only be added once Google documents support for that path
|
||||
- Embeddings (`test_live_embeddings.py`): single embed, batched embed, batch-vs-single alignment, and chunk-to-id mapping for every configured embedding model. `gemini-embedding-2*` is the reason this exists — those models collapse a list of bare strings into one document (#745), and only a live call catches it
|
||||
- OpenAI-compatible embedding providers (e.g. OpenRouter's `google/gemini-embedding-001`): the #932 surface. Those providers reject a base64 embedding request outright (HTTP 400) or answer HTTP 200 with empty data, so the whole matrix fails without `encoding_format="float"`. Real OpenAI accepts base64 happily, so only a third-party provider catches it. Note that OpenRouter load-balances across upstreams, so the base64 failure is per-attempt rather than guaranteed: a retry can land on an endpoint that accepts it. `test_live_openai_float_encoding_matches_base64` covers the other side, that the float switch must not move vectors on real OpenAI
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
|
|
@ -51,6 +52,11 @@ def require_provider_key(model_spec: LiveModelSpec) -> None:
|
|||
|
||||
|
||||
def require_embedding_key(spec: LiveEmbeddingSpec) -> str:
|
||||
if spec.api_key_env:
|
||||
key = os.getenv(spec.api_key_env)
|
||||
if not key:
|
||||
pytest.skip(f"Missing {spec.api_key_env} for live embedding {spec.id}")
|
||||
return key
|
||||
key = {
|
||||
"openai": settings.LLM.OPENAI_API_KEY,
|
||||
"gemini": settings.LLM.GEMINI_API_KEY,
|
||||
|
|
@ -72,8 +78,10 @@ def make_embedding_client(
|
|||
"vector_dimensions": spec.dimensions,
|
||||
"max_input_tokens": 2048,
|
||||
"max_tokens_per_request": 300_000,
|
||||
# Both providers accept an explicit dimension request for these models.
|
||||
"send_dimensions": True,
|
||||
"send_dimensions": spec.send_dimensions,
|
||||
# Pinned rather than resolved from settings: the matrix exists to exercise
|
||||
# the float path that `auto` only picks for third-party providers.
|
||||
"encoding_format": "float",
|
||||
}
|
||||
kwargs.update(overrides)
|
||||
return _EmbeddingClient(
|
||||
|
|
@ -81,6 +89,7 @@ def make_embedding_client(
|
|||
transport=spec.transport,
|
||||
model=spec.model,
|
||||
api_key=require_embedding_key(spec),
|
||||
base_url=spec.base_url,
|
||||
),
|
||||
**kwargs,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -14,6 +14,13 @@ class LiveEmbeddingFamily:
|
|||
dimensions: int
|
||||
default_models: tuple[str, ...] = ()
|
||||
docs_url: str | None = None
|
||||
base_url: str | None = None
|
||||
# Falls back to the transport's own key when unset.
|
||||
api_key_env: str | None = None
|
||||
dimensions_env: str | None = None
|
||||
base_url_env: str | None = None
|
||||
send_dimensions: bool = True
|
||||
send_dimensions_env: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -24,6 +31,9 @@ class LiveEmbeddingSpec:
|
|||
env_var: str
|
||||
dimensions: int
|
||||
docs_url: str | None = None
|
||||
base_url: str | None = None
|
||||
api_key_env: str | None = None
|
||||
send_dimensions: bool = True
|
||||
|
||||
@property
|
||||
def id(self) -> str:
|
||||
|
|
@ -55,6 +65,25 @@ EMBEDDING_FAMILIES: tuple[LiveEmbeddingFamily, ...] = (
|
|||
default_models=("text-embedding-3-small",),
|
||||
docs_url="https://platform.openai.com/docs/guides/embeddings",
|
||||
),
|
||||
# OpenAI transport pointed at an OpenAI-compatible provider. This is the
|
||||
# regression surface for #932: the openai SDK asks for base64 embeddings
|
||||
# unless told otherwise, and third-party providers reject or empty out that
|
||||
# request. Empty default_models → skipped unless set.
|
||||
LiveEmbeddingFamily(
|
||||
transport="openai",
|
||||
family="openai_compatible_embedding",
|
||||
env_var="LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS",
|
||||
dimensions=3072,
|
||||
dimensions_env="LIVE_EMBEDDING_OPENAI_COMPATIBLE_DIMENSIONS",
|
||||
base_url="https://openrouter.ai/api/v1",
|
||||
base_url_env="LIVE_EMBEDDING_OPENAI_COMPATIBLE_BASE_URL",
|
||||
api_key_env="OPENROUTER_API_KEY",
|
||||
# Mirrors honcho's own behaviour once VECTOR_DIMENSIONS is set; turn off
|
||||
# for a provider that rejects the param.
|
||||
send_dimensions=True,
|
||||
send_dimensions_env="LIVE_EMBEDDING_OPENAI_COMPATIBLE_SEND_DIMENSIONS",
|
||||
docs_url="https://openrouter.ai/docs/api-reference/embeddings",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -72,6 +101,17 @@ def get_live_embedding_specs(
|
|||
if transport is not None and family.transport != transport:
|
||||
continue
|
||||
models = _parse_env_models(os.getenv(family.env_var)) or family.default_models
|
||||
dimensions = family.dimensions
|
||||
if family.dimensions_env:
|
||||
dimensions = int(os.getenv(family.dimensions_env) or family.dimensions)
|
||||
base_url = family.base_url
|
||||
if family.base_url_env:
|
||||
base_url = os.getenv(family.base_url_env) or family.base_url
|
||||
send_dimensions = family.send_dimensions
|
||||
if family.send_dimensions_env:
|
||||
raw = os.getenv(family.send_dimensions_env)
|
||||
if raw is not None:
|
||||
send_dimensions = raw.strip().lower() in {"1", "true", "yes"}
|
||||
for model in models:
|
||||
specs.append(
|
||||
LiveEmbeddingSpec(
|
||||
|
|
@ -79,8 +119,11 @@ def get_live_embedding_specs(
|
|||
family=family.family,
|
||||
model=model,
|
||||
env_var=family.env_var,
|
||||
dimensions=family.dimensions,
|
||||
dimensions=dimensions,
|
||||
docs_url=family.docs_url,
|
||||
base_url=base_url,
|
||||
api_key_env=family.api_key_env,
|
||||
send_dimensions=send_dimensions,
|
||||
)
|
||||
)
|
||||
return tuple(specs)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from .conftest import cosine_similarity, make_embedding_client
|
||||
from .embedding_matrix import LiveEmbeddingSpec, get_live_embedding_specs
|
||||
|
|
@ -18,6 +21,9 @@ BATCH_TEXTS: list[str] = [
|
|||
|
||||
ALL_SPECS = get_live_embedding_specs()
|
||||
GEMINI_SPECS = get_live_embedding_specs(transport="gemini")
|
||||
OPENAI_NATIVE_SPECS = tuple(
|
||||
spec for spec in ALL_SPECS if spec.family == "openai_embedding"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -114,6 +120,32 @@ async def test_live_batch_embed_maps_chunks_to_their_ids(
|
|||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("spec", OPENAI_NATIVE_SPECS, ids=lambda spec: spec.id)
|
||||
async def test_live_openai_float_encoding_matches_base64(
|
||||
spec: LiveEmbeddingSpec,
|
||||
) -> None:
|
||||
"""Guard for #938, which switched the openai paths to `encoding_format="float"`.
|
||||
|
||||
Requesting floats must return the same vectors the SDK's base64 default
|
||||
decoded to, so existing stored embeddings stay comparable.
|
||||
"""
|
||||
client = make_embedding_client(spec)
|
||||
openai_client = cast(AsyncOpenAI, client.client)
|
||||
base64_kwargs: dict[str, Any] = {"model": spec.model, "input": [BATCH_TEXTS[0]]}
|
||||
if spec.send_dimensions:
|
||||
base64_kwargs["dimensions"] = spec.dimensions
|
||||
|
||||
float_vector = await client.embed(BATCH_TEXTS[0])
|
||||
# No encoding_format → SDK sends base64 and decodes it, the pre-#938 path.
|
||||
base64_response = await openai_client.embeddings.create(**base64_kwargs)
|
||||
|
||||
similarity = cosine_similarity(float_vector, base64_response.data[0].embedding)
|
||||
assert (
|
||||
similarity > 0.99999
|
||||
), f"{spec.id}: float encoding diverges from base64 (cosine={similarity:.8f})"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("spec", GEMINI_SPECS, ids=lambda spec: spec.id)
|
||||
async def test_live_gemini_batch_embed_survives_batch_split(
|
||||
|
|
|
|||
|
|
@ -1,10 +1,16 @@
|
|||
import array
|
||||
import base64
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
from google.genai import types as genai_types
|
||||
|
||||
from src.config import EmbeddingModelConfig, resolve_embedding_model_config
|
||||
from src.config import (
|
||||
EmbeddingEncodingFormat,
|
||||
EmbeddingModelConfig,
|
||||
resolve_embedding_model_config,
|
||||
)
|
||||
from src.embedding_client import (
|
||||
BatchItem,
|
||||
_EmbeddingClient, # pyright: ignore[reportPrivateUsage]
|
||||
|
|
@ -20,6 +26,9 @@ class FakeOpenAIEmbeddingsAPI:
|
|||
def __init__(self, embedding: list[float]) -> None:
|
||||
self.embedding: list[float] = embedding
|
||||
self.calls: list[dict[str, Any]] = []
|
||||
# Simulate a provider answering 200 with missing embeddings.
|
||||
self.returns_no_data: bool = False
|
||||
self.truncate_data_to: int | None = None
|
||||
|
||||
async def create(
|
||||
self,
|
||||
|
|
@ -31,10 +40,21 @@ class FakeOpenAIEmbeddingsAPI:
|
|||
call: dict[str, Any] = {"model": model, "input": input}
|
||||
call.update(kwargs)
|
||||
self.calls.append(call)
|
||||
# Mirror the SDK: a named encoding_format skips its base64 decode, so the
|
||||
# response carries the raw string instead of floats.
|
||||
payload: Any = self.embedding
|
||||
if kwargs.get("encoding_format") == "base64":
|
||||
payload = base64.b64encode(
|
||||
array.array("f", self.embedding).tobytes()
|
||||
).decode()
|
||||
if isinstance(input, list):
|
||||
data = [SimpleNamespace(embedding=self.embedding) for _ in input]
|
||||
data = [SimpleNamespace(embedding=payload) for _ in input]
|
||||
else:
|
||||
data = [SimpleNamespace(embedding=self.embedding)]
|
||||
data = [SimpleNamespace(embedding=payload)]
|
||||
if self.returns_no_data:
|
||||
data = []
|
||||
elif self.truncate_data_to is not None:
|
||||
data = data[: self.truncate_data_to]
|
||||
return SimpleNamespace(data=data)
|
||||
|
||||
|
||||
|
|
@ -69,7 +89,11 @@ async def test_openai_embedding_client_uses_configured_model_and_dimensions(
|
|||
|
||||
assert embedding == [0.1] * 8
|
||||
assert fake_embeddings.calls == [
|
||||
{"model": "text-embedding-3-small", "input": ["hello world"]}
|
||||
{
|
||||
"model": "text-embedding-3-small",
|
||||
"input": ["hello world"],
|
||||
"encoding_format": "float",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -204,6 +228,7 @@ def _build_openai_client(
|
|||
send_dimensions: bool,
|
||||
vector_dimensions: int,
|
||||
max_batch_size: int | None = None,
|
||||
encoding_format: EmbeddingEncodingFormat = "float",
|
||||
) -> tuple[_EmbeddingClient, FakeOpenAIEmbeddingsAPI]:
|
||||
fake_embeddings = FakeOpenAIEmbeddingsAPI(embedding)
|
||||
|
||||
|
|
@ -226,6 +251,7 @@ def _build_openai_client(
|
|||
max_input_tokens=8192,
|
||||
max_tokens_per_request=300_000,
|
||||
send_dimensions=send_dimensions,
|
||||
encoding_format=encoding_format,
|
||||
)
|
||||
return client, fake_embeddings
|
||||
|
||||
|
|
@ -248,6 +274,7 @@ async def test_openai_embed_forwards_dimensions_when_send_dimensions_true(
|
|||
{
|
||||
"model": "text-embedding-3-small",
|
||||
"input": ["hello"],
|
||||
"encoding_format": "float",
|
||||
"dimensions": 768,
|
||||
}
|
||||
]
|
||||
|
|
@ -267,7 +294,13 @@ async def test_openai_embed_omits_dimensions_when_send_dimensions_false(
|
|||
|
||||
await client.embed("hello")
|
||||
|
||||
assert fake.calls == [{"model": "text-embedding-3-small", "input": ["hello"]}]
|
||||
assert fake.calls == [
|
||||
{
|
||||
"model": "text-embedding-3-small",
|
||||
"input": ["hello"],
|
||||
"encoding_format": "float",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -437,6 +470,86 @@ async def test_openai_batch_embed_forwards_dimensions(
|
|||
assert fake.calls[0]["dimensions"] == 768
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_embed_requests_float_encoding_format(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The single-query path must request float embeddings explicitly.
|
||||
|
||||
Without an explicit encoding_format, the openai SDK defaults to base64,
|
||||
which OpenAI-compatible providers such as OpenRouter answer with empty
|
||||
embedding data for models that don't support base64 encoding.
|
||||
"""
|
||||
client, fake = _build_openai_client(
|
||||
monkeypatch,
|
||||
embedding=[0.1] * 8,
|
||||
model="text-embedding-3-small",
|
||||
send_dimensions=False,
|
||||
vector_dimensions=8,
|
||||
)
|
||||
|
||||
await client.embed("hello")
|
||||
|
||||
assert fake.calls[0]["encoding_format"] == "float"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_batch_embed_requests_float_encoding_format(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The batch path must request float embeddings explicitly, like embed()."""
|
||||
client, fake = _build_openai_client(
|
||||
monkeypatch,
|
||||
embedding=[0.1] * 8,
|
||||
model="text-embedding-3-small",
|
||||
send_dimensions=False,
|
||||
vector_dimensions=8,
|
||||
)
|
||||
|
||||
await client.batch_embed({"a": "hello", "b": "world"})
|
||||
|
||||
assert len(fake.calls) == 1
|
||||
assert fake.calls[0]["encoding_format"] == "float"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_embed_reports_missing_embedding_data(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""An explicit encoding_format turns off the SDK's own empty-data check, so
|
||||
a provider answering 200 with no embeddings must still fail legibly."""
|
||||
client, fake = _build_openai_client(
|
||||
monkeypatch,
|
||||
embedding=[0.1] * 8,
|
||||
model="text-embedding-3-small",
|
||||
send_dimensions=False,
|
||||
vector_dimensions=8,
|
||||
)
|
||||
fake.returns_no_data = True
|
||||
|
||||
with pytest.raises(ValueError, match="Embedding count mismatch"):
|
||||
await client.embed("hello")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_batch_embed_reports_short_embedding_data(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A batch answered with fewer embeddings than inputs must name the counts
|
||||
rather than surface a bare zip() error."""
|
||||
client, fake = _build_openai_client(
|
||||
monkeypatch,
|
||||
embedding=[0.1] * 8,
|
||||
model="text-embedding-3-small",
|
||||
send_dimensions=False,
|
||||
vector_dimensions=8,
|
||||
)
|
||||
fake.truncate_data_to = 1
|
||||
|
||||
with pytest.raises(ValueError, match="Expected 2, got 1"):
|
||||
await client.batch_embed({"a": "hello", "b": "world"})
|
||||
|
||||
|
||||
def _build_embedding_settings(
|
||||
env: dict[str, str],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
|
|
@ -449,6 +562,8 @@ def _build_embedding_settings(
|
|||
"EMBEDDING_MODEL_CONFIG__MODEL",
|
||||
"EMBEDDING_MODEL_CONFIG__TRANSPORT",
|
||||
"EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE",
|
||||
"EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE",
|
||||
"EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL",
|
||||
"EMBEDDING_MODEL_CONFIG__MAX_BATCH_SIZE",
|
||||
):
|
||||
monkeypatch.delenv(key, raising=False)
|
||||
|
|
@ -457,6 +572,70 @@ def _build_embedding_settings(
|
|||
return EmbeddingSettings()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("env", "expected"),
|
||||
[
|
||||
# No base_url means real OpenAI, which serves base64 at ~1/3.6 the bytes.
|
||||
({}, "base64"),
|
||||
(
|
||||
{
|
||||
"EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL": "https://api.openai.com/v1"
|
||||
},
|
||||
"base64",
|
||||
),
|
||||
(
|
||||
{
|
||||
"EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL": "https://openrouter.ai/api/v1"
|
||||
},
|
||||
"float",
|
||||
),
|
||||
(
|
||||
{"EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL": "http://localhost:8000/v1"},
|
||||
"float",
|
||||
),
|
||||
({"EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE": "float"}, "float"),
|
||||
(
|
||||
{
|
||||
"EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE": "base64",
|
||||
"EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL": "https://openrouter.ai/api/v1",
|
||||
},
|
||||
"base64",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_resolve_encoding_format(
|
||||
env: dict[str, str], expected: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
s = _build_embedding_settings(env, monkeypatch)
|
||||
assert s.resolve_encoding_format() == expected
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_base64_mode_omits_encoding_format_and_returns_floats(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""base64 mode must request by omission on both paths.
|
||||
|
||||
Naming `base64` explicitly makes the SDK skip its own decode and hand back
|
||||
the raw string, which then fails the dimension check.
|
||||
"""
|
||||
client, fake = _build_openai_client(
|
||||
monkeypatch,
|
||||
embedding=[0.1] * 8,
|
||||
model="text-embedding-3-small",
|
||||
send_dimensions=False,
|
||||
vector_dimensions=8,
|
||||
encoding_format="base64",
|
||||
)
|
||||
|
||||
embedding = await client.embed("hello")
|
||||
batched = await client.batch_embed({"a": "hello", "b": "world"})
|
||||
|
||||
assert all("encoding_format" not in call for call in fake.calls)
|
||||
assert len(embedding) == 8
|
||||
assert [len(vectors[0]) for vectors in batched.values()] == [8, 8]
|
||||
|
||||
|
||||
def test_resolve_send_dimensions_auto_default_dim_returns_false(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
|
|
|
|||
Loading…
Reference in New Issue