* test(embedding): add reproducer for missing encoding_format on openai paths
The openai SDK defaults encoding_format to base64 when it is not passed. OpenAI-compatible providers that don't support base64 embeddings (e.g. OpenRouter with nvidia/nemotron-3-embed-1b:free) return HTTP 200 with empty data, and every embedding call fails with 'No embedding data received'.
* fix(embedding): request float encoding_format on openai embedding calls
The openai SDK defaults encoding_format to base64 when the caller does not pass one. OpenAI-compatible providers that don't support base64 embeddings (e.g. OpenRouter hosting nvidia/nemotron-3-embed-1b:free) answer HTTP 200 with empty embedding data, and every embedding call fails with 'No embedding data received', breaking conclusions, semantic search, and the deriver. Pass encoding_format='float' explicitly on both the single-query and batch call paths.
* test(embedding): cover openai-compatible providers in the live embedding matrix
The existing openai family runs against real OpenAI, which serves base64
embeddings happily, so the matrix passes with or without the #932 fix. Adds an
`openai_compatible_embedding` family (openai transport, third-party base_url)
so the matrix can reach a provider that rejects base64. Empty default_models
keeps it skipped unless LIVE_EMBEDDING_OPENAI_COMPATIBLE_MODELS is set.
Also adds test_live_openai_float_encoding_matches_base64, which pins the other
direction: switching the wire format must not move vectors on real OpenAI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(embedding): keep an explicit embedding-count check on the openai paths
Passing `encoding_format` disables the openai SDK's own empty-data guard, so a
provider answering 200 with missing embeddings surfaced as `IndexError: list
index out of range` on the single path and `zip() argument 2 is shorter than
argument 1` on the batch path. The latter is also #745's signature, which would
have left it with two unrelated causes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(live-llm): correct the openai-compatible embedding matrix env docs
The documented default dimensions said 2048 after the family moved to 3072, and
LIVE_EMBEDDING_OPENAI_COMPATIBLE_SEND_DIMENSIONS was missing entirely. Also
points the example and the coverage note at a model that is actually reachable,
and records that OpenRouter load-balances, so the base64 failure is per-attempt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(embedding): resolve openai encoding_format by mode instead of pinning float
Requesting float unconditionally costs ~3.6x the response bytes of base64 and up
to +83% latency on a 500-item batch, which the default deployment on real OpenAI
pays for nothing: only third-party OpenAI-compatible providers reject base64.
Adds EMBEDDING_MODEL_CONFIG__ENCODING_FORMAT_MODE, mirroring dimensions_mode.
`auto` keeps base64 when no base_url override is set or it points at
api.openai.com, and picks float elsewhere. The format is still always sent
explicitly, since the SDK otherwise injects base64 on its own.
Also corrects the _validate_embedding_count docstring, which said "fewer" where
the guard is an inequality.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(embedding): request base64 embeddings by omission, not by name
The openai SDK decodes a base64 response only when it injected the default
itself; naming any format makes it skip the decode and hand back the raw string,
which then fails the dimension check with "Expected 1536, got 8192". base64 mode
therefore has to omit the kwarg rather than pass it.
The unit fake returned float lists whatever was asked for, so it could not catch
this. It now mirrors the SDK and returns a base64 string for a named base64
request, which fails against the previous commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Aakash Kattelu <aakash@plasticlabs.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix: resolve tiktoken encoding without constructing the embedding client
EmbeddingClient.encoding forced full client construction, which raises
'OpenAI API key is required' even though tiktoken needs no credentials.
The document dedup tie-break (src/crud/document.py) only needs .encoding
for token counting, so any test hitting that path fails in environments
without embedding keys — notably CI for pull requests from forks, where
repo secrets are unavailable (e.g. #908's test-python job failing on
tests/crud/test_document.py::test_duplicate_rejection_reinforces_existing).
Resolve the encoding from the configured model directly, falling back to
cl100k_base, and only reuse the underlying client's encoding when it has
already been constructed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: make embedding batch size configurable
Add optional max_batch_size to the embedding model config
(EMBEDDING_MODEL_CONFIG__MAX_BATCH_SIZE) to cap texts per request for
OpenAI-compatible providers with smaller limits than OpenAI's, such as
DashScope text-embedding-v4 (10) and Alibaba Bailian
qwen3.7-text-embedding (20). When unset, native provider defaults are
preserved (OpenAI 2048, Gemini 100).
Fixes#687.
* test(embedding): cover Gemini batching and config fallbacks per review
- Gemini transport now tested for configured batch splitting and the 100
default fallback
- OpenAI unset default (2048, single request) explicitly covered
- env-parsing test now asserts the value survives resolve_embedding_model_config
- docs: 100 is the client's conservative Gemini default, not a native limit
* test(embedding): assert provider batch-size defaults
---------
Co-authored-by: adavyas <adavyasharma@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(llm): support per-request provider timeouts
* fix(llm): convert Gemini timeout to milliseconds
* fix(llm): validate Gemini HTTP options
* test(llm): type Anthropic stream context args
* test(llm): live per-request timeout coverage for all providers
Two live checks per provider: a generous timeout asserted at the SDK
call boundary, and a tight timeout that must abort well under the 600s
client default. Gemini's async transport can be aiohttp, so its tight
timeout surfaces as asyncio.TimeoutError rather than httpx.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(tests): drop extra blank line in anthropic backend test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(llm): validate provider_params.timeout at config load
Move the timeout coercion into src.config as coerce_provider_timeout and
run it from a field validator on ModelOverrideSettings.provider_params, so
a bad value in config.toml/env fails at startup with the exact config path
instead of surfacing per-request as a retried 500. Good values normalize
to float seconds at load. The per-request guard in src.llm.backend now
delegates to the same coercion (wrapping ValueError in ValidationException)
and continues to cover extra_params passed programmatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: document provider_params.timeout load-time validation and gotchas
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(llm): address review nits on timeout plumbing
Apply eisene's review feedback:
- Rename PROVIDER_TIMEOUT_ERROR → PROVIDER_TIMEOUT_ERROR_TEXT
- Move request_timeout_from_extra_params from backend.py (pure
dataclasses) to request_builder.py (request assembly)
- Add comment explaining Gemini's ms timeout conversion
- Generalize _normalize_extra_params with _strip_none_params helper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Aakash Kattelu <aakash@plasticlabs.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: add model config option for json_object mode
* fix: catch possible validation error from structured output
* fix(llm): harden structured_output_mode json_object path
Follow-up fixes to the json_object structured-output mode for
OpenAI-compatible providers without Structured Outputs support:
- runtime: carry structured_output_mode onto the per-attempt fallback
config (select_model_config_for_attempt dropped it, silently sending
json_schema to a provider that can't parse it)
- backend: return a graceful empty on a contentless json_object
response instead of raising, matching the json_schema path, and
preserve token usage by normalizing the response
- backend: narrow the parse-failure catch to BadRequestError only, so
transient JSONDecodeError/ValidationError propagate to retry/fallback
instead of being swallowed to empty on the first attempt
- config: reject structured_output_mode on non-openai transports
(silent no-op otherwise); trim docs to the deriver, the only
structured-output feature
- backend: validate clean JSON before repair, cache the schema
instruction, and share json_object setup between complete()/stream()
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(llm): consolidate structured-output repair, drop dead seam
Fold the OpenAI backend's three structured-output repair sites
(LengthFinishReasonError, parsed=None, json_object) into the one shared
_parse_or_repair_structured_content helper, gated by an empty_on_missing
flag: json_object returns a graceful empty on a contentless response so a
loose provider can't crash the call, while json_schema raises so the
retry/fallback chain engages.
Delete the dead execute_structured_output_call seam and its only
collaborators (attempt_structured_output_repair, StructuredOutputFailurePolicy)
— it was never called and its single-shot validate/repair/empty model
conflicts with the retry behavior in honcho_llm_call.
No behavior change. Adds tests covering the json_schema parse fallbacks
(repair, refusal passthrough, no-content raise).
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(deriver): Remove connection retry logic and add jitter to polling interval
* chore(docs): Update changelog and document new configurations
* chore: increment version numbers
* feat(db): add connection retry, adaptive deriver polling, and pool metrics
Add resilience and visibility for DB connection handling under transaction-
pooler (Supavisor) saturation, where client-connection limits get exhausted
across many tenants.
- get_db/tracked_db now force an eager pool checkout with bounded exponential
backoff (tenacity), retrying SQLAlchemy TimeoutError + OperationalError so
transient pooler rejections degrade gracefully instead of 500ing. Toggle via
DB_CONNECTION_RETRY_ENABLED (+ delay/backoff knobs); ~10s default budget.
- Deriver polling backs off when idle or erroring (base -> max, x2 each cycle)
and snaps back to base on claimed work, cutting steady-state query load.
Toggle via DERIVER_POLLING_BACKOFF_ENABLED (+ max/multiplier).
- Add scrape-time db_pool_connections Prometheus gauge (checked_out/checked_in/
size/overflow, labeled api|deriver), registered in both the API lifespan and
the deriver metrics server.
- Make SqlalchemyIntegration explicit in both Sentry inits; wrap connection
acquisition in a db.pool.acquire span and capture live pool stats on
retry-exhaustion.
* feat(db): add acquisition counter and in-flight query gauge
Build on the pool-connection metrics with two signals that turn detection
into diagnosis under transaction-pooler saturation:
- db_connection_acquisitions{outcome=ok|retried|exhausted}: counts how often
connection checkout retries through pooler rejection — the alertable early
warning before requests start failing.
- db_queries_in_flight: statements actually executing on the wire (via
SQLAlchemy cursor-execute events, drift-proof across query errors). Pairs
with checked_out: the gap reveals connections held but parked (the "idle in
transaction during an external call" antipattern). Labeled namespace +
instance_type only; gated on METRICS.ENABLED for zero overhead when off.
Add DB-free unit tests for retry outcomes, polling backoff, and in-flight
gauge drift handling.
* fix: address CodeRabbit review on PR #758
- db: roll back the session on a retryable checkout failure before
retrying — a failed autobegin can leave it pending-rollback, making the
next db.connection() raise instead of re-checking-out cleanly. Cheap
Python-side cleanup when no connection was bound.
- metrics: guard DBPoolCollector.collect() so a pool-read/import hiccup
can't raise and abort the whole /metrics scrape (Prometheus drops ALL
metrics if any collector raises) — log and fall back to empty.
* fix(db): lazy retrying session + review fixes for connection backoff
Address Codex/CodeRabbit review on PR #758.
- Replace eager checkout with HonchoAsyncSession: a lazy AsyncSession that
checks out its connection (with retry) on the first DB-touching call, not at
construction. Request handlers doing non-DB work (embedding/file/LLM) before
their first query no longer pin a connection across it, while the API path
still gets checkout retry. Only the checkout is retried — the statement runs
once via super(), so writes are never duplicated. Tracing's set_config moves
into the same lazy acquire hook.
- Roll the session back on a retryable checkout failure before retrying, so a
failed autobegin can't leave it pending-rollback.
- Lower default POOL_TIMEOUT to 5s and validate it stays under the retry budget
for pooled (non-null) POOL_CLASS; update config.toml.example and v2/v3 docs.
- Clamp pool overflow gauge to >= 0 (was negative before the pool fills).
- Remove double-sleep in the deriver idle poll (true backoff cap, not 2x);
make in-flight instrumentation registration idempotent.
- Tests: HonchoAsyncSession lazy/idempotent acquire, statement-runs-once,
tracing, commit/rollback flag reset, get_db no-acquire-at-entry, polling-loop
single-sleep, and the POOL_TIMEOUT/retry-budget validator.
* fix(db): cover all DB-touching session methods; clear flag on close/reset
Address Codex follow-up review on PR #758 (polish, no behavior-critical bug).
- HonchoAsyncSession: wrap get/get_one/stream/stream_scalars/delete in addition
to execute/scalar/scalars/flush/merge/refresh/commit, so the "lazy checkout
with retry on first DB use" guarantee has no holes. connection() stays
unwrapped (acquire_connection_with_retry calls it — wrapping would recurse).
- Reset the acquired flag on close()/reset() too, so a session reused after
close/reset re-acquires (and re-wraps retry) on its next DB use.
- Fix stale comments: connection retry now applies lazily to the request path
via HonchoAsyncSession (config.py), and the FakeSession helper note.
- Tests: close/reset flag reset, and get/delete route through acquisition.
* feat(embedding): add dimensions_mode for OpenAI dimensions= forwarding
Add EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE (auto|always|never) controlling
whether the dimensions= parameter is forwarded on OpenAI embeddings.create
calls. auto (default) sends it when the operator explicitly set
EMBEDDING_VECTOR_DIMENSIONS and the configured model is not on the
known-rejecting allowlist (currently text-embedding-ada-002).
The provenance check (was VECTOR_DIMENSIONS explicitly set?) lives as
EmbeddingSettings.resolve_send_dimensions() because it needs access to
model_fields_set, which the standalone resolver does not have. The
resolved boolean is passed into _EmbeddingClient at construction time;
the client never inspects mode or provenance.
Also pins cloudevents <2.0 — 2.0.0 reorganized the package and dropped
cloudevents.conversion and cloudevents.http, which src/telemetry/emitter.py
imports. The original `>=1.12.0` constraint allowed the broken 2.0 resolve.
With the pin, the imports resolve cleanly and the basedpyright warning
cascade (37+ warnings about unknown types) disappears.
Drive-by cleanups (all unnecessary cast/ignore comments flagged by
basedpyright after the cloudevents downgrade):
- vector_store/lancedb.py, tests/conftest.py, and
tests/deriver/test_vector_reconciliation.py — drop dead pyright ignores
- sdks/python/src/honcho/http/{async_,}client.py — drop unnecessary
cast(datetime, ...) (parsedate_to_datetime already returns datetime)
- vector_store/turbopuffer.py — cast(Any, rows) for the upsert_rows
TypedDict that the SDK exposes but our row builder doesn't satisfy
- tests/test_datetime_parsing.py — ignore reportArgumentType on the
test that deliberately passes wrong types to assert raises
* feat(models): honor EMBEDDING_VECTOR_DIMENSIONS in pgvector columns
* feat(startup): atomic swap dim-vs-MIGRATED guard for runtime schema validator
Add src/startup/embedding_validator.py that introspects the actual pgvector
column dim at boot and refuses to start if it does not match
EMBEDDING_VECTOR_DIMENSIONS. Runs after the DB pool is up and before the
embedding client is constructed, in both src/main.py (FastAPI lifespan) and
src/deriver/__main__.py.
Implementation details:
- Schema-qualified pg_attribute join through pg_class/pg_namespace respects
DB.SCHEMA rather than relying on search_path
- Bounded retry (3 attempts, 1s backoff) for transient introspection failure,
then fail-closed with "could not validate embedding schema" — uncertainty
is not a green light to serve traffic
- External-store sampler (turbopuffer, lancedb) enumerates workspaces from
the application DB and probes their lazy-created namespaces; current
per-namespace probe is a no-op stub since the SDKs do not expose
uniform dim introspection — full enumeration is left to
`configure_embeddings --report` in Phase 3
Atomic guard swap: deletes the old dim-vs-MIGRATED config validator (which
forbade non-1536 pgvector unless MIGRATED=True) in the same commit as the
new runtime validator. There is no release window where non-1536 pgvector
can start unprotected. The 9 dual-write branches that use VECTOR_STORE.MIGRATED
remain untouched and load-bearing for legacy-tenant backend swaps.
VECTOR_STORE_DIMENSIONS deprecation: drop the "must match" raise; in
propagate_namespace, check model_fields_set and emit logger.warning +
DeprecationWarning (DeprecationWarning alone is filtered by Python's default
config and would not reach operators). Always overwrite with
EMBEDDING.VECTOR_DIMENSIONS regardless.
Test changes:
- tests/test_models_vector_dim.py: Phase 1's VECTOR_STORE_TYPE=lancedb +
MIGRATED=true escape hatches removed; the test now passes on plain
EMBEDDING_VECTOR_DIMENSIONS=768
- tests/llm/test_model_config.py: the two tests asserting the old guards
replaced with tests for the new deprecation + acceptance behavior
- tests/startup/test_embedding_validator.py: 10 new tests — dim assertion
logic (pass/mismatch/missing/unbounded/non-public-schema), fail-closed
retry budget, real-test-DB pass, real-DB ALTER-then-validate, deprecation
warning capture, non-1536 + pgvector + MIGRATED=false at config time
* feat(scripts): add configure_embeddings bootstrap CLI
Adds scripts/configure_embeddings.py alongside the other one-off scripts
(provision_db, migrate_db, generate_jwt_secret, etc.). Invoked as
`uv run python scripts/configure_embeddings.py` — same convention as the
existing scripts in that directory, including the sys.path shim that
lets src.* imports resolve when run directly.
Bootstrap step for self-hosted installs at a non-default
EMBEDDING_VECTOR_DIMENSIONS — runs between `alembic upgrade head` and
starting the API/deriver.
pgvector ALTER safety (single transaction):
- LOCK TABLE {schema}.documents, {schema}.message_embeddings IN ACCESS
EXCLUSIVE MODE — closes the TOCTOU window between population check
and ALTER
- COUNT(*) WHERE embedding IS NOT NULL on both tables; refuse with a
non-zero exit if either is populated (ALTER ... USING NULL would
silently wipe those vectors)
- Snapshot HNSW index DDL from pg_indexes; drop, ALTER, recreate from
the captured DDL so operator-set HNSW params (m, ef_construction)
survive the round trip
External vector stores (turbopuffer, lancedb) are never created or
modified — namespaces are per-workspace and lazy-created on first write.
The --report mode enumerates workspaces and collections from the
application DB, derives the expected namespaces via
get_vector_namespace(), and prints a per-namespace status table.
CLI modes (mutually exclusive):
- (default) interactive: print plan, prompt to confirm
- --dry-run: print plan and exit 0 without touching the DB
- --yes: apply without prompt
- --report: print external-store namespace inventory and exit
Also updates src/startup/embedding_validator.py error-message paths and
docs/v3/contributing/configuration.mdx invocations to point at the new
script location.
Tests cover plan no-op, plan needs-alter, plan raises on missing column,
ALTER + HNSW round-trip, refuse-when-populated (monkeypatched count to
avoid wiring the full workspace/peer/collection/document FK chain just
to land one vector row), and idempotency.
* docs: add changing-embeddings operations page
Document the supported way to change EMBEDDING_VECTOR_DIMENSIONS or
EMBEDDING_MODEL_CONFIG__MODEL on a Honcho deployment: provision a new
deployment at the desired configuration, replay source data out of
band, cut over at the application layer.
The page explains the asymmetry:
- Dimension is machine-enforced as immutable. The startup validator
introspects pg_attribute and crashes the API/deriver on mismatch.
- Model is operator-owned. There is no persistent metadata recording
which model produced each vector, so a same-dim model swap is
silently undetectable — flagged with a Warning callout.
Also documents the truncation edge case (text-embedding-3-large truncated to 1536 with EMBEDDING_VECTOR_DIMENSIONS left at default)
and the DIMENSIONS_MODE=always mitigation, plus a pointer that
storage-backend swap (VECTOR_STORE_MIGRATED + reconciler) is a distinct operation unaffected by this work.
Registers the page in docs/docs.json under the Self-Hosting nav group
and cross-links from configuration.mdx.
* fix(embedding): correct turbopuffer regex + tighten DIMENSIONS_MODE docs
- Turbopuffer attribute type for a vector column is `[N]f32` / `[N]f16` /
`[N]i8`, not `f32_vector(N)` as the earlier probe assumed. The earlier
regex returned None for the real SDK format, so existing Turbopuffer
namespaces would have been reported as "missing" instead of validated
for mismatch. Regex switched to `\[(\d+)\]` which is the
vendor-stable shape. Test cases rewritten to lock the actual format.
- docs/v3/contributing/configuration.mdx had a contradictory pair of
bullets: 223 said explicit 1536 makes `auto` forward dimensions=, 224
said `auto` would skip the parameter because 1536 is the default.
Operators reading both would (rightly) conclude they need `always`
even when `auto` would work. Rewrote both bullets so:
- `auto` is provenance-driven (explicit-set, not non-default-value).
- `always` is positioned as defense-in-depth for config layers that
might strip explicit default-valued envs, not the only path for
same-as-default truncation.
* fix(embedding): address PR #678 review comments
CodeRabbit + Rajat review feedback. All actionable items addressed
except two false-positives (responded on PR).
Bug fixes:
- deriver telemetry leak: validator was called outside try/finally so
shutdown_telemetry() did not run on validation failure. Moved inside.
- _emit_report printed "no effect with pgvector" unconditionally,
including from implicit post-apply calls. Added is_report_mode flag;
only print on explicit --report.
- LanceDB and Turbopuffer probes returned None when the namespace
existed but its schema was malformed (no vector field / unparseable
type string), silently bucketing real corruption as "missing"
(lazy-create) and letting it pass the startup validator. Now raise
VectorStoreError with actionable diagnostics; None remains valid only
for "namespace does not exist."
- Startup validator only sampled message namespaces; added a parallel
Collection-row sample so document namespaces are probed too, with the
same dim assertion. Mirrors the --report path.
Hygiene:
- StartupValidationError now subclasses HonchoException so existing
exception handlers recognize it. ValidationException is @final and
has 422 request-validation semantics that would be misleading here.
- scripts/configure_embeddings.py main() no longer spins up two event
loops. engine.dispose() moved into a try/finally inside _async_main
so cleanup runs in the same loop as the pipeline.
- Replaced hand-rolled retry loop with tenacity.AsyncRetrying; same
fail-closed semantics, less code, before_sleep_log for visibility.
- Added _validate_identifier() defense-in-depth: DB.SCHEMA and HNSW
index names are regex-checked against [A-Za-z_][A-Za-z0-9_]* before
SQL interpolation. Operator config + DB catalog are not user input
under the current threat model, but the constraint is cheap to gate.
Test + docs:
- test_app_settings_accepts_non_1536_with_any_vector_store_configuration
now actually exercises turbopuffer (was missing); supplies a dummy
TURBOPUFFER_API_KEY to satisfy the model_validator.
- changing-embeddings.mdx: hyphenated "out-of-band" per reviewer style.
* fix: modify conftest to fix ci
* fix: ci tests for typescript server
* fix: Add JSON repair for truncated LLM responses across all providers and Gemini thinking budget support
LengthFinishReasonError from OpenAI-compatible providers (custom, openai, groq) was crashing the deriver
with 14k+ occurrences in production. The vLLM path already had repair logic but it was gated on
provider=="vllm", unreachable when routing through litellm as a custom provider.
- Extract shared _repair_response_model_json() helper for all providers
- Catch LengthFinishReasonError in OpenAI/custom parse() path and repair truncated JSON
- Add repair fallback to Anthropic and Gemini response_model paths
- Add repair fallback to Groq response_model path
- Pass thinking_budget_tokens to Gemini 2.5 models via thinking_config
- Add 14 tests covering repair paths for all providers and Gemini thinking budget
Fixes HONCHO-YC
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: live llm integration tests
* feat: Consistent Model Config Protocol
* fix: migrate the remaining app callers off the legacy llm_settings path
* fix: Docs and regression tests
* fix: refactor llm runtime path to model-config-only API
* fix: refactor config to nested model-config source of truth
* fix: refactor llm streaming and tool dispatch through backends
* fix: cut over llm config to nested model_config only
* fix: collapse vllm and custom into openai_compatible transport
* feat: refactor llm config to explicit transports and bare model ids
* feat: (embed) Add configurability for embedding model
* fix: tests for embedding provider
* fix: Address Review Comments
* fix: (llm) remove Groq backend and per-vendor base URLs
* chore: move llm tests
* fix: (llm) address review findings — config regressions, backend bugs, dead code
* fix: address backend end silly errors
* chore: (docs) update configuration and self-hosting guides
* chore: fix tests
* fix: address code rabbit comments
* fix: add validation to the dream settings
* fix: further address code rabbit comments
* fix: Address Code Rabbit Comments
* fix: Another round of code rabbit
* fix: Address Code Rabbit Nits
* fix: tests
* refactor: rename thinking validator to reflect transport scope
_validate_anthropic_thinking_minimum only enforces the >=1024 rule for
Anthropic and no-ops for other transports, so the name was misleading
now that it's shared across ConfiguredModelSettings, FallbackModelSettings,
and ModelConfig. Renamed to _validate_thinking_constraints with a docstring
clarifying per-transport behavior. No logic change.
* fix(config): drop transport-specific thinking params when env override changes transport
_fill_defaults_for_nested_field previously preserved the default MODEL_CONFIG's
thinking_budget_tokens/thinking_effort across a transport override. This leaked
Gemini-family defaults (e.g. thinking_budget_tokens=1024) into OpenAI-transport
overrides, and the OpenAI backend then correctly rejected the unsupported param
at call time (OpenAI uses reasoning.effort, not a token budget).
The helper now strips thinking_budget_tokens and thinking_effort from the
default dict when the env override supplies a transport different from the
default's. Explicit thinking params in the override are preserved.
* fix(config): apply thinking-param strip to dialectic level merge too
DialecticSettings._merge_level_defaults does its own inline MODEL_CONFIG
merge (parallel to _fill_defaults_for_nested_field), so the previous fix
missed dialectic-level overrides. E.g. flipping
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__TRANSPORT from gemini (default)
to openai still leaked the default thinking_budget_tokens=0 into the
openai config, which the OpenAI backend then rejected at call time.
The level-merge path now applies the same 'strip transport-specific
thinking params when transport changes' rule as the generic helper.
Added a regression test exercising the merge validator directly.
* refactor(llm): wire ModelConfig knobs through, prune clients.py migration leftovers
Three connected fixes to finish carving the LLM stack out of src/utils/clients.py
and into src/llm/:
1. Propagate ModelConfig tuning knobs into backend calls.
honcho_llm_call_inner built extra_params from only {json_mode, verbosity},
silently dropping top_p, top_k, frequency_penalty, presence_penalty, seed,
and operator-supplied provider_params from any ModelConfig. Thread the
selected config through ProviderSelection and merge
build_config_extra_params(selected_config) into extra_params; per-call
kwargs still win over provider_params defaults. Makes
_build_config_extra_params public as build_config_extra_params so
clients.py and request_builder.py share one translation. Adds
TestModelConfigExtraParamsPropagation covering OpenAI/Anthropic knob
propagation, provider_params passthrough, and per-call override
precedence.
2. Drop dead extract_openai_* duplicates in clients.py.
extract_openai_reasoning_content, extract_openai_reasoning_details, and
extract_openai_cache_tokens had no callers outside their own definitions
— the live implementations live in src/llm/backends/openai.py. -103
lines from clients.py.
3. Unify on ModelTransport, delete SupportedProviders.
The "google" vs "gemini" split forced a _provider_for_model_config
translation shim in two places. Replace all SupportedProviders usages
with ModelTransport, rename CLIENTS["google"] → CLIENTS["gemini"],
update provider branches + LLMError labels + reasoning-trace entries
accordingly. Trace JSONL now writes "provider": "gemini" instead of
"google" — consistent with the broader env-var rename cutover.
Also tidies up pre-existing basedpyright findings in tests/llm/test_model_config.py
(pydantic before-validator dict inputs + descriptor-proxy call).
ruff: clean. basedpyright: 0 errors, 0 warnings. Tests: 153/153 pass across
tests/utils/test_clients.py, tests/utils/test_length_finish_reason.py,
tests/llm/, tests/dialectic/, tests/deriver/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): finish the src/utils/clients.py → src/llm/ migration
honcho_llm_call_inner now delegates to request_builder.execute_completion
and execute_stream instead of re-implementing backend call scaffolding
inline. The new _effective_config_for_call helper carries per-call kwargs
(temperature, stop_seqs, thinking_budget_tokens, reasoning_effort) onto
the selected ModelConfig — or synthesizes a minimal config for the
test-only callers that pass provider+model directly. max_output_tokens
is zeroed on the effective config to preserve the current
"per-call max_tokens wins" semantic; honoring ModelConfig.max_output_tokens
is a separable correctness concern.
Side effect of routing through the new path: ConfiguredModelSettings'
thinking_budget_tokens validator now fires on synthesized configs.
test_anthropic_thinking_budget was asserting that a sub-1024 budget
propagated to Anthropic — bumped to 1024 to match what Anthropic actually
accepts.
Unified client construction. Promoted the cached client factories in
src/llm/__init__.py (get_anthropic_client, get_openai_client,
get_gemini_client, get_{anthropic,openai,gemini}_override_client) to
public API and added them to __all__. Promoted
credentials._default_transport_api_key → default_transport_api_key.
Deleted the duplicate _build_client and _default_credentials_for_provider
from clients.py; _client_for_model_config now falls through to the
public factories. CLIENTS dict and _get_backend_for_provider stay as the
mockable seam for the ~50 patch.dict(CLIENTS, {...}) test call sites.
Wired operator-configurable Gemini cached-content reuse end-to-end.
PromptCachePolicy moved from src/llm/caching.py into src/config.py so
ModelConfig can reference it as a field without a circular import;
caching.py re-exports the name for existing imports. Added
cache_policy: PromptCachePolicy | None on ConfiguredModelSettings,
FallbackModelSettings, ResolvedFallbackConfig, and ModelConfig.
resolve_model_config, _resolve_fallback_config, and
_select_model_config_for_attempt copy the field through.
honcho_llm_call_inner passes effective_config.cache_policy into
execute_completion / execute_stream, so operators opt in via
e.g. DERIVER_MODEL_CONFIG__CACHE_POLICY__MODE=gemini_cached_content
and the selection actually fires instead of sitting on a dead path.
New regression test test_cache_policy_reaches_gemini_backend asserts the
PromptCachePolicy object reaches the Gemini backend's extra_params.
ruff + basedpyright: clean. Tests: 154/154 pass across
tests/utils/test_clients.py, tests/utils/test_length_finish_reason.py,
tests/llm/, tests/dialectic/, tests/deriver/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(llm): move all LLM orchestration into src/llm/ and delete clients.py
The 1624-line src/utils/clients.py has been carved up into focused modules
under src/llm/ and deleted. There is now one golden path for LLM
orchestration and no dual entrypoint.
New module layout:
src/llm/
__init__.py thin stable re-export surface
api.py public honcho_llm_call with retry + fallback + tool
loop delegation
executor.py honcho_llm_call_inner (single-call executor); bridges
to request_builder.execute_completion / execute_stream
tool_loop.py execute_tool_loop + stream_final_response, plus
assistant-tool-message and tool-result formatting
runtime.py AttemptPlan dataclass (replaces the loose
ProviderSelection NamedTuple), effective_config_for_call,
plan_attempt, per-retry temperature bump, attempt
ContextVar
registry.py single owner of CLIENTS dict + cached default and
override SDK-client factories + backend/history-adapter
selection + high-level get_backend(config)
conversation.py count_message_tokens, tool-aware message grouping,
truncate_messages_to_fit
types.py HonchoLLMCallResponse, HonchoLLMCallStreamChunk,
StreamingResponseWithMetadata, IterationData,
IterationCallback, ReasoningEffortType, VerbosityType,
ProviderClient
request_builder.py low-level request assembly (ModelConfig → backend
complete/stream); no longer owns credential resolution
credentials.py default_transport_api_key, resolve_credentials
caching.py gemini_cache_store; re-exports PromptCachePolicy
from src.config
backend.py Protocol + normalized result types
history_adapters.py provider-specific assistant/tool message shapes
structured_output.py
backends/ AnthropicBackend, OpenAIBackend, GeminiBackend
handle_streaming_response had no production callers; it is deleted. The
three tests that used it now drive honcho_llm_call_inner(stream=True,
client_override=...) directly, which exercises the same code path the
public API uses.
Dead credential passthrough removed. The ProviderBackend Protocol and
all three concrete backends no longer accept api_key / api_base — those
are baked into the underlying SDK client at registry construction time
and were being del'd everywhere they appeared. request_builder also
stops resolving and forwarding them.
Client construction is unified. The cached default-client factories
(get_anthropic_client, get_openai_client, get_gemini_client) and override
factories (get_*_override_client) are promoted to public API; the
module-level CLIENTS dict populates from them and remains the
patch.dict(CLIENTS, {...}) mocking seam tests rely on. Old duplicate
helpers (_build_client, _default_credentials_for_provider) are gone.
default_transport_api_key is promoted to public.
Application imports now come from src.llm (dreamer, dialectic, deriver,
summarizer, telemetry-adjacent tests). No code imports from
src.utils.clients anywhere in the repo.
ruff: clean. basedpyright: 0 errors, 0 warnings. Tests: 1013/1013 pass
across the entire non-infra test suite (excluding tests/unified,
tests/bench, tests/live_llm, tests/alembic).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(llm): sanitize tool schemas for Gemini's function_declarations validator
Gemini's native-transport function-declarations validator accepts a narrow
subset of JSON-Schema / OpenAPI: type, format, description, nullable, enum,
properties, required, items, minItems, maxItems, minimum, maximum, title.
Anything else — additionalProperties, allOf, if/then/else, $ref, anyOf,
oneOf, $defs, patternProperties — triggers an INVALID_ARGUMENT 400 at call
time.
Our agent tool schemas in src/utils/agent_tools.py use several of those
(additionalProperties: false, allOf + if/then conditionals) because they
were authored for OpenAI strict-mode + Anthropic, which need the richer
vocabulary. GeminiBackend._convert_tools was passing them straight through.
Add _sanitize_schema(): walks the parameters tree and drops unsupported
keywords while preserving semantics for the keywords that hold user data
(properties maps field-name → sub-schema; required / enum are lists of
literals; items is a single sub-schema). Other backends are untouched and
continue to receive the full strict schemas.
Regression tests:
- test_gemini_sanitize_schema_strips_unsupported_keywords: confirms
additionalProperties, allOf + if/then, and $defs are stripped at nested
levels while legitimate fields survive.
- test_gemini_convert_tools_sanitizes_parameters_schema: end-to-end
_convert_tools output has no forbidden keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: fix tool calling syntax for gemini
* refactor(llm): normalize defaults, widen OpenAI reasoning-model routing
* chore: fix test
* fix(llm): address post-migration review feedback
* fix(llm): gemini robustness + dreamer specialist ergonomics
* chore: addres review comments
* chore: (docs) unrelease changelog addition
* chore: (docs) merge commit changes
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Erosika <eri@plasticlabs.ai>
* fix: Inconsistencies in Docs, health endpoint, troubleshooting guide
* fix: (docs) maintain consistency on postgres db name
* chore: (docs) update v2 contributing docs with updates db paths
* docs: overhaul self-hosting docs for provider-agnostic setup
- .env.template: lead with provider options (custom, vllm, google,
anthropic, openai, groq) instead of baking in vendor-specific keys.
All provider/model settings commented out so server fails fast until
configured. Separate endpoint config from per-feature provider+model
from tuning knobs.
- docker-compose.yml.example: fix healthcheck -d honcho -> -d postgres
to match POSTGRES_DB=postgres.
- config.toml.example: reorder and document LLM key section with
OpenRouter and vLLM examples.
- self-hosting.mdx: replace multi-vendor key table with provider options
table. Add examples for OpenRouter, vLLM/Ollama, and direct vendor
keys. Remove duplicated key lists from Docker/manual setup sections.
- configuration.mdx: replace scattered provider docs with provider types
table. Fix Docker Compose snippet to match actual compose file. Note
code defaults as fallback, not recommended path.
- troubleshooting.mdx: add alternative provider issues section (custom
provider config, model name format, Docker localhost, structured
output failures).
* docs: add Docker build troubleshooting for permission errors
- Document BuildKit requirement (RUN --mount syntax)
- AppArmor/SELinux blocking Docker builds on Linux
- Volume mount UID mismatch between host and container app user
- Note in self-hosting docs that Docker path builds from source
* docs: reframe self-hosting as contributor/dev path, point to cloud service
* Revert "docs: reframe self-hosting as contributor/dev path, point to cloud service"
This reverts commit 3e766eb1a9.
* docs: add production compose, model guidance, thinking budget docs
- Add docker-compose.prod.yml for VM/server deployment: no source
mounts, restart policies, 127.0.0.1-bound ports, cache enabled
- Add model tier guidance and community quick-start link to self-hosting
- Document THINKING_BUDGET_TOKENS gotcha for non-Anthropic providers
- Add reverse proxy examples (Caddy + nginx) to production section
- Add backup/restore commands to production considerations
* docs: simplify self-hosting to single provider, restructure config guide
Self-hosting page now defaults to one OpenAI-compatible endpoint
with one model for all features. Moved model tiers, alternative
providers, and per-feature tuning into the configuration guide.
Eliminated duplicate config priority sections, dev/prod split,
and redundant TOML examples.
* docs: merge compose files, restore provider/model to feature sections in .env.template
Single docker-compose.yml.example with dev sections commented out.
Moved PROVIDER and MODEL back alongside each feature in .env.template
so settings stay colocated with their module. Updated self-hosting
docs to reference single compose file.
* fix: broken anchor links, redundant migration step, minor inconsistencies
Fix 4 broken internal links (#llm-provider-setup, #llm-api-keys,
#which-api-keys-do-i-need, #alternative-providers) to point to
correct headings. Remove redundant Docker migration step (entrypoint
already runs alembic). Fix cache URL missing ?suppress=true in
reference config. Fix uv install command to use official method.
* docs: env template ready to use, simplify self-hosting flow
.env.template now has provider/model lines uncommented with
placeholder values — user just sets endpoint, key, and model name.
Thinking budgets default to 0 for non-Anthropic providers.
Self-hosting page: removed 30-line env var wall, LLM setup now
points to the template. Merged duplicate verify sections.
Removed api_key from SDK examples (auth off by default).
* docs: reorder next steps, configuration guide first
* fix: default embedding provider to openrouter for single-endpoint setup
Without this, embeddings default to openai which requires a separate
LLM_OPENAI_API_KEY. Setting to openrouter routes embeddings through
the same OpenAI-compatible endpoint as everything else.
* fix: review issues — hermes page, thinking budget, production wording
Hermes integration page: replaced inline Docker/manual setup with
link to self-hosting guide, added elkimek community link. Removed
old env var names (OPENAI_API_KEY without LLM_ prefix).
Troubleshooting: removed "or 1" from thinking budget guidance.
Self-hosting: softened "production-ready" to "production-oriented"
since auth is disabled by default.
* docs: model examples in template, expanded LLM setup, better verify flow
.env.template: added "e.g. google/gemini-2.5-flash" hints next to
model placeholders so users know the expected format.
Self-hosting: expanded LLM Setup to show the 3 things users need to
set (endpoint, key, model name) with find-replace tip. Added build
time note, deriver log check, and real smoke test (create workspace)
to verify section. Health check now notes it doesn't verify DB/LLM.
* fix: smoke test uses v3 API path, not v1
* docs: clarify deriver metrics port vs Prometheus host port
* fix: remove deprecated memoryMode from hermes config example
* docs: update hermes page to match current memory provider config
Updated config to match hermes-agent docs: removed apiKey (not needed
for self-hosted), added hermes memory setup CLI command, added config
fields table (recallMode, writeFrequency, sessionStrategy, etc.).
Better verification tests: store-and-recall across sessions, direct
tool calling test. Links to upstream hermes docs for full field list.
* fix: invalid THINKING_BUDGET_TOKENS=0 and missing docker/ in image
Comment out THINKING_BUDGET_TOKENS=0 in .env.template — deriver,
summary, and dream validators require gt=0. Dialectic levels also
commented out since non-thinking models don't need the override.
Add COPY for docker/ directory in Dockerfile so entrypoint.sh is
available when docker-compose.yml.example references it.
* chore: Additional troubleshooting step
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
* chore: 3.0 honcho and 2.0 sdks changelog
fix: use PeerContextResponse in peer.ts
* chore: move docs to /v3/, build SDKs
* chore: code review
* feat: [WIP] migrate away from stainless in typescript sdk
* chore: move api from /v2/ to /v3/
* feat: no-stainless typescript with real tests
* feat: migrate python sdk off of stainless
* feat: clean typescript sdk
* chore: add tests for ts http client
* fix: rewrite entire python sdk in new format, update typescript sdk to use `configuration` not `config` for consistency with API
* fix: clean up SDKs, synchronize
* chore: update sdk examples
* chore: update OpenAPI documentation and SDK examples to reflect changes
* fix: better test
* fix: install deps in test runner, improve robustness of streaming in sdk, coderabbit nits
* fix: standardize around camelCase in TS SDK
* refactor: update configuration handling in SDKs to use typed models for workspace, session, and peer configurations
* docs: clarify queue status usage and remove polling methods from SDKs
add claude skills for migrations
* chore: fix links in docs
* feat: add deriver flush mode to bypass batch token threshold
- Introduced `is_deriver_flush_enabled` function to check if flush mode is active.
- Updated `QueueManager` to conditionally apply batch token thresholds based on flush mode.
- Enhanced `UnifiedTestExecutor` to enable flush mode via Redis.
- Added `flush` parameter to test cases to facilitate testing of flush mode behavior.
- Updated various test cases to utilize the new flush functionality.
* feat: implement schedule_dream functionality in SDKs, use in unified test runner
- Added `schedule_dream` method to both Python and TypeScript SDKs for scheduling dream tasks.
- Updated HTTP routes to include endpoint for scheduling dreams.
- Enhanced test runner to utilize the new `schedule_dream` method for scheduling actions.
- Updated TypeScript client to support the new scheduling functionality with appropriate parameters.
* feat: update single deriver task to support multiple observers
- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.
* refactor: update enqueue tests to support deduplication of queue items with multiple observers
- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.
* fix: add backwards compatibility for representation work unit keys and payload observers
* feat: update dialectic configuration and introduce cost calculator
- Adjusted LLM and dialectic settings in `.env.template`, `config.toml.example`, and `src/config.py` to reduce maximum tool output characters and session history tokens for cost efficiency.
- Implemented a new `dialectic_cost_calculator.py` script to estimate costs based on reasoning levels and model pricing.
- Enhanced `DialecticAgent` to utilize minimal tools and adjusted output token settings based on reasoning level to optimize performance and reduce costs.
* feat: add reasoning level to chat input in unified test runner
- Enhanced the `UnifiedTestExecutor` to include a `reasoning_level` parameter in the chat method call.
- Updated the `QueryAction` model to support the new `reasoning_level` attribute, allowing for more nuanced chat interactions.
* feat: run deriver once for multiple observers (#335)
* feat: update single deriver task to support multiple observers
- Changed the `observer` parameter to `observers` as a list in multiple functions across the deriver module.
- Updated the processing logic to handle multiple observers for representation tasks.
- Adjusted related payload and queue management functions to accommodate the new observers structure.
- Modified tests to reflect changes in the representation task handling and ensure proper functionality.
* refactor: update enqueue tests to support deduplication of queue items with multiple observers
- Modified tests in `test_enqueue.py` to reflect changes in the queue item structure, where each message now results in a single queue item containing a list of observers.
- Updated assertions to validate that the `observers` field correctly includes all relevant peers, ensuring proper functionality of the deduplication logic.
- Removed redundant payload matching logic to streamline test cases and improve clarity.
* fix: add backwards compatibility for representation work unit keys and payload observers
* feat: refactor benchmark runners to share common functionality
- Introduced a new `runner_common.py` module containing shared utilities for benchmark test runners, including common argument parsing, client creation, and queue management.
- Updated `BEAMRunner`, `LoCoMoRunner`, and `LongMemEvalRunner` to inherit from `RunnerMixin`, leveraging shared functionality for metrics collection and logging.
- Added `reasoning_level` and `redis_url` parameters to runner constructors for enhanced configuration.
- Streamlined argument parsing by utilizing `add_common_arguments` for shared command-line options across all runners.
* fix: update last_user_message handling to use message content instead of ID
* fix: standardize config vs configuration
---------
Co-authored-by: Vineeth Voruganti <13438633+VVoruganti@users.noreply.github.com>
Rajat Ahuja
Vansh Sharma
JUNZE
Alexei Vedernikov
Eugene Eisenstein
Aakash Kattelu
Harish Kukreja
Vineeth Voruganti
adavyas
Eri Barrett
doria